search

ORAC-CC / orac

Optimal Retrieval of Aerosol and Cloud
GNU General Public License v3.0
30 stars 19 forks source link

Add external SEVIRI_ML to preproc #74

Closed danielphilipp closed 1 month ago

danielphilipp commented 2 years ago

Again, a try to include external SEVIRI_ML (https://github.com/danielphilipp/seviri_ml) functionalities to the ORAC preprocessor. Functionalities are:

(1) a SEVIRI-specific CMA/CPH as from the beginning (2) a CTP estimate + variance which are written to a new preproc file (.ctp.nc), exactly similar to Gareths approach (3) a multilayer flag + multilayer probability + uncertainty, which are written to the .clf.nc file

With the preproc driver switches USE_SEVIRI_ML_CMA_CPH, USE_SEVIRI_ML_CTP_FG and USE_SEVIRI_ML_MLAY one can turn those functionalities on or off.

If desired, the external Tensorflow ANNs are called in cloud_typing_pavolonis.F90. I tested the external ANNs quite extensively. They work on basic Linux machines but also on large batch queue systems where MPI is used around ORAC.

To compile the Fortran (and C) interfaces of SEVIRI_ML just run seviri_ml/get_py_config.sh to get your Python include/lib directory and the numpy include directory of your currently activated python. Then insert the Python and Numpy include directory into the seviri_ml/make.config file. Change compiler and flags if desired and run make.

I added seviri_ml placeholders to the ORAC_LIB file. Open the ORAC_LIB file and insert the SEVIRI_ML directory as SEVIRI_ML_LIB and SEVIRI_ML_INCLUDE. Also insert the Python library and include path toSEVIRI_ML_PYTHON_LIB SEVIRI_ML_PYTHON_INCLUDE (as shown by seviri_ml/get_py_config.sh). Uncomment the last two lines in the ORAC_LIB file to actually link and include the necessary seviri_ml stuff. In the LIBS += ... please change the -lpython3.X to the library name of your python version. (also shown by seviri_ml/get_py_config.sh).

I recommend Python 3.8 with Tensorflow 2.4.1

simonrp84 commented 2 years ago

Thanks for making this PR! I'll try to take a look next week, and hopefully someone else can do so in the near future too.

danielphilipp commented 2 years ago

in commit c02d564 "Further improvements and fixes" I basically did:

  1. Allowed RTTOV calculations up to higher satellite zenith angles of 85 degrees, as RTTOV 13 is now default.
  2. Only remove Rayleigh scattering if opts%rt_ir%addsolar is True as otherwise Rayleigh scattering is removed without being included (as Martin showed).
  3. Simplified seviri_neural_net_preproc.F90 with some subroutines.
  4. Added missing concatenation of the CTP filename string.
  5. Added 't' to the variables to be read from ECMWF file.
simonrp84 commented 2 years ago

Sorry to be annoying @danielphilipp but could you make that recent commit as a separate PR? The changes you made there are distinct from the seviri ML and it would be easier to treat as two separate PRs, otherwise we risk confusion and stale code. The changes in c02d564 all look OK to me, so it should be simple to merge once a PR is made :)

danielphilipp commented 2 years ago

You are right. I removed c02d564 because it is unrelated to SEVIRI_ML integration but I kept (3) and (4) from this commit as those are related to SEVIRI_ML. Once this PR is merged I will open a new one with the other changes from c02d564.

danielphilipp commented 2 years ago

If you already downloaded SEVIRI_ML I would recommend to update the code with the latest version on Github because the Tensorflow-based CTP models were corrupt and are correctly working now. Theano-based CTP models were okay.