search

rigoudyg / climaf

CliMAF - a Climate Model Analysis Framework - doc at : http://climaf.readthedocs.org/
Other
18 stars 7 forks source link

Importing CliMAF silently #190

Open jypeter opened 3 years ago

jypeter commented 3 years ago

I know I requested some detailed information when importing climaf in #136 , and it is working quite nicely

Now, I'm wondering if there is also a way to import climaf without displaying any information. I thought we were getting the full details when using from climaf.api import * and that it would be more quiet when using just import climaf, but it's just the same

$ python
Python 3.6.11 | packaged by conda-forge | (default, Aug  5 2020, 20:09:42)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import climaf
python => 3.6.11 | packaged by conda-forge | (default, Aug  5 2020, 20:09:42)
[GCC 7.5.0]
---
Required softwares to run CliMAF => you are using the following versions/installations:
ncl 6.6.2 => /modfs/modtools/miniconda3/envs/analyse_3.6_test/bin/ncl
cdo 1.9.6 => /opt/nco/1.9/bin/cdo
nco (ncks) 4.5.2 => /opt/nco-4.5.2/bin/ncks
ncdump such => /prodigfs/ipslfs/dods/jservon/miniconda/envs/cesmep_env/bin/ncdump
Check stamping requirements
nco (ncatted) found -> /opt/nco-4.5.2/bin/ncatted
convert found -> /usr/bin/convert
pdftk found -> /usr/bin/pdftk
exiv2 found -> /ciclad-home/jservon/Evaluation/CliMAF/climaf_installs/climaf_V2.0.0/bin/exiv2
---
CliMAF version = 2.0.0
CliMAF install => /ciclad-home/jservon/Evaluation/CliMAF/climaf_installs/climaf_V2.0.0
Cache directory set to : /data/jypmce/climafcache (use $CLIMAF_CACHE if set)
Cache directory for remote data set to : /data/jypmce/climafcache/remote_data (use $CLIMAF_REMOTE_CACHE if set)
Available macros read from ~/.climaf.macros are : []
>>>

I guess a way to do this could be to set a specific environment variable before the import, and CliMAF could test the value of the variable (if defined) when it starts, and act accordingly, 

>>> os.environ['CLIMAF_INIT'] = 'silent'
>>> os.environ['CLIMAF_INIT']
'silent'
>>> os.getenv('CLIMAF_INIT')
'silent'
>>> import climaf
senesis commented 3 years ago

IMHO, if we avoid to display utilities version number, we could even go a step further, and bypass (as fas as possible) the tests for these utilities, which slows CliMAF startup.

jypeter commented 3 years ago

See an example of changing a CliMAF environment variable on-the-fly before importing CliMAF

The idea would be to disable printing the debug information that gets printed before the actual output of a script (example below)

[jypmce@ciclad-ng bdd]$ python ~/PMIP4/climaf_search_v2.py lgm tas Amon

python => 3.6.11 | packaged by conda-forge | (default, Aug  5 2020, 20:09:42)
[GCC 7.5.0]
---
Required softwares to run CliMAF => you are using the following versions/installations:
ncl 6.6.2 => /modfs/modtools/miniconda3/envs/analyse_3.6_test/bin/ncl
cdo 1.9.6 => /opt/nco/1.9/bin/cdo
nco (ncks) 4.5.2 => /opt/nco-4.5.2/bin/ncks
ncdump such => /prodigfs/ipslfs/dods/jservon/miniconda/envs/cesmep_env/bin/ncdump
Check stamping requirements
nco (ncatted) found -> /opt/nco-4.5.2/bin/ncatted
convert found -> /usr/bin/convert
pdftk found -> /usr/bin/pdftk
exiv2 found -> /ciclad-home/jservon/Evaluation/CliMAF/climaf_installs/climaf_V2.0.0/bin/exiv2
---
CliMAF version = 2.0.0
CliMAF install => /ciclad-home/jservon/Evaluation/CliMAF/climaf_installs/climaf_V2.0.0
Cache directory set to : /data/jypmce/climafcache (use $CLIMAF_CACHE if set)
Cache directory for remote data set to : /data/jypmce/climafcache/remote_data (use $CLIMAF_REMOTE_CACHE if set)
Available macros read from ~/.climaf.macros are : []

*** Starting: Wed Jun 30 13:17:33 2021

Looking for variable "tas @ Amon", for experiment "lgm @ CMIP6"...

Found 4 model(s):

model_list = ['AWI-ESM-1-1-LR', 'INM-CM4-8', 'MIROC-ES2L', 'MPI-ESM1-2-LR']
model_list = AWI-ESM-1-1-LR,INM-CM4-8,MIROC-ES2L,MPI-ESM1-2-LR

AWI                 AWI-ESM-1-1-LR      r1i1p1f1       gn
                    [3901-4000]
                    Nb years =   100
                    Nb files =  20 - Total size =   50.27 (Mb)
                    Size / year =    0.50 (Mb / year)
                    root = /bdd/CMIP6/PMIP/AWI/AWI-ESM-1-1-LR/lgm/r1i1p1f1/Amon/tas/gn/latest

INM                 INM-CM4-8           r1i1p1f1       gr1
                    [1900-2099]
                    Nb years =   200
                    Nb files =   2 - Total size =  118.59 (Mb)
                    Size / year =    0.59 (Mb / year)
                    root = /bdd/CMIP6/PMIP/INM/INM-CM4-8/lgm/r1i1p1f1/Amon/tas/gr1/latest

MIROC               MIROC-ES2L          r1i1p1f2       gn
                    [3200-3299]
                    Nb years =   100
                    Nb files =   1 - Total size =   21.83 (Mb)
                    Size / year =    0.22 (Mb / year)
                    root = /bdd/CMIP6/PMIP/MIROC/MIROC-ES2L/lgm/r1i1p1f2/Amon/tas/gn/latest

MPI-M               MPI-ESM1-2-LR       r1i1p1f1       gn
                    [1850-2349]
                    Nb years =   500
                    Nb files =  25 - Total size =  194.92 (Mb)
                    Size / year =    0.39 (Mb / year)
                    root = /bdd/CMIP6/PMIP/MPI-M/MPI-ESM1-2-LR/lgm/r1i1p1f1/Amon/tas/gn/latest

Accumulated (selected) data size for all models =    0.38 (Gb)

Models that had no obvious problems:
[['AWI-ESM-1-1-LR', 'r1i1p1f1', 'gn'], ['INM-CM4-8', 'r1i1p1f1', 'gr1'], ['MIROC-ES2L', 'r1i1p1f2', 'gn'], ['MPI-ESM1-2-LR', 'r1i1p1f1', 'gn']]

  Longuest period available (at least one model) = 500
  Longuest period for ALL models = 100

*** Script finished: Wed Jun 30 13:19:53 2021
senesis commented 3 years ago

Solved through #216 . Messages go through the log mechanism, and can so be discarded by setting a high log level (e.g. 'critical')