Code freeze for v2.2.0

[jvegreg]

Today we freeze the code for the 2.2.0 release. I ask for your help to review and merge as many pending issues in the milestone before today at 22:00 CES. Thanks!

@ESMValGroup/scientific-lead-development-team @ESMValGroup/technical-lead-development-team

[valeriupredoi]

@jvegasbsc I see you are more lenient than @bouweandela and myself for last release, we decided to freeze it in the afternoon :grin: For the freeze, you know the drill - create a branch called something like release_2.2.0 and ask @mattiarighi to set nobody-move-a-finger permissions on it ie nobody pushes to it; we should also ask the people to try run as many recipes as possible with the release branch; also, we should get on a zoom call next week when we do the release :+1:

[jvegreg]

Branch release_2.2.0 created. @remi-kazeroni can you please protect it?

[remi-kazeroni]

Branch release_2.2.0 created. @remi-kazeroni can you please protect it?

I don't think I have the right to do that. I can't access the settings of the repo. My guess is that only @axel-lauer can do that at the moment.

[axel-lauer]

Done!

[valeriupredoi]

cheers @axel-lauer - for the sake of redundancy it'd be very good if you allowed @remi-kazeroni and at least another core member privileges for these types of operations (of your choice, could be @bouweandela , myself or @jvegasbsc ) :+1:

[axel-lauer]

@valeriupredoi good point. I propose to give @remi-kazeroni as "successor" of @mattiarighi admin rights right now but to leave it up to the new steering committee to decide on how admin rights are handled in the future.

[jvegreg]

Please, @ESMValGroup/esmvaltool-developmentteam go ahead and test the branch release_2.2.0 with your favourite recipes so we can get proper testing in the new releases. Please, bear in min that you may need ESMValGroup/ESMValTool#1874 to make them work with Iris 3

[valeriupredoi]

@valeriupredoi good point. I propose to give @remi-kazeroni as "successor" of @mattiarighi admin rights right now but to leave it up to the new steering committee to decide on how admin rights are handled in the future.

cheers @axel-lauer - it all sounds very reasonable to me :beer:

[valeriupredoi]

Please, @ESMValGroup/esmvaltool-developmentteam go ahead and test the branch release_2.2.0 with your favourite recipes so we can get proper testing in the new releases. Please, bear in min that you may need ESMValGroup/ESMValTool#1874 to make them work with Iris 3

@ESMValGroup/esmvaltool-developmentteam to be able to test, the easiest way is to install the iris3 branch from https://github.com/ESMValGroup/ESMValTool/pull/1874 in development mode then install the release_2.2.0 ESMValCore branch, note that the iris3 branch from Tool has a modified environment, so it's best you created an environment for it:

## first install esmvalcore
cd ESMValCore
git fetch -v
git checkout release_2.2.0
conda env create -n test_core_release -f environment.yml
conda activate test_core_release
pip install -e .[develop]

## then install esmvaltool in development
cd ESMValTool
git fetch -v
git checkout iris3
## comment out the "- esmvalcore>=2.1.0,<2.2" line in environment.yml
conda env update -n test_core_release -f environment.yml
conda activate test_core_release
pip install -e .[develop]

[schlunma]

I'm getting the following error now when installing the tool in development mode with the current master of ESMValCore and iris3 of ESMValTool (I commented out the esmvalcore dependency in environment.yml) during the pip install command on ESMValTool:

ERROR: Command errored out with exit status 1:
   command: /mnt/lustre02/work/bd0854/b309141/miniconda3/envs/esm/bin/python3.9 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-kw0tzxha/llvmlite_05e52ad606c7441389048af7d71f0f12/setup.py'"'"'; __file__='"'"'/tmp/pip-install-kw0tzxha/llvmlite_05e52ad606c7441389048af7d71f0f12/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-ei_mrgps                      
       cwd: /tmp/pip-install-kw0tzxha/llvmlite_05e52ad606c7441389048af7d71f0f12/                                                                                                                                                                                              
  Complete output (26 lines):                                                                                                                                                                                                                                                 
  running bdist_wheel                                                                                                                                                                                                                                                         
  /mnt/lustre02/work/bd0854/b309141/miniconda3/envs/esm/bin/python3.9 /tmp/pip-install-kw0tzxha/llvmlite_05e52ad606c7441389048af7d71f0f12/ffi/build.py                                                                                                                        
  LLVM version... Traceback (most recent call last):                                                                                                                                                                                                                          
    File "/tmp/pip-install-kw0tzxha/llvmlite_05e52ad606c7441389048af7d71f0f12/ffi/build.py", line 105, in main_posix                                                                                                                                                          
      out = subprocess.check_output([llvm_config, '--version'])                                                                                                                                                                                                               
    File "/mnt/lustre02/work/bd0854/b309141/miniconda3/envs/esm/lib/python3.9/subprocess.py", line 420, in check_output                                                                                                                                                       
      return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,                                                                                                                                                                                                        
    File "/mnt/lustre02/work/bd0854/b309141/miniconda3/envs/esm/lib/python3.9/subprocess.py", line 501, in run                                                                                                                                                                
      with Popen(*popenargs, **kwargs) as process:                                                                                                                                                                                                                            
    File "/mnt/lustre02/work/bd0854/b309141/miniconda3/envs/esm/lib/python3.9/subprocess.py", line 947, in __init__                                                                                                                                                           
      self._execute_child(args, executable, preexec_fn, close_fds,                                                                                                                                                                                                            
    File "/mnt/lustre02/work/bd0854/b309141/miniconda3/envs/esm/lib/python3.9/subprocess.py", line 1819, in _execute_child                                                                                                                                                    
      raise child_exception_type(errno_num, err_msg, err_filename)                                                                                                                                                                                                            
  FileNotFoundError: [Errno 2] No such file or directory: 'llvm-config'                                                                                                                                                                                                       

  During handling of the above exception, another exception occurred:                                                                                                                                                                                                         

  Traceback (most recent call last):                                                                                                                                                                                                                                          
    File "/tmp/pip-install-kw0tzxha/llvmlite_05e52ad606c7441389048af7d71f0f12/ffi/build.py", line 191, in <module>                                                                                                                                                            
      main()                                                                                                                                                                                                                                                                  
    File "/tmp/pip-install-kw0tzxha/llvmlite_05e52ad606c7441389048af7d71f0f12/ffi/build.py", line 181, in main                                                                                                                                                                
      main_posix('linux', '.so')                                                                                                                                                                                                                                              
    File "/tmp/pip-install-kw0tzxha/llvmlite_05e52ad606c7441389048af7d71f0f12/ffi/build.py", line 107, in main_posix                                                                                                                                                          
      raise RuntimeError("%s failed executing, please point LLVM_CONFIG "                                                                                                                                                                                                     
  RuntimeError: llvm-config failed executing, please point LLVM_CONFIG to the path for llvm-config                                                                                                                                                                            
  error: command '/mnt/lustre02/work/bd0854/b309141/miniconda3/envs/esm/bin/python3.9' failed with exit code 1                                                                                                                                                                
  ----------------------------------------                                                                                                                                                                                                                                    
  ERROR: Failed building wheel for llvmlite

ERROR: Command errored out with exit status 1:
     command: /mnt/lustre02/work/bd0854/b309141/miniconda3/envs/esm/bin/python3.9 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-kw0tzxha/llvmlite_05e52ad606c7441389048af7d71f0f12/setup.py'"'"'; __file__='"'"'/tmp/pip-install-kw0tzxha/llvmlite_05e52ad606c7441389048af7d71f0f12/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-6sfce9aq/install-record.txt --single-version-externally-managed --compile --install-headers /mnt/lustre02/work/bd0854/b309141/miniconda3/envs/esm/include/python3.9/llvmlite
         cwd: /tmp/pip-install-kw0tzxha/llvmlite_05e52ad606c7441389048af7d71f0f12/
    Complete output (29 lines):
    running install
    running build
    got version from file /tmp/pip-install-kw0tzxha/llvmlite_05e52ad606c7441389048af7d71f0f12/llvmlite/_version.py {'version': '0.34.0', 'full': 'c5889c9e98c6b19d5d85ebdd982d64a03931f8e2'}
    running build_ext
    /mnt/lustre02/work/bd0854/b309141/miniconda3/envs/esm/bin/python3.9 /tmp/pip-install-kw0tzxha/llvmlite_05e52ad606c7441389048af7d71f0f12/ffi/build.py
    LLVM version... Traceback (most recent call last):
      File "/tmp/pip-install-kw0tzxha/llvmlite_05e52ad606c7441389048af7d71f0f12/ffi/build.py", line 105, in main_posix
        out = subprocess.check_output([llvm_config, '--version'])
      File "/mnt/lustre02/work/bd0854/b309141/miniconda3/envs/esm/lib/python3.9/subprocess.py", line 420, in check_output
        return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
      File "/mnt/lustre02/work/bd0854/b309141/miniconda3/envs/esm/lib/python3.9/subprocess.py", line 501, in run
        with Popen(*popenargs, **kwargs) as process:
      File "/mnt/lustre02/work/bd0854/b309141/miniconda3/envs/esm/lib/python3.9/subprocess.py", line 947, in __init__
        self._execute_child(args, executable, preexec_fn, close_fds,
      File "/mnt/lustre02/work/bd0854/b309141/miniconda3/envs/esm/lib/python3.9/subprocess.py", line 1819, in _execute_child
        raise child_exception_type(errno_num, err_msg, err_filename)
    FileNotFoundError: [Errno 2] No such file or directory: 'llvm-config'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/tmp/pip-install-kw0tzxha/llvmlite_05e52ad606c7441389048af7d71f0f12/ffi/build.py", line 191, in <module>
        main()
      File "/tmp/pip-install-kw0tzxha/llvmlite_05e52ad606c7441389048af7d71f0f12/ffi/build.py", line 181, in main
        main_posix('linux', '.so')
      File "/tmp/pip-install-kw0tzxha/llvmlite_05e52ad606c7441389048af7d71f0f12/ffi/build.py", line 107, in main_posix
        raise RuntimeError("%s failed executing, please point LLVM_CONFIG "
    RuntimeError: llvm-config failed executing, please point LLVM_CONFIG to the path for llvm-config
    error: command '/mnt/lustre02/work/bd0854/b309141/miniconda3/envs/esm/bin/python3.9' failed with exit code 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /mnt/lustre02/work/bd0854/b309141/miniconda3/envs/esm/bin/python3.9 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-kw0tzxha/llvmlite_05e52ad606c7441389048af7d71f0f12/setup.py'"'"'; __file__='"'"'/tmp/pip-install-kw0tzxha/llvmlite_05e52ad606c7441389048af7d71f0f12/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-6sfce9aq/install-record.txt --single-version-externally-managed --compile --install-headers /mnt/lustre02/work/bd0854/b309141/miniconda3/envs/esm/include/python3.9/llvmlite Check the logs for full command output.

[valeriupredoi]

this is a common hurdle when installing numba, but we've got rid of it, all goes fine on my machine, can you pls tell me which wheel is building while this is being executed? (add a few more lines of stdout before the error kicks in)

[schlunma]

I'm repeating the installation with the release_2.2.0 branch, hang tight

[valeriupredoi]

OK I have replicated your issue - and that's because you should always read screen stdout :gun: - installing master of ESMValCore results in:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
esmvaltool 2.1.1 requires esmvalcore<2.2,>=2.1.0, but you have esmvalcore 2.2.0 which is incompatible.

and it rightly does, then going further to install ESMValTool results in your error

[schlunma]

It doesn't say it for me, I swear :grimacing: Here's the full log:

install.log#

Just got the same error with the release branch. Apparently it tries to install numba for me...

I did the following:

conda env remove -n esm
cd ~/ESMValCore
git checkout release_2.2.0
git pull
conda env create -n esm -f environment.yml
cd ~/ESMValTool
git checkout iris3
git pull
# change environment.yml file
conda env update -n esm -f environment.yml
source activate esm
pip install -e ".[develop]"

The last command results in the error...

[valeriupredoi]

you missing a couple steps: conda activate esm && pip install -e .[develop] while still inside ESMValCore

[schlunma]

No, the core needs to be installed after the tool, otherwise you end up with the pip package and not the development version (at least it did this some months ago). I'm installing the tool like this for months now, it always worked

[schlunma]

(This is also the order given by the documentation: https://docs.esmvaltool.org/en/latest/quickstart/installation.html)

[valeriupredoi]

why do you think you're commenting out the environment line before updating it , Sherlock :grin: Trust me, am an engineer :hammer:

[valeriupredoi]

(This is also the order given by the documentation: https://docs.esmvaltool.org/en/latest/quickstart/installation.html)

this is an unorthodox manner, the documentation has its limitations but they're not exposed due to the workflow following a logical mainstream path. This type of installation as we are doing here is highly dubious if a user would have to do it on a regular basis

[valeriupredoi]

also note that if you installed it after, in pip devel mode, with esmvalcore initially being downloaded and installed from conda, you will run in a hefty dependency issue with iris3 not bein installed, and you'd have to update the env from core again, but that'll get you in a catch22 because esmvaltool will not be compatible with some deps - just follow my instructions and we'll all be happy :grin:

[schlunma]

Okay, now I get your point (and why you commented out esmvalcore in the environment.yml). I agree that this is okay for testing purposes now, but shouldn't we have two branches (in the ideal case the two master branches) where the regular development installation works just fine? Or is this only possible after the release?

[valeriupredoi]

Okay, now I get your point (and why you commented out esmvalcore in the environment.yml). I agree that this is okay for testing purposes now, but shouldn't we have two branches (in the ideal case the two master branches) where the regular development installation works just fine? Or is this only possible after the release?

yep, the current core-master is at odds with tool-master coz of the upping of version number in core (something that @jvegasbsc pointed out well and I agree with). For now consider the core-master to be release_2.2.0, Javi has fixed that in https://github.com/ESMValGroup/ESMValCore/pull/984 and I fully thumbs-upped it. Also release_2.2.0 is the one that needs testing because we are still merging stuff in master (the core one) that we may or may not cherry-pick for the release, may not most prob)

[schlunma]

Okay, apparently the installation with my (traditional) method works just fine with the release_2.2.0 and iris3 branch if I do not change anything of the branches (so no commenting out stuff of the environment). Seems like the release branch still counts as version 2.1 (I guess since #977 has only been merged into master), so there's no version conflict.

You should be careful with the tips you're giving, @valeriupredoi :joy:

[valeriupredoi]

tips-shnips, of course it works, coz I realized I had just committed the pinning for iris3 this morning in the iris3 branch of esmvaltool :grin: I am quite surprised, if not alarmed, why it works though (without commenting out the package requirement for esmvalcore) since that should throw an environment insolubility with the conda esmvalcore not being built for iris3 - you sure you not picking up a cached installation?

[schlunma]

But the conda esmvalcore environment IS built for iris3, no? https://github.com/ESMValGroup/ESMValCore/blob/fd7022639684c57b5949a714df7629bbcf4b75c8/environment.yml#L10

[valeriupredoi]

no, not the conda package one isn't, but I think conda is being dumb since you already have esmvalcore installed from develop, and iris3 is already in the environment due to that installation, it will not try and re-install it from esmvalgroup even if the requirement is there (if left uncommented). OK problem solved: blame conda :man_facepalming: Also - that will work only if you've installed esmvalcore in advance, so my tips are correct - next time you make fun of my tips I'll leave you to the elements :rofl:

[remi-kazeroni]

I have just tested all the recipes/examples (except the julia one). Everything works fine except the following 2 ones:

• [ ] recipe_preprocessor_derive_test.yml File "/mnt/lustre01/pf/b/b309192/ESMValCore/esmvalcore/_recipe.py", line 1039, in _initialize_variables raw_datasets + raw_variable.pop('additional_datasets', [])) TypeError: can only concatenate list (not "NoneType") to list

• [ ] recipe_preprocessor_test.yml ERROR fiona._env:215 Unable to open EPSG support file gcs.csv. Try setting the GDAL_DATA environment variable to point to the directory containing EPSG csv files.

Could someone double check these please?

[bouweandela]

@jvegasbsc @valeriupredoi Could one of you check the problems reported by @remi-kazeroni please?

[valeriupredoi]

@bouweandela @remi-kazeroni on me to-do list, will do it in a bit :+1:

[jvegreg]

I will check this evening, it's being a crazy day

[valeriupredoi]

@remi-kazeroni the second issue is not impacting us - fiona imports well and runs fine, it's this one here https://github.com/Toblerity/Fiona/issues/897 and if you scroll to the bottom of it you'll see there is some dud file that is missing in the new GDAL, that fiona thinks it should be reading - which is not the case, the $GDAL_DATA variable is set OK in the environment and is getting picked up well. A reinstallation of fiona in a new env solves the problem.

The second issue (first, in your post) is a bit more tricksy - I get it too, but in a slightly different shape (I am running with --skp_nonexistent), in my case the tool is trying to run concatenation on 0-len cubes list, will investigate a bit more...

[valeriupredoi]

OK I think I know what the problem with the 0-len cube list is - --skip_nonexistent has changed its functionality - with previous release, running that recipe, and missing data, but running with the skip flag I still get data missing error esmvalcore._recipe_checks.RecipeError: Did not find any input data for task cmip5/sm_derive_input_mrsos which is fair since that should be raised if input data for a variable is missing, but the current implementation skips over this and tries to assemble 0-len data @bouweandela @jvegasbsc who of you two fellas changed the flag's (mis)behaviour coz I sure didn't :grin:

[valeriupredoi]

OK - I dug deeper in the fiona issue: it seems that the pypi fiona (that we install currently) has been built with gdal=3 earlier that still had that dud file, installing it from conda-forge completely solves the issue, I am not going to open an issue on the fiona github page since last time I did that I had to haggle and sneer at them to actually pay correct attention to it and fix it, we'll just have to install it from conda-forge. There was a note in the setup file I left saying that we'd eventually have to install fiona from conda since they kinda gave up uploading wheels on pypi, so might as well we do it now

[valeriupredoi]

OK I think I know what the problem with the 0-len cube list is - --skip_nonexistent has changed its functionality - with previous release, running that recipe, and missing data, but running with the skip flag I still get data missing error esmvalcore._recipe_checks.RecipeError: Did not find any input data for task cmip5/sm_derive_input_mrsos which is fair since that should be raised if input data for a variable is missing, but the current implementation skips over this and tries to assemble 0-len data @bouweandela @jvegasbsc who of you two fellas changed the flag's (mis)behaviour coz I sure didn't grin

@remi-kazeroni it'd be best if you opened an issue about this supplying the command you ran (if you used or not --skip_nonexistent) and the debug log file please. It's clear that it is an issue with how the derivation is handling the missing data, and this has been an issue with the previous release too, since I've run it with the previous one and get a different yet still non-descriptive error, but at least it tells me there is no data:

running with esmvaltool=2.1.1 and esmvalcore=2.1.0 (off the JASMIN central installation):

esmvaltool run examples/recipe_preprocessor_derive_test.yml --skip_nonexistent

returns:

2021-02-05 16:57:18,966 UTC [13676] INFO    Skipping: no data found for {'short_name': 'mrsos', 'project': 'CMIP5', 'mip': 'Lmon', 'exp': 'historical', 'start_year': 2004, 'end_year': 2005, 'derive': True, 'force_derivation': False, 'variable_group': 'sm_derive_input_mrsos', 'diagnostic': 'cmip5', 'preprocessor': 'default', 'dataset': 'GFDL-CM3', 'ensemble': 'r1i1p1', 'recipe_dataset_index': 0, 'institute': ['NOAA-GFDL'], 'alias': 'GFDL-CM3', 'original_short_name': 'mrsos', 'standard_name': 'moisture_content_of_soil_layer', 'long_name': 'Moisture in Upper Portion of Soil Column', 'units': 'kg m-2', 'modeling_realm': ['land'], 'frequency': 'mon', 'filename': '/home/users/valeriu/esmvaltool_var_test/esmvaltool_output/recipe_preprocessor_derive_test_20210205_165653/preproc/cmip5/sm_derive_input_mrsos/CMIP5_GFDL-CM3_Lmon_historical_r1i1p1_mrsos_2004-2005.nc'}
2021-02-05 16:57:19,306 UTC [13676] INFO    Maximum memory used (estimate): 0.2 GB
2021-02-05 16:57:19,307 UTC [13676] INFO    Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur.
2021-02-05 16:57:19,314 UTC [13676] ERROR   Program terminated abnormally, see stack trace below for more information
Traceback (most recent call last):
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/_main.py", line 430, in run
    fire.Fire(ESMValTool())
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/fire/core.py", line 138, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/fire/core.py", line 463, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/fire/core.py", line 672, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/_main.py", line 407, in run
    process_recipe(recipe_file=recipe, config_user=cfg)
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/_main.py", line 94, in process_recipe
    recipe = read_recipe_file(recipe_file, config_user)
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/_recipe.py", line 71, in read_recipe_file
    return Recipe(raw_recipe,
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/_recipe.py", line 951, in __init__
    self.tasks = self.initialize_tasks() if initialize_tasks else None
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/_recipe.py", line 1299, in initialize_tasks
    task = _get_preprocessor_task(
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/_recipe.py", line 908, in _get_preprocessor_task
    task = _get_single_preprocessor_task(
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/esmvalcore/_recipe.py", line 784, in _get_single_preprocessor_task
    raise RecipeError(
esmvalcore._recipe_checks.RecipeError: Did not find any input data for task cmip5/sm_derive_input_mrsos
2021-02-05 16:57:19,363 UTC [13676] INFO    If you suspect this is a bug or need help, please open an issue on https://github.com/ESMValGroup/ESMValTool/issues and attach the run/recipe_*.yml and run/main_log_debug.txt files from the output directory.

This is much better than what we have now, since the current codebase is actually attempting derivation (hence going and trying to concatenate). In any case, it's just a matter of understanding where the stop point has moved, and outputting a reasonable error message, but I'm not gonna try fix that on a Friday evening and it's really a corner case that would fail anyway, so no reason to halt the release :+1: :beer:

[jvegreg]

I will look into it. I probably messed it up when changing the error messages for the missing data

[axel-lauer]

I am also having trouble getting an environment installed. I followed @valeriupredoi 's instructions (except for adding quotes to the pip commands) but no success. pip install for ESMValTool gives me this error

`Collecting cartopy>=0.18 Using cached Cartopy-0.18.0.tar.gz (14.4 MB) ERROR: Command errored out with exit status 1: command: /mnt/lustre02/work/bd0854/b380103/miniconda3/bin/python3.8 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-z4ovgl_o/cartopy_db74203ccd7a42cca3b436e98fcd3e2b/setup.py'"'"'; file='"'"'/tmp/pip-install-z4ovgl_o/cartopy_db74203ccd7a42cca3b436e98fcd3e2b/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-v_jd7n0j cwd: /tmp/pip-install-z4ovgl_o/cartopy_db74203ccd7a42cca3b436e98fcd3e2b/ Complete output (12 lines): Traceback (most recent call last): File "/tmp/pip-install-z4ovgl_o/cartopy_db74203ccd7a42cca3b436e98fcd3e2b/setup.py", line 43, in import numpy as np ModuleNotFoundError: No module named 'numpy' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "", line 1, in File "/tmp/pip-install-z4ovgl_o/cartopy_db74203ccd7a42cca3b436e98fcd3e2b/setup.py", line 45, in raise ImportError('NumPy 1.10+ is required to install cartopy.') ImportError: NumPy 1.10+ is required to install cartopy.

WARNING: Discarding https://files.pythonhosted.org/packages/46/c1/04e50c9986842f00f7db0e7a65caa896840050d7328f74e5b7437aa01179/Cartopy-0.18.0.tar.gz#sha256=7ffa317e8f8011e0d965a3ef1179e57a049f77019867ed677d49dcc5c0744434 (from https://pypi.org/simple/cartopy/) (requires-python:>=2.7, !=3.0., !=3.1., !=3.2., !=3.3., !=3.4.*). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output. ERROR: Could not find a version that satisfies the requirement cartopy>=0.18 (from esmvaltool[develop]) ERROR: No matching distribution found for cartopy>=0.18`

I also tried various other ways to create a new environment but all attempts have failed. I think by now I tried everything that comes to my mind. Why is it so complicated to create a new environment for testing v2.2?

[valeriupredoi]

@axel-lauer the trace tells me you are using python=3.8 which is not good since if things worked with the installation of esmvalcore (as per my instructions) your environment should have been created with and updated for python=3.9 so that means something went awry before that. Could you pls post the steps you took here and give me some output of python -V && conda list esmvalcore && conda list iris && conda env list && conda list numpy after you've installed esmvalcore and then after you've updated the env for esmvaltool, just before you run pip install -e .[develop] that errored for you? :beer:

[valeriupredoi]

also pls make sure you are on release_2.2.0 branch :+1:

[schlunma]

I just repeated @valeriupredoi's instructions

conda update -n base -c defaults conda
conda --version  # gives conda 4.9.2

cd ~/ESMValCore
git checkout release_2.2.0
git pull
conda env create -n esmtest -f environment.yml
source activate esmtest
pip install -e ".[develop]"
conda deactivate

cd ~/ESMValTool
git checkout iris3
git pull
# comment out the "- esmvalcore>=2.1.0,<2.2" line in environment.yml
conda env update -n esmtest -f environment.yml
source activate esmtest 
pip install -e ".[develop]"

and got a successful installation of the tool (I tried esmvaltool version and an example recipe). For me, this picks up Python 3.9.1. Here is the full environment:

env.txt

One question though: Does this ensure that the usual installation in development mode works after the release of ESMValCore the and the merge of the iris3 branch into the ESMValTool master?

[valeriupredoi]

cheers Manu! yes, the usual development installation procedure will work fine after the release provided that we make sure the esmvalcore package version is pinned to >=2.2.0 and mpich is also pinned from https://github.com/ESMValGroup/ESMValTool/pull/2007

[valeriupredoi]

also note that the conda deactivate command after installing core should not be needed (but it's not unneeded) :beer:

[remi-kazeroni]

@remi-kazeroni it'd be best if you opened an issue about this supplying the command you ran (if you used or not --skip_nonexistent) and the debug log file please.

Will do so. I also had the --skip_nonexistent option.

[axel-lauer]

I just tried to follows @schlunma's steps using a brand new anaconda installation. Everything seems to work fine until the second "pip install":

(esmtest) mistralpp3{b380103} /pf/b/b380103/workesm/ESMValTool> pip install -e ".[develop]" Obtaining file:///mnt/lustre02/work/bd0854/b380103/ESMValTool Installing build dependencies ... done Getting requirements to build wheel ... done Installing backend dependencies ... done Preparing wheel metadata ... done Requirement already satisfied: jinja2 in /mnt/lustre02/work/bd0854/b380103/anaconda3/lib/python3.8/site-packages (from ESMValTool==2.1.1) (2.11.2) Collecting xgboost Using cached xgboost-1.3.3-py3-none-manylinux2010_x86_64.whl (157.5 MB) Collecting cdo Using cached cdo-1.5.4.tar.gz (23 kB) Processing /mnt/lustre01/pf/b/b380103/.cache/pip/wheels/72/04/4a/aaab8318bdf227fba7a48e25696f01aed93060ec17b3cf8efe/seawater-3.3.4-py3-none-any.whl Collecting netCDF4 Using cached netCDF4-1.5.5.1-cp38-cp38-manylinux2010_x86_64.whl (4.7 MB) ERROR: Could not find a version that satisfies the requirement ESMPy (from ESMValTool==2.1.1) (from versions: none) ERROR: No matching distribution found for ESMPy (from ESMValTool==2.1.1)

I am close to giving up. This is a nightmare.

[bouweandela]

ESMPy is provided by the ESMF package from conda-forge, that should be installed when you use the environment.yml file from ESMValCore to create or update the environment (you can check that it is listed there).

[valeriupredoi]

@axel-lauer for the sake of sanity, I have repeated the exact same steps @schlunma posted above and the ones you followed too; I have no issues installing ESMValTool (Manu - side note, having looked closer at the commands - I recommend not mixing source activate with conda deactivate - in fact, source activate is obsolete and only conda activate/deactivate should be used); Axel, I still see your environment gets built with Python 3.8 - and if that's not a problem on its own since we still support 3.8, it's a clue that something is done a little different from what we others are doing, not sure what, but the only thing different I can think of is that you mentioned the use of Anaconda rather than Miniconda (what I am using and what Manu is also using and what the tests are using). For the sake of sanity again, could you try the same steps but starting from a fresh Miniconda install this time around? No biggie if you don't have time, we'll push the release anyway, maybe you'd have time to test after, when it's gonna be much easier once the package's out on conda :+1:

[axel-lauer]

@valeriupredoi just ran into the same problem with a fresh miniconda installation. Tomorrow I'll try switching from tcsh to bash. Maybe that does the trick...

[valeriupredoi]

@axel-lauer it might be that your conda init tcsh is not working properly, as I've heard from other places, try the solution in this post, the last comment since the starred one is very old. The problem I think is happening in your case is that even if at command prompt it shows that you are in the environment esmtest it is actually not sourced properly and is missing a heapload of paths to dep packages, if you did a conda env list and the star is at the esmtest env that is not enough, look for stuff like where the python executable is called fromwhich python and do a conda list too after you've installed esmvalcore to make sure the deps newly installed are in the actual virt env and not in base

[valeriupredoi]

BTW it'd be nice if you could document the fixes you apply and the steps you take to make it work - don't worry about this particular case with conda update etc, just try and install esmvalcore in pure development mode then esmvaltool, and make sure all deps are there where they should be - this way we can document the procedure and include it in the documentation for the next oddball that's using tcsh or csh :grin:

[schlunma]

BTW the main reason I'm using source activate instead of conda activate is that I had trouble with conda init on mistral at some point (I don't remember the details, though; it was long time ago)...

[bouweandela]

@axel-lauer Sometimes conda says it activates the environment, but it actually doesn't. You could try running which python to see if it actually activated the environment it says it activated. If that happens, you just need to deactivate and activate again.