zklaus
commented
3 years ago Thanks, @valeriupredoi, this is good information. I wonder though, what the associated action is? What is the criterion for closing this issue?
Closed valeriupredoi closed 2 years ago
zklaus
commented
3 years ago Thanks, @valeriupredoi, this is good information. I wonder though, what the associated action is? What is the criterion for closing this issue?
valeriupredoi
commented
3 years ago Hang on, still testing, it's not an issue perse, it's a place where I put test results, @bouweandela and meself decided to put mamba through the paces before we start using it proper
valeriupredoi
commented
3 years ago installing esmvaltool from conda gets us pretty much the same differences in envs (note the mamba install is MUCH faster)
(conda_install_conda) valeriu@valeriu-PORTEGE-Z30-C:~/ESMValTool$ diff ~/esmvaltool_fromconda_mamba.txt ~/esmvaltool_fromconda_conda.txt
1c1
< name: conda_install_mamba
---
> name: conda_install_conda
16d15
< - binutils=2.36.1=hdd6e379_1
29d27
< - c-compiler=1.1.1=h516909a_0
49d46
< - compilers=1.1.1=0
52d48
< - cxx-compiler=1.1.1=hc9558a2_0
58c54
< - decorator=4.4.2=py_0
---
> - decorator=5.0.9=pyhd8ed1ab_0
70c66
< - esmvaltool-r=2.2.0=0
---
> - esmvaltool-r=2.1.0=0
82d77
< - fortran-compiler=1.1.1=he991be0_0
87c82
< - gcc_impl_linux-64=7.5.0=habd7529_19
---
> - gcc_impl_linux-64=7.5.0=hda68d29_13
105c100
< - gxx_impl_linux-64=7.5.0=hd0bb8aa_19
---
> - gxx_impl_linux-64=7.5.0=h64c220c_13
149d143
< - libgcc-devel_linux-64=7.5.0=hda03d7c_19
169d162
< - libstdcxx-devel_linux-64=7.5.0=hb016644_19
197c190
< - networkx=2.6.1=pyhd8ed1ab_1
---
> - networkx=2.5=py_0
227c220
< - pyke=1.1.1=py39hf3d152e_1003
---
> - pyke=1.1.1=pyhd8ed1ab_1004
242a236,237
> - r-curl=4.3.2=r40hcfec24a_0
> - r-udunits2=0.13=r40hcfec24a_1004
252c247
< - seaborn=0.11.1=ha770c72_0
---
> - seaborn=0.11.1=hd8ed1ab_1
314c309
< prefix: /home/valeriu/miniconda3/envs/conda_install_mamba
---
> prefix: /home/valeriu/miniconda3/envs/conda_install_condaOne thing I noticed is that when conda (and mamba for the matter) install esmvaltool from conda, the version displayed via conda list esmvaltool is 2.1:
(conda_install_conda) valeriu@valeriu-PORTEGE-Z30-C:~/ESMValTool$ conda list esmvaltool
# packages in environment at /home/valeriu/miniconda3/envs/conda_install_conda:
#
# Name Version Build Channel
esmvaltool 2.1.0 0 esmvalgroup
esmvaltool-julia 2.2.0 0 esmvalgroup
esmvaltool-ncl 2.2.0 0 esmvalgroup
esmvaltool-python 2.2.0 pyhd8ed1ab_2 conda-forge
esmvaltool-r 2.1.0 0 esmvalgroupwhich is dodgy (the actual package installed is 2.2.0 indeed). I looked all over https://anaconda.org/esmvalgroup/esmvaltool and can't see any reference for 2.1 -> @zklaus do you know where the conda displayed version comes from?
valeriupredoi
commented
3 years ago nevermind my last comment, have no idea why that 2.1 creeped up in me env, did a fresh install and got the right versioning:
# packages in environment at /home/valeriu/miniconda3-fresh/envs/esmvaltool-conda:
#
# Name Version Build Channel
esmvaltool 2.2.0 0 esmvalgroup
esmvaltool-julia 2.2.0 0 esmvalgroup
esmvaltool-ncl 2.2.0 0 esmvalgroup
esmvaltool-python 2.2.0 py_0 esmvalgroup
esmvaltool-r 2.2.0 0 esmvalgroup~but note that that's off esmvalgroup where we should pick it up from conda-forge~ no, am a dumber, it's not made it yet to conda-forge :man_facepalming:
bouweandela
commented
3 years ago I tried installing ESMValTool on Mistral with mamba instead of conda and ran into problems. When I install with conda things work just fine.
When I try to run a recipe in the mamba created environment, the tool crashes with:
ERROR:esmvalcore._main:Program terminated abnormally, see stack trace below for more information:
Traceback (most recent call last):
File "/pf/b/b381141/conda/envs/esmvaltool/lib/python3.9/site-packages/esmvalcore/_main.py", line 433, in run
fire.Fire(ESMValTool())
File "/pf/b/b381141/conda/envs/esmvaltool/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/pf/b/b381141/conda/envs/esmvaltool/lib/python3.9/site-packages/fire/core.py", line 466, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/pf/b/b381141/conda/envs/esmvaltool/lib/python3.9/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/pf/b/b381141/conda/envs/esmvaltool/lib/python3.9/site-packages/esmvalcore/_main.py", line 357, in run
from ._recipe import TASKSEP
File "/pf/b/b381141/conda/envs/esmvaltool/lib/python3.9/site-packages/esmvalcore/_recipe.py", line 14, in <module>
from . import _recipe_checks as check
File "/pf/b/b381141/conda/envs/esmvaltool/lib/python3.9/site-packages/esmvalcore/_recipe_checks.py", line 12, in <module>
from .preprocessor import TIME_PREPROCESSORS, PreprocessingTask
File "/pf/b/b381141/conda/envs/esmvaltool/lib/python3.9/site-packages/esmvalcore/preprocessor/__init__.py", line 9, in <module>
from .._provenance import TrackedFile
File "/pf/b/b381141/conda/envs/esmvaltool/lib/python3.9/site-packages/esmvalcore/_provenance.py", line 7, in <module>
from PIL import Image
File "/pf/b/b381141/conda/envs/esmvaltool/lib/python3.9/site-packages/PIL/Image.py", line 114, in <module>
from . import _imaging as core
ImportError: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /pf/b/b381141/conda/envs/esmvaltool/lib/python3.9/site-packages/PIL/_imaging.cpython-39-x86_64-linux-gnu.so)Versions: mamba 0.7.3 conda 4.10.3
It looks like the problem is that mamba installs the pillow package from the defaults channel instead of from conda-forge. The mamba installed pillow package tries to load a system library (/lib64/libc.so.6) that was probably present on the system that it was built on and subsequently fails because it is not present on the system that I'm trying to run the tool on.
bouweandela
commented
3 years ago It looks like all required packages are available on conda-forge, but we would need some way to tell mamba to not install anything from the defaults channel.
valeriupredoi
commented
3 years ago right on Bouwe, but @zklaus reported the same issue with what I believe to be a non-mamba installation (and it might be something Mistral-specific), can you maybe try the same install process but with conda (not rely on an older installation and say that works fine)
valeriupredoi
commented
3 years ago It looks like all required packages are available on conda-forge, but we would need some way to tell mamba to not install anything from the defaults channel.
we should be able to do that by removing the defaults channel from the condarc or mambarc (if there's one?) conf file
bouweandela
commented
3 years ago can you maybe try the same install process but with conda (not rely on an older installation and say that works fine)
That is exactly what I did above. It is a development installation, so from the environment.yml. suprisingly conda seems to pick up the older esmvalcore (v2.3.0 instead of v2.3.1). Maybe that needs to be pinned?
valeriupredoi
commented
3 years ago ah fair do's, OK - I'll test too, can you pls post the exact commands you used to be able to reproduce the behavior? About 2.3.0 - there is absolutely no reason why conda should pick that up unless you using Python 3.6 - I checked that with the GA tests https://github.com/ESMValGroup/ESMValTool/actions/runs/1073166287 - we'll pin anyway, pinning is always a good idea even if only for restricting conda going on a long walk
bouweandela
commented
3 years ago ah fair do's, OK - I'll test too, can you pls post the exact commands you used to be able to reproduce the behavior?
Clone repo:
git clone https://github.com/esmvalgroup/esmvaltool
cd esmvaltoolMamba install:
mamba env create -n mamba -f environment.yml
conda activate mamba
pip install -e .Conda install:
conda create -n conda -c conda-forge python
conda env update -n conda -f environment.yml
conda activate conda
pip install -e .Three comments:
First, the glibc thing really is a bit mistral specific in that it has an ancient version of OS and with it glibc, see this DKRZ blog post.
Second, perhaps you should update your mamba, see this blog post, particularly the comment on channel priorities. Mamba 0.7 really is maybe a bit old.
Third, for mamba you use mamba env create, but for conda conda create + conda env update. Is there a reason for this? Does it make any difference if you also use mamba create + mamba env update?
These things notwithstanding, do we want to take everything only from conda-forge or allow for mixing?
valeriupredoi
commented
3 years ago hey @bouweandela I tried exactly your workflow (on my laptop, I'll try on JASMIN too but I don't expect any change) and all goes fone for me:
(mamba) valeriu@valeriu-PORTEGE-Z30-C:~/ESMValCore$ conda list pillow
# packages in environment at /home/valeriu/miniconda3/envs/mamba:
#
# Name Version Build Channel
pillow 8.2.0 py39hf95b381_1 conda-forge
(mamba) valeriu@valeriu-PORTEGE-Z30-C:~/ESMValCore$ mamba --version
mamba 0.15.0
conda 4.10.3and esmvaltool run works fine. Can you please update your mamba and try again? As @zklaus mentioned, your mamba's old, buddy :grin:
valeriupredoi
commented
3 years ago OK I visited my beloved Jasmin and there I had a mamba=0.13 so I used it instead of upgrading it, and lo and behold pillow got installed from defaults, bit even so, the installation and run of esmvaltool went fine (I ran examples/recipe_python.yml no problemo); then I updated mamba to 0.15.2 and repeated the process, this time around just fftw gets installed from defaults/main, pillow is nice and conda-forgy (8.2.0, same build as on my laptop) and again, recipe ran fine. So that's a defo Mistral issue, it'd be worth updating your mamba to 0.15+ @bouweandela and test again, my bets are you'll run in the same glibc issue even with pillow from conda-forge, and that will give mamba a clean sheet and the Mistral folk need to fix that issue :+1:
egalytska
commented
3 years ago Dear all, Unfortunatelly I ran into the same issue in Mistral. Trying to install ESMValTool (from scratch) with conda would give me weird message "/lib64/libc.so.6: version `GLIBC_2.15' not found (required by /.../python)". But mamba helped. Below I post in details what I encountered.
(base) b380971@mlogin100% conda --version
conda 4.10.3
(base) b380971@mlogin100% python --version
Python 3.8.8To install ESMValTool I follow instructions from here.
base) b380971@mlogin100% git clone https://github.com/ESMValGroup/ESMValTool
(base) b380971@mlogin100% cd ESMValTool
(base) b380971@mlogin100% conda create --name esmvaltool 'python=3.9'
(base) b380971@mlogin100% conda env update --name esmvaltool --file environment.ymlAfter some good 50 mins the environment was created.
base) b380971@mlogin100% conda activate esmvaltool
(esmvaltool) b380971@mlogin100% pip install --editable '.[develop]'
/work/bd0854/b380971/anaconda3/envs/esmvaltool/bin/python: /lib64/libc.so.6: version `GLIBC_2.15' not found (required by /work/bd0854/b380971/anaconda3/envs/esmvaltool/bin/python)
/work/bd0854/b380971/anaconda3/envs/esmvaltool/bin/python: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /work/bd0854/b380971/anaconda3/envs/esmvaltool/bin/python)
/work/bd0854/b380971/anaconda3/envs/esmvaltool/bin/python: /lib64/libc.so.6: version `GLIBC_2.17' not found (required by /work/bd0854/b380971/anaconda3/envs/esmvaltool/bin/python)!Note: The same error message I got when I was trying to install Julia for EsmValTool.
So I tired to use mamba.
(base) b380971@mlogin100% conda install mamba -n base -c conda-forge
Btw, while solving environments I am getting quite often the following message: Solving environment: failed with initial frozen solve. Retrying with flexible solve.
(base) b380971@mlogin100% mamba --version
mamba 0.7.3
conda 4.10.30_0 I thought that apriori one of the recent versions of mamba would be installed. I did not update mamba because any conda install- or update- actions take for me for ever. So the following I did with mamba:
(base) b380971@mlogin100% mamba env create -n esmvaltool2 -f environment.yml
(base) b380971@mlogin100% conda activate esmvaltool2
(esmvaltool2) b380971@mlogin100% pip install -e .And finally after couple days of suffer I saw "....Successfully installed ESMValTool".
Double check
(esmvaltool2) b380971@mlogin100% esmvaltool --help
Printed a help message.
ESMValCore was also successfully installed.
Thanks to @schlunma who pointed to this issue and @valeriupredoi who introduced mamba. Does it mean that from now on we won't be able to install ESMValTool on Mistral with conda anymore?
valeriupredoi
commented
3 years ago @egalytska cheers for the heads up!
Btw, while solving environments I am getting quite often the following message: Solving environment: failed with initial frozen solve. Retrying with flexible solve.
no problem there, conda tries different strategies to solve the environment (all slow, mind you) - the first one is a more restrictive and most of the times fails, then it tries another one that's a bit more inclusive, and that usually leads to the env being solved
About mamba being old, you can try installing it with conda install -c conda-forge mamba==0.15 - that should work fine
Thanks for reporting this, it is now clear that the issue is with the library not being correctly installed on Minstral, this needs to be fixed upstream by the site admins, maybe @schlunma and/or @remi-kazeroni could push this a bit and see if the post that @zklaus linked in https://github.com/ESMValGroup/ESMValCore/issues/1227#issuecomment-888962499 can become a ticket or something :beer:
zklaus
commented
3 years ago Well, GLIBC is so fundamental, that updating it means moving to the next generation of the operating system. This will simply not happen on mistral, which has been installed in 2015 and will be replaced by the new BullSequana based system, which in turn is supposed to become available mid-2021, i.e. now(ish). See the press release.
However, conda-forge is, for now, commited to CentOS6 as a base (see the last announcement of 2020). Unfortunately, the sudden and largely unexpected discontinuation of CentOS has create a lot of uncertainty around that, and I imagine also for DKRZ brought up the question on what to base the next HPC.
Bottom line: If we restrict ourselves to conda-forge, we'll be fine for now, but there simply is a bit of uncertainty on where things are going for the next couple of months.
remi-kazeroni
commented
3 years ago Thanks for reporting this, it is now clear that the issue is with the library not being correctly installed on Minstral, this needs to be fixed upstream by the site admins, maybe @schlunma and/or @remi-kazeroni could push this a bit and see if the post that @zklaus linked in #1227 (comment) can become a ticket or something 🍺
We have already encountered that issue with different software used in our group on Mistral. As @zklaus pointed out, there is no chance to get such libraries updated a few months before Mistral will be decommisioned (see schedule). We have tried to ask unsuccessfully in the past.
Thanks to @schlunma who pointed to this issue and @valeriupredoi who introduced mamba. Does it mean that from now on we won't be able to install ESMValTool on Mistral with conda anymore?
I have also run into the issue described by @egalytska when installing the latest version of the ESMValTool. The only way I found to solve the environment with conda was to create an environment based on the Core environment.yml file and update this environment with the Tool environment.yml:
cd ESMValCore
conda env create -f environment.yml -n esmvaltool
cd ../ESMValTool
conda env update -f environment.yml -n esmvaltoolIs that advisable @valeriupredoi? The whole process took about 10 minutes and the ESMValTool seems to run fine for me.
valeriupredoi
commented
3 years ago Is that advisable @valeriupredoi?
perfectly fine, in fact encouraged, since updating a base env takes much shorter than creating the full blown env from scratch :+1:
bouweandela
commented
3 years ago Is that advisable @valeriupredoi? The whole process took about 10 minutes and the ESMValTool seems to run fine for me.
@remi-kazeroni @valeriupredoi This may result in problems with solving the environment. At the moment, the recommended installation procedure is to first create an environment with just Python in it and then update that with the environment.yml file, see https://docs.esmvaltool.org/en/latest/quickstart/installation.html#install-from-source. That may change of course, as we would like switch to recommending mamba instead of conda, provided that it works.
These things notwithstanding, do we want to take everything only from conda-forge or allow for mixing?
@zklaus Everything from conda forge would be best, because mixing may lead to problems. Also, some of us may need to buy an anaconda commercial subscription if we use packages from anaconda.
bouweandela
commented
3 years ago First, the glibc thing really is a bit mistral specific in that it has an ancient version of OS and with it glibc, see this DKRZ blog post.
The point is that packages installed from conda should not try to use system libraries, but the files that are shipped with the conda compilers. However, we are installing the compilers from conda forge, so if a package then comes from another channel, it will not have been compiled against these libraries and fall back on system libraries.
For me, the problem with the glibc library on mistral does not happen on the login node, where it is available, but the compute nodes. Anyway, will try again with an updated mamba when I have time.
valeriupredoi
commented
3 years ago @remi-kazeroni @valeriupredoi This may result in problems with solving the environment. At the moment, the recommended installation procedure is to first create an environment with just Python in it and then update that with the environment.yml file
not when you want to install ESMValCore locally with pip in development mode; the alternative is to install Core as a dependency of Tool (from conda-forge) at Tool install point then go to your esmvalcore dir and run a pip install -e .[develop] - that will supersede the conda package install from site-packages but this is a dirty installation procedure and may result in mixed paths for libraries. I thought Remi wanted to have both Core and Tool installed in dev mode, in which case creating an env for Core, installing Core locally in it then updating that env for Tool and installing Tool in it is perfectly fine :+1:
valeriupredoi
commented
3 years ago @remi-kazeroni @valeriupredoi This may result in problems with solving the environment. At the moment, the recommended installation procedure is to first create an environment with just Python in it and then update that with the environment.yml file
not when you want to install ESMValCore locally with
pipin development mode; the alternative is to install Core as a dependency of Tool (from conda-forge) at Tool install point then go to your esmvalcore dir and run apip install -e .[develop]- that will supersede the conda package install fromsite-packagesbut this is a dirty installation procedure and may result in mixed paths for libraries. I thought Remi wanted to have both Core and Tool installed in dev mode, in which case creating an env for Core, installing Core locally in it then updating that env for Tool and installing Tool in it is perfectly fine +1
also note that I experienced issues with mamba doing this (create Core env + install Core + update env for Tool + install Tool); but conda has no issues doing that. In fact many CI tests use conda update as a main call, rarely they conda create any bulky environments
bouweandela
commented
3 years ago I thought Remi wanted to have both Core and Tool installed in dev mode, in which case creating an env for Core, installing Core locally in it then updating that env for Tool and installing Tool in it is perfectly fine
Except when the installing core creates an environment that conda is unable to update for the tool installation, as has happened so often in the past with conda. If it works it's fine, but it's not a procedure I would recommend, because chances of ending up with an environment that cannot be solved are pretty high.
valeriupredoi
commented
3 years ago I thought Remi wanted to have both Core and Tool installed in dev mode, in which case creating an env for Core, installing Core locally in it then updating that env for Tool and installing Tool in it is perfectly fine
Except when the installing core creates an environment that conda is unable to update for the tool installation, as has happened so often in the past with conda. If it works it's fine, but it's not a procedure I would recommend, because chances of ending up with an environment that cannot be solved are pretty high.
indeed, this was endemic in the past, but more recently all is good with this eg see Remi's attempt too
remi-kazeroni
commented
3 years ago I thought Remi wanted to have both Core and Tool installed in dev mode, in which case creating an env for Core, installing Core locally in it then updating that env for Tool and installing Tool in it is perfectly fine 👍
Yes this is exactly what I did: installing the Tool and then the Core in dev mode once the conda environment was solved (created from the Core and updated with the Tool).
valeriupredoi
commented
3 years ago For me, the problem with the glibc library on mistral does not happen on the login node, where it is available, but the compute nodes. Anyway, will try again with an updated mamba when I have time.
Sounds good, Bouwe, I bet you my boots it'll be okay. Here's the question though - I thought the base Python in any conda env (including base) needs the system GLIBC no? As @zklaus mentions, newer conda's struggle on CentOS's (see https://github.com/conda/conda/issues/10657 ) and 1. the system GLIBC gets too old and is not recommended to upgrade it and 2. Anaconda does supply a GLIBC package but that seems to be older than Joe Biden https://anaconda.org/rmg/glibc - anyway it looks like setting conda env variables may allow overriding system libs https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-virtual.html
remi-kazeroni
commented
3 years ago @zklaus Everything from conda forge would be best, because mixing may lead to problems. Also, some of us may need to buy an anaconda commercial subscription if we use packages from anaconda.
I also wanted to bring that up. It may need a separate issue but the link above says:
How do you define commercial usage?
We clarified our definition of commercial usage in our Terms of Service in an update on Sept. 30, 2020. The new language states that use by individual hobbyists, students, universities, non-profit organizations, or businesses with less than 200 employees is allowed, and all other usage is considered commercial and thus requires a business relationship with Anaconda.
How do I know if I’m not in compliance?
You’re not in compliance with our Terms of Service if you are using Anaconda’s repositories in a for-profit or government organization with over 200 employees outside of a business relationship with Anaconda. You are also not in compliance if you are using mirroring software to create copies of the Anaconda repositories without a license.
Is this (going to be) a problem for us? One viable alternative could be miniforge. The central esmvaltool module set up by DKRZ on Mistral is based on that but I was not involved in the process so I don't know much.
zklaus
commented
3 years ago That conda/conda issue really is a red herring. That guy is asking how to run the current, 2021 conda on a RHEL 5.4 server. That was released in 2011. And he also want's to do this on the beta. Everything from today will struggle there.
First, the glibc thing really is a bit mistral specific in that it has an ancient version of OS and with it glibc, see this DKRZ blog post.
The point is that packages installed from conda should not try to use system libraries, but the files that are shipped with the conda compilers. However, we are installing the compilers from conda forge, so if a package then comes from another channel, it will not have been compiled against these libraries and fall back on system libraries.
In general you are right, but glibc really is special. From their website
The GNU C Library project provides the core libraries for the GNU system and GNU/Linux systems, as well as many other systems that use Linux as the kernel. These libraries provide critical APIs including ISO C11, POSIX.1-2008, BSD, OS-specific APIs and more. These APIs include such foundational facilities as open, read, write, malloc, printf, getaddrinfo, dlopen, pthread_create, crypt, login, exit and more.
In other words, it is tied so intimately to the kernel that you cannot just install a new version, but rather using glibc gives you another layer of separation from the kernel that allows you to work with a more stable api. That's why it is the only library that is among the virtual packages that @valeriupredoi already mentioned.
@remi-kazeroni, I think we should discuss the commercial usage thing somewhere else. Either its own issue, or a github discussion.
valeriupredoi
commented
3 years ago oh an OS from 2011 - worse than me and my Ubuntu 14.04 :rofl:
egalytska
commented
3 years ago Dear @valeriupredoi, thanks for explaining.
About mamba being old, you can try installing it with
conda install -c conda-forge mamba==0.15- that should work fine
Btw. It did not work =/ Mamba stayed the same 0.7.3
And here comes the worst part: seems that I am still not over GLIBC issue. Despite ESMValTool and Core were installed successfully, I receive similar error while trying to run a recipe.
(esmvaltool) b380971@mlogin104% esmvaltool run examples/recipe_python.yml
ERROR:esmvalcore._main:Program terminated abnormally, see stack trace below for more information:
Traceback (most recent call last):
File "/mnt/lustre02/work/bd0854/b380971/ESMValCore/esmvalcore/_main.py", line 432, in run
fire.Fire(ESMValTool())
File "/work/bd0854/b380971/anaconda3/envs/esmvaltool/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/work/bd0854/b380971/anaconda3/envs/esmvaltool/lib/python3.9/site-packages/fire/core.py", line 466, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/work/bd0854/b380971/anaconda3/envs/esmvaltool/lib/python3.9/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/mnt/lustre02/work/bd0854/b380971/ESMValCore/esmvalcore/_main.py", line 357, in run
from ._recipe import TASKSEP
File "/mnt/lustre02/work/bd0854/b380971/ESMValCore/esmvalcore/_recipe.py", line 14, in <module>
from . import _recipe_checks as check
File "/mnt/lustre02/work/bd0854/b380971/ESMValCore/esmvalcore/_recipe_checks.py", line 12, in <module>
from .preprocessor import TIME_PREPROCESSORS, PreprocessingTask
File "/mnt/lustre02/work/bd0854/b380971/ESMValCore/esmvalcore/preprocessor/__init__.py", line 9, in <module>
from .._provenance import TrackedFile
File "/mnt/lustre02/work/bd0854/b380971/ESMValCore/esmvalcore/_provenance.py", line 7, in <module>
from PIL import Image
File "/work/bd0854/b380971/anaconda3/envs/esmvaltool/lib/python3.9/site-packages/PIL/Image.py", line 114, in <module>
from . import _imaging as core
ImportError: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /work/bd0854/b380971/anaconda3/envs/esmvaltool/lib/python3.9/site-packages/PIL/_imaging.cpython-39-x86_64-linux-gnu.so)I tried the @remi-kazeroni method:
The only way I found to solve the environment with conda was to create an environment based on the Core
environment.ymlfile and update this environment with the Toolenvironment.yml:cd ESMValCore conda env create -f environment.yml -n esmvaltool cd ../ESMValTool conda env update -f environment.yml -n esmvaltool
and all works. Finally. Hopefully.
remi-kazeroni
commented
3 years ago About mamba being old, you can try installing it with
conda install -c conda-forge mamba==0.15- that should work fineBtw. It did not work =/ Mamba stayed the same 0.7.3
Using conda install -c conda-forge mamba==0.15 from the base environment on a Mistral login node worked fine for me:
$ mamba --version
mamba 0.15.0
conda 4.10.3@egalytska which version of conda are you using? It can be updated with: conda update --name base conda
I tried the @remi-kazeroni method:
The only way I found to solve the environment with conda was to create an environment based on the Core
environment.ymlfile and update this environment with the Toolenvironment.yml:cd ESMValCore conda env create -f environment.yml -n esmvaltool cd ../ESMValTool conda env update -f environment.yml -n esmvaltooland all works. Finally. Hopefully.
Good to hear. To summarize, here's how I installed the ESMValTool v2.3 on Mistral (both Core and Tool in developer mode):
cd dir_to_ESMValCore
conda env create -f environment.yml -n esmvaltool
cd dir_to_ESMValTool
conda env update -f environment.yml -n esmvaltool
conda activate esmvaltool
pip install -e .[develop] # install the Tool in dev mode
cd dir_to_ESMValCore
pip install -e .[develop] # install the Core in dev modeThis installation finished in about 10 minutes.
To install the ESMValTool only, I couldn't find a way using conda (job killed after several hours) even following the instructions. In that case I used mamba:
cd dir_to_ESMValTool
mamba env create -n tool -file environment.yml
conda activate tool
pip install -e .[develop] # install the Tool in dev modeThis installation finished in about 20 minutes. Note that pre-solving the environment withconda create --name esmvaltool 'python=3.9' led to a GLIBC issue.
bouweandela
commented
3 years ago @remi-kazeroni Could you try with conda create --name esmvaltool 'python=3.9' -c conda-forge instead of conda create --name esmvaltool 'python=3.9'?
remi-kazeroni
commented
3 years ago @remi-kazeroni Could you try with
conda create --name esmvaltool 'python=3.9' -c conda-forgeinstead ofconda create --name esmvaltool 'python=3.9'?
Good point, I could solve the environment in 40 minutes on Mistral by doing:
conda create --name esmvaltool 'python=3.9' -c conda-forge
conda env update --name esmvaltool --file environment.ymlWould it make sense to change our instructions to add conda-forge
egalytska
commented
3 years ago This worked for me as well.
@remi-kazeroni Could you try with
conda create --name esmvaltool 'python=3.9' -c conda-forgeinstead ofconda create --name esmvaltool 'python=3.9'?@egalytska which version of conda are you using? It can be updated with:
conda update --name base conda@remi-kazeroni, it is 4.10.3. But for now I leave mamba as is.
mamba is a fast alternative to
conda; oftentimes env solving is done factors faster with mamba than with conda, and the results should be identical. I am running a systematic comparisonmambavscondahere so we start using it instead of Madame Conda.Operating system
Conda/Mamba versions
mambabuild:ESMValCore Tests
Environment comparison mamba-conda
conda env create -n esmvaltool -f environment.ymlvsmamba env create -n esmvaltool -f environment.ymlenv export after installing
esmvalcoremamba-env.txt conda-env.txt
Env differences:
-> nothing major at all!
Test session with
mambaandcondaMamba:
pytest)Conda:
pytest)mamba install from conda-forge vs conda install from conda-forge
Directive:
environment: esmvalcore_mamba.txt time to install: <2min
Directive:
environment: esmvalcore_conda.txt time to install: about 6min
Environment differences:
-> again, minor differences it seems mamba picks up a couple packages slightly older than conda, that's about it.
mamba build vs conda build
Do we even want to bother testing this one since
mamba buildusesconda buildunder the hood anyway?Conclusion:
mambais as safe as Bank of England for esmvalcoreESMValTool Tests
mamba vs conda env creation
mamba env create -n esmvaltool -f environment.ymlduration: about 6min to full env creation (includes downloading most packages) env: esmvaltool_mamba.txt test environment:tests pass: OK help+version: OK does it actually run: yes
conda create -n esmvaltool-conda "python=3.9.6" && conda env update -n esmvaltool-conda(note that I freeze the Python version to have the same setup as the mamba env that has Python=3.9.6) duration: max memory: about 1.5G duration: about 3min (nothing was downloaded, conda recycled the already downloaded packages for the mamba env) env: esmvaltool_conda.txtEnvironment differences:
Nothing to worry, it looks like mamba installs an extra
libgcc-devel_linux-64=7.5.0=hda03d7c_19that doesn't show up in the conda env, not sure why, but it's good there are no missing libs.