Closed valeriupredoi closed 4 years ago
Maybe you could open an issue about this with conda? https://github.com/conda/conda/issues
It looks like similar issues have been reported before, e.g. https://github.com/conda/conda/issues/7787, https://github.com/conda/conda/issues/8650
My installation finished:
$ \time -v conda create -n esmvaltool_deps -c esmvalgroup -c conda-forge esmvaltool
resulted in
Command being timed: "conda create -n esmvaltool_deps -c esmvalgroup -c conda-forge esmvaltool"
User time (seconds): 4681.48
System time (seconds): 80.85
Percent of CPU this job got: 109%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:12:41
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 7334184
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 1168
Minor (reclaiming a frame) page faults: 23161833
Voluntary context switches: 294555
Involuntary context switches: 455039
Swaps: 0
File system inputs: 1657920
File system outputs: 4395656
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
i.e. about 1 hour to solve the environment and around 7GB or RAM.
tiptop, mate, tiptop :rofl:
I can't see your comment about the env solving quickly w/o ncl
- did you remove it?
Sorry, yes, because it was confusing, e.g.
conda create -n test -c esmvalgroup -c conda-forge esmvaltool-r imagemagick ncl
takes about a minute to resolve and 1.2 GB or RAM and installs the exact same packages as conda create -n test -c esmvalgroup -c conda-forge esmvaltool
, which takes an hour and needs 7GB RAM.
what does -r
do?
OK @bouweandela some progress twoards understanding this:
conda env create ...
is actually very quick and only taking 1.5G which is normal; why is conda create ...
so bulky is puzzling;esmvalcore
fails:
(base) valeriu@valeriu:~/ESMValTool$ \time conda install -c esmvalgroup esmvalcore
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: \
Found conflicts! Looking for incompatible packages.
This can take several minutes. Press CTRL-C to abort.
failed
UnsatisfiableError: The following specifications were found to be incompatible with each other:
Output in format: Requested package -> Available versions
Command exited with non-zero status 1 31.94user 0.84system 0:39.12elapsed 83%CPU (0avgtext+0avgdata 212256maxresident)k 104568inputs+8outputs (48major+73946minor)pagefaults 0swaps
-> it would be very useful if you could create an up to date esmvalcore package so we can see if the issues with installing `esmvaltool` stem from conflicts with `esmvalcore` deps - usually conda decides it can still solve an env that is otherwise insolvable (if fewer deps were in the list) and goes on forever with no avail
UPDATE: \time conda install -c esmvalgroup -c conda-forge esmvalcore
works (duh!) we need to change the instructions at https://anaconda.org/ESMValGroup/esmvalcore
stats: 85.24user 8.15system 3:10.18elapsed 49%CPU (0avgtext+0avgdata 504072maxresident)k 721960inputs+2167944outputs (94major+418051minor)pagefaults 0swaps
OK we really need a new esmvalcore
package, am hitting this (proj
-related issue that we solved a month or so ago):
(base) valeriu@valeriu:~/ESMValTool$ esmvaltool --help
ERROR 1: PROJ: proj_create_from_database: Open of /home/valeriu/miniconda3/share/proj failed
PROJ: proj_create_from_database: Cannot find proj.db
Traceback (most recent call last):
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/ESMF/interface/loadESMF.py", line 128, in <module>
mode=ct.RTLD_GLOBAL)
File "/home/valeriu/miniconda3/lib/python3.7/ctypes/__init__.py", line 364, in __init__
self._handle = _dlopen(self._name, mode)
OSError: dlopen: cannot load any more object with static TLS
Traceback (most recent call last):
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/ESMF/interface/loadESMF.py", line 128, in <module>
mode=ct.RTLD_GLOBAL)
File "/home/valeriu/miniconda3/lib/python3.7/ctypes/__init__.py", line 364, in __init__
self._handle = _dlopen(self._name, mode)
OSError: dlopen: cannot load any more object with static TLS
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/valeriu/miniconda3/bin/esmvaltool", line 11, in <module>
load_entry_point('ESMValCore==2.0.0b9', 'console_scripts', 'esmvaltool')()
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 490, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2854, in load_entry_point
return ep.load()
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2445, in load
return self.resolve()
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2451, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/_main.py", line 42, in <module>
from ._recipe import TASKSEP, read_recipe_file
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/_recipe.py", line 14, in <module>
from . import _recipe_checks as check
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/_recipe_checks.py", line 12, in <module>
from .preprocessor import PreprocessingTask, TIME_PREPROCESSORS
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/preprocessor/__init__.py", line 13, in <module>
from ._derive import derive
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/preprocessor/_derive/__init__.py", line 32, in <module>
ALL_DERIVED_VARIABLES = _get_all_derived_variables()
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/preprocessor/_derive/__init__.py", line 27, in _get_all_derived_variables
f'esmvalcore.preprocessor._derive.{short_name}')
File "/home/valeriu/miniconda3/lib/python3.7/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/preprocessor/_derive/sispeed.py", line 6, in <module>
from .._regrid import regrid
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/preprocessor/_regrid.py", line 16, in <module>
from ._regrid_esmpy import ESMF_REGRID_METHODS
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/preprocessor/_regrid_esmpy.py", line 4, in <module>
import ESMF
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/ESMF/__init__.py", line 70, in <module>
from ESMF.api.esmpymanager import *
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/ESMF/api/esmpymanager.py", line 11, in <module>
from ESMF.interface.cbindings import *
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/ESMF/interface/cbindings.py", line 13, in <module>
from ESMF.interface.loadESMF import _ESMF
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/ESMF/interface/loadESMF.py", line 131, in <module>
raise ImportError('The ESMF shared library did not load properly.')
ImportError: The ESMF shared library did not load properly.
@bouweandela can you pls produce a new esmvalcore
package, ship it to esmvalgroup
, change the install instructions on the page and let me know when you've done that, please mate? I will test the installation from conda right away that's done, I suspect our conda woes will be fixed by doing this :beer:
what does -r do?
It installs only the esmvaltool-r
subpackage, as described in our installation instructions.
we need to change the instructions at https://anaconda.org/ESMValGroup/esmvalcore
I've done that a few times already, but these instructions get overwritten with the default every time I upload a new package, so I'm not going to do that anymore. I reckon people should just use the installation instructions from our documentation.
OK we really need a new esmvalcore package, am hitting this (proj-related issue that we solved a month or so ago):
Could you please share all commands you ran to create this environment, so I can try to reproduce your issue?
conda install -c conda-forge -c esmvalgroup esmvalcore
esmvaltool --help
am trying it on Jasmin as well at the mo, will report back :beer:
Could you also share the commands you used to create the environment that you're installing into?
no other commands man, installing in the conda (base) - ideally when a user does conda install ...
they should not create any environment prior to :beer:
I just downloaded conda (version 4.8.3) and ran
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
# chose ~/conda-test as an installation directory
. conda-test/etc/profile.d/conda.sh
conda activate base
conda install -c conda-forge -c esmvalgroup esmvalcore
esmvaltool --help
and everything worked perfectly. What version of conda are you using? Can you provide a link for downloading from the archive: https://repo.anaconda.com/miniconda/
Or maybe a link to a docker container that runs the conda version that has the problem?
here are the traceback steps Monsieur Le Bouwe:
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
conda install -c conda-forge -c esmvalgroup esmvalcore
note that I accepted conda to prepend the installation and activate base
right away, equivalent to the snippet in .bashrc
:
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/home/valeriu/miniconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "/home/valeriu/miniconda3/etc/profile.d/conda.sh" ]; then
. "/home/valeriu/miniconda3/etc/profile.d/conda.sh"
else
export PATH="/home/valeriu/miniconda3/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda initialize <<<
and for the big finale:
(base) valeriu@valeriu:~$ esmvaltool --help
Traceback (most recent call last):
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/ESMF/interface/loadESMF.py", line 128, in <module>
mode=ct.RTLD_GLOBAL)
File "/home/valeriu/miniconda3/lib/python3.7/ctypes/__init__.py", line 364, in __init__
self._handle = _dlopen(self._name, mode)
OSError: dlopen: cannot load any more object with static TLS
Traceback (most recent call last):
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/ESMF/interface/loadESMF.py", line 128, in <module>
mode=ct.RTLD_GLOBAL)
File "/home/valeriu/miniconda3/lib/python3.7/ctypes/__init__.py", line 364, in __init__
self._handle = _dlopen(self._name, mode)
OSError: dlopen: cannot load any more object with static TLS
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/valeriu/miniconda3/bin/esmvaltool", line 11, in <module>
load_entry_point('ESMValCore==2.0.0b9', 'console_scripts', 'esmvaltool')()
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 490, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2854, in load_entry_point
return ep.load()
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2445, in load
return self.resolve()
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2451, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/_main.py", line 42, in <module>
from ._recipe import TASKSEP, read_recipe_file
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/_recipe.py", line 14, in <module>
from . import _recipe_checks as check
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/_recipe_checks.py", line 12, in <module>
from .preprocessor import PreprocessingTask, TIME_PREPROCESSORS
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/preprocessor/__init__.py", line 13, in <module>
from ._derive import derive
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/preprocessor/_derive/__init__.py", line 32, in <module>
ALL_DERIVED_VARIABLES = _get_all_derived_variables()
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/preprocessor/_derive/__init__.py", line 27, in _get_all_derived_variables
f'esmvalcore.preprocessor._derive.{short_name}')
File "/home/valeriu/miniconda3/lib/python3.7/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/preprocessor/_derive/sispeed.py", line 6, in <module>
from .._regrid import regrid
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/preprocessor/_regrid.py", line 16, in <module>
from ._regrid_esmpy import ESMF_REGRID_METHODS
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/esmvalcore/preprocessor/_regrid_esmpy.py", line 4, in <module>
import ESMF
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/ESMF/__init__.py", line 70, in <module>
from ESMF.api.esmpymanager import *
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/ESMF/api/esmpymanager.py", line 11, in <module>
from ESMF.interface.cbindings import *
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/ESMF/interface/cbindings.py", line 13, in <module>
from ESMF.interface.loadESMF import _ESMF
File "/home/valeriu/miniconda3/lib/python3.7/site-packages/ESMF/interface/loadESMF.py", line 131, in <module>
raise ImportError('The ESMF shared library did not load properly.')
ImportError: The ESMF shared library did not load properly.
a few pointers:
(base) valeriu@valeriu:~$ which conda
/home/valeriu/miniconda3/bin/conda
(base) valeriu@valeriu:~$ conda --version
conda 4.8.3
(base) valeriu@valeriu:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.6 LTS
Release: 14.04
Codename: trusty
(base) valeriu@valeriu:~$ conda list esmvalcore
# packages in environment at /home/valeriu/miniconda3:
#
# Name Version Build Channel
esmvalcore 2.0.0b9 py_0 esmvalgroup
moar pointers:
(base) valeriu@valeriu:~$ conda list python
# packages in environment at /home/valeriu/miniconda3:
#
# Name Version Build Channel
antlr-python-runtime 4.7.2 py37_1001 conda-forge
msgpack-python 1.0.0 py37h99015e2_1 conda-forge
python 3.7.6 cpython_h8356626_6 conda-forge
python-dateutil 2.8.1 py_0 conda-forge
python-stratify 0.1.1 py37h03ebfcd_1002 conda-forge
python_abi 3.7 1_cp37m conda-forge
(base) valeriu@valeriu:~$ conda list esmf
# packages in environment at /home/valeriu/miniconda3:
#
# Name Version Build Channel
esmf 8.0.1 nompi_hbeb3ca6_0 conda-forge
Note that on Jasmin the install didn't even finish, it was going in circles trying to solve the environment, on my laptop it was very snappy (as pointed above)
Massive hint: when you uninstall esmvalcore and install esmf
(since it'll be removed with the uninstall of esmvalcore) that one is compiled correctly against mpi with mpich:
(base) valeriu@valeriu:~$ conda list esmf
# packages in environment at /home/valeriu/miniconda3:
#
# Name Version Build Channel
esmf 8.0.1 mpi_mpich_h213fab7_100 conda-forge
right after this if you re-install esmvalcore, then it'll work nicely. You need to check that the requirement for esmf
in the conda recipe for esmvalcore is set correctly for mpi with mpich :beer:
strange, I have repeated the install steps on Jasmin and all works fine there out the box with
(base) [valeriu@jasmin-sci2 ~]$ conda list esmf
# packages in environment at /home/users/valeriu/miniconda3:
#
# Name Version Build Channel
esmf 8.0.1 nompi_hbeb3ca6_0 conda-forge
I still reckon it's best to run a few installation tests on different OS's after the package gets built. It might be that my OS is old (Ubuntu 14 is ancient, 6 years old) but even so it shouldn't matter
jasmin's OS:
(base) [valeriu@jasmin-sci2 ~]$ lsb_release -a
LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Distributor ID: RedHatEnterpriseServer
Description: Red Hat Enterprise Linux Server release 6.10 (Santiago)
Release: 6.10
Codename: Santiago
It also works fine on my laptop and on CircleCI, so I wouldn't be suprised if it's somehow related to your OS or user profile.
what are you running on your laptop - if it's a Debian-based OS then am OK to call this solved (well, the installation of esmvalcore
that is, for now, I am still struggling with esmvaltool
) :beer:
I'm running ubuntu 20.04. Trying to reproduce your issue now with ubuntu 14.04.
Yes, I managed to reproduce it!
excellent! did you build a container with that ye olde OS? Got good news - I managed to install esmvaltool
on Jasmin straight from conda (well, it's installing now, but the env solving lasted about a smoke's time :grin: ) - this was after I've installed esmvalcore
from conda so it may be that either this is a workaround or conda fixed their troubles (I'll uninstall them both and try installing esmvaltool
from scratch)
I'm running ubuntu 20.04. Trying to reproduce your issue now with ubuntu 14.04.
way to go updated OS, man! What's that one called, Flying Dutchman? :grin:
Indeed I did
docker pull ubuntu:14.04
docker run -it ubuntu:14.04
and that reproduced the problem. I also found lots of other people having the same issue: https://github.com/pytorch/pytorch/issues/2575#issuecomment-523667875
It looks like the publicly available Ubuntu 14.04 is no longer maintained: https://ubuntu.com/about/release-cycle, so maybe now would be a good time to upgrade?
aw mate, 14.04 hasn't been supported since April last year :grin: but you know, updating an OS that is 11 version increments old is probably suicidal. Nah, it's fine, we should log this somewhere and if some oldtimer like me complains in a future issue we will know what he problem is
new roadblock; while installing esmvaltool
from conda (conda install -c conda-forge -c esmvalgroup esmvaltool
) it complains about Julia:
ERROR conda.core.link:_execute(700): An error occurred while installing package 'esmvalgroup::esmvaltool-julia-2.0.0b4-0'.
Rolling back transaction: \
full output:
Rolling back transaction: -
done
LinkError: post-link script failed for package esmvalgroup::esmvaltool-julia-2.0.0b4-0
location of failed script: /home/users/valeriu/miniconda3/bin/.esmvaltool-julia-post-link.sh
==> script messages <==
Executable julia not found! Exiting...
==> script output <==
stdout:
stderr:
return code: 1
()
2005.28user 528.46system 42:49.91elapsed 98%CPU (0avgtext+0avgdata 1311888maxresident)k
2302216inputs+9258752outputs (1063major+34605267minor)pagefaults 0swaps
and esmvaltool --help
dont work :cry:
that's it for me for today, off to make dinner :pizza:
new roadblock; while installing esmvaltool from conda (conda install -c conda-forge -c esmvalgroup esmvaltool) it complains about Julia:
It would be really helpful if you could read the installation instructions. Quoting from there:
In order to install the Conda package, you will need both conda and Julia pre-installed, this is because Julia cannot be installed from conda.
you reckon I don't know that? :grin: I am not happy with the conda installation to fail because of that - we need to make it install and throw a big warning at the end saying this and that Julia but not to fail altogether; this is in line with that R diagnostic test that is still failing if R is not installed (bump: approve that PR, mate) :beer:
we need to make it install and throw a big warning at the end saying this and that Julia
Maybe you could open a pull request to improve the error message?
oh I will and it'll contain a few changes:
conda install -c conda-forge -c esmvalgroup esmvaltool
) to the installation without needing Julia? That would be absolutely awesome because it'll be very straightforwardOK I managed to catch this loose tiger that conda pretends to be: if you want to install esmvaltool from conda you will end up spending the good part of your day looking at that spindle spin and not solve the environment - if you install esmvalcore first, esmvaltool env will be solved in 3min and chuck at most 2G of mem. I will make a PR with updated instructions including how to install and softlink julia before installing :beer:
can we default the installation from conda (conda install -c conda-forge -c esmvalgroup esmvaltool) to the installation without needing Julia? That would be absolutely awesome because it'll be very straightforward
No, I think not, because then it will not work if people try to run a Julia diagnostic. For advanced users that know they do not want Julia diagnostics, we have the subpackages, e.g. conda install esmvaltool-python esmvaltool-r esmvaltool-ncl
will get you esmvaltool without Julia. Maybe we could make it possible to install the julia dependencies later, once we have the new command line interface. Or we could try to change the script so it installs Julia as well as the Julia dependencies, but I'm afraid that might be bit error prone.
OK let's keep it this way then, I am happier now that we are actually telling the user how to install Julia and make the executable available. I will try today or tomorrow see if we can get Julia back inside the conda environment, although doubt much has changes in the meantime :beer:
closing this since it works now, beware a reopen :grin:
ah the dreaded reopen :grin: As per the instructions installing from conda via simple conda invocation gets stuck in conflict zone:
conda install -c esmvalgroup -c conda-forge esmvalcore
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: \
Found conflicts! Looking for incompatible packages.
This can take several minutes. Press CTRL-C to abort.
on an anaconda install, in a base environment:
valeriu@valeriu-PORTEGE-Z30-C:~/ESMValCore$ conda -V
conda 4.8.3
valeriu@valeriu-PORTEGE-Z30-C:~/ESMValCore$ python -V
Python 3.8.3
I am adding the conda install test in #700 now and we can see this distributed to both Linux and OSX for different Python versions :beer:
too much fluff here, closing this and reopening a new, more precise one :beer:
results in successive failures of conda to solve the environment and piling up memory consumption in the process (on my puny 4GB RAM machine the conda python process reached 3GB and went straight to swap twice, forcing me to restart the whole machine twice). I am aware this is partly conda's fault for trying to be something that'll never be but it's also our fault for not optimizing the package; we need to fix this since people are trying to install it this way and not from source and giving up in the process (one case reported to me in the last 24h); I will look into this tomorrow but I ask @bouweandela and @bjoernbroetz to have a think too :beer: