search

pkgw / pwkit

Miscellaneous science/astronomy tools.
MIT License
26 stars 6 forks source link

some dependency trap? #8

Closed caseyjlaw closed 7 years ago

caseyjlaw commented 7 years ago

I just wanted to flag this for you, in case it is tripping others up. I just got myself out of some weird dependency trap. My installation uses casa-python (which I test with import casac), but after a recent update it failed to find the _atmosphere library. I've had this channel configuration for a while now, as suggested by you:

> conda config --get channels
--add channels 'defaults'   # lowest priority
--add channels 'http://conda.anaconda.org/pkgw-forge/channel/main'
--add channels 'conda-forge'   # highest priority

And here is where things got stuck after a conda update --all:

> conda list | egrep 'casa|pwkit|boost|icu'
boost                     1.63.0              np111py27_6    conda-forge
boost-cpp                 1.63.0                        2    conda-forge
casa-data                 r20170909                     0    pkgw-forge
casa-nrao-casacore        4.6.0                         3    pkgw-forge/channel/main
casa-python               4.6.0                    py27_4    pkgw-forge
casa-tools                4.6.0                         5    pkgw-forge
icu                       58.1                          1    conda-forge
pwkit                     0.8.8                    py27_0    pkgw-forge

I saw that casa-python 5.0.0 is available, but not getting picked up. After some work, I saw that qt4dbus was conflicting with more recent casa work by you. I removed it and then the casa-python 5.0.0 was picked up properly. Now I can import casac just fine after a conda update --all.

pkgw commented 7 years ago

Ah, OK, thanks. I think what's happening here is a consequence of the fact that the conda packaging system doesn't have great tools for migrating package names. In this case, I ended up superseding qt4dbus by a new package called qt4 but there's no way for users like to you be guided onto the right upgrade path.

Similarly, the casacore library is now best provided by casacore from conda-forge, not casa-nrao-casacore from pkgw-forge. As long as things are working, whatever, but at some point you should remove the pkgw-forge package and reinstall based on the conda-forge package.

pkgw commented 7 years ago

I will close this since this is a known condition, but feel free to add relevant comments — this issue could be useful for other users’ reference in the future.

caseyjlaw commented 7 years ago

Actually, something isn't quite right with that suggestion. I removed casa-nrao-casacore and then got the _atmosphere error. Then I put it back in and it works. Here's what the update looked like:

gygax$ conda install casa-nrao-casacore
Fetching package metadata .............
Solving package specifications: .

Package plan for installation in environment /users/claw/miniconda:

The following NEW packages will be INSTALLED:

    casa-nrao-casacore: 4.6.0-6           pkgw-forge 

The following packages will be DOWNGRADED:

    boost:              1.64.0-py27_4     conda-forge --> 1.63.0-py27_5     conda-forge
    boost-cpp:          1.64.0-1          conda-forge --> 1.63.0-2          conda-forge
    casa-python:        5.0.0-np113py27_0 pkgw-forge  --> 4.7.2-np113py27_5 pkgw-forge 
    casa-tools:         5.0.0-0           pkgw-forge  --> 4.7.2-1           pkgw-forge 
    casacore:           2.3.0-py27_1      conda-forge --> 2.2.0-1           conda-forge
    hdf5:               1.8.18-1          conda-forge --> 1.8.17-11         conda-forge

Lots of downgrading to accommodate the older casacore. What gives?

pkgw commented 7 years ago

The package is just called casacore, without the casa-nrao- prefix. Does that work better?

caseyjlaw commented 7 years ago

Ok, I needed to uninstall both kinds of casacore and then reinstall casacore to get it back to working.

caseyjlaw commented 5 years ago

No need to panic, but I wanted to document another instance of this error: ModuleNotFoundError: No module named '_atmosphere'.

It was resolved by moving to Python 3.7.1 and running conda update --all. Here is the detailed error message, in case others want to find this thread:

(deployment3) gygax$ python -c 'import casac'
Traceback (most recent call last):
  File "/home/cbe-master/realfast/anaconda/envs/deployment3/lib/python3.7/site-packages/__casac__/atmosphere.py", line 14, in swig_import_helper
    return importlib.import_module(mname)
  File "/home/cbe-master/realfast/anaconda/envs/deployment3/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 670, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 583, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1043, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: /home/cbe-master/realfast/anaconda/envs/deployment3/lib/python3.7/site-packages/__casac__/../../../././libsingledishfiller.so.54.0.0: undefined symbol: _ZN8casacore5ArrayINS_7QuantumIdEEE15postTakeStorageEv

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/cbe-master/realfast/anaconda/envs/deployment3/lib/python3.7/site-packages/casac.py", line 1, in <module>
    import __casac__
  File "/home/cbe-master/realfast/anaconda/envs/deployment3/lib/python3.7/site-packages/__casac__/__init__.py", line 54, in <module>
    from . import atmosphere
  File "/home/cbe-master/realfast/anaconda/envs/deployment3/lib/python3.7/site-packages/__casac__/atmosphere.py", line 17, in <module>
    _atmosphere = swig_import_helper()
  File "/home/cbe-master/realfast/anaconda/envs/deployment3/lib/python3.7/site-packages/__casac__/atmosphere.py", line 16, in swig_import_helper
    return importlib.import_module('_atmosphere')
  File "/home/cbe-master/realfast/anaconda/envs/deployment3/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named '_atmosphere'
pkgw commented 5 years ago

Thanks @caseyjlaw. Basically, the "No module name _atmosphere" error means that the backing shared libraries aren't all correctly matched up, which can happen for any number of reasons in the fast-evolving conda-forge system.

Perhaps casa-python should be patched to provide a better error message in this case, with a link to problem-solving suggestions ...

caseyjlaw commented 5 years ago

It may be worth while, if you think there are users out there. I have run into this in different contexts, so others may be too. Does this error trigger any ideas for you? As it turns out, I do need to solve this. I thought I could upgrade to python 3.7 to avoid it, but I cannot.

caseyjlaw commented 5 years ago

I solved this by blasting my environment and installing from scratch. So nothing needed on my end.

caseyjlaw commented 5 years ago

Sorry for all this back and forth, but it is not solved by starting from scratch.

(deployment3) gygax$ python -c 'import casac'
Traceback (most recent call last):
  File "/home/cbe-master/realfast/anaconda/envs/deployment3/lib/python3.6/site-packages/__casac__/atmosphere.py", line 14, in swig_import_helper
    return importlib.import_module(mname)
  File "/home/cbe-master/realfast/anaconda/envs/deployment3/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 658, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 571, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 922, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: /home/cbe-master/realfast/anaconda/envs/deployment3/lib/python3.6/site-packages/__casac__/../../.././libsingledishfiller.so.54.0.0: undefined symbol: _ZN8casacore6StringC1ERNSt7__cxx1119basic_ostringstreamIcSt11char_traitsIcESaIcEEE

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/cbe-master/realfast/anaconda/envs/deployment3/lib/python3.6/site-packages/casac.py", line 1, in <module>
    import __casac__
  File "/home/cbe-master/realfast/anaconda/envs/deployment3/lib/python3.6/site-packages/__casac__/__init__.py", line 54, in <module>
    from . import atmosphere
  File "/home/cbe-master/realfast/anaconda/envs/deployment3/lib/python3.6/site-packages/__casac__/atmosphere.py", line 17, in <module>
    _atmosphere = swig_import_helper()
  File "/home/cbe-master/realfast/anaconda/envs/deployment3/lib/python3.6/site-packages/__casac__/atmosphere.py", line 16, in swig_import_helper
    return importlib.import_module('_atmosphere')
  File "/home/cbe-master/realfast/anaconda/envs/deployment3/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named '_atmosphere'
caseyjlaw commented 5 years ago

FWIW, these versions work:

(development3) gygax$ conda list | grep casa
casa-data                 r20180426                     0    pkgw-forge
casa-python               5.1.2                    py36_0    pkgw-forge
casa-tools                5.1.2                         0    pkgw-forge
casacore                  2.4.1                    py36_2    conda-forge
(development3) gygax$ source activate deployment3

and these (newer ones) don't:

(deployment3) gygax$ conda list | grep casa
casa-data                 r20180919                     0    pkgw-forge
casa-python               5.4.1            py36hc5275ef_0    pkgw-forge
casa-tools                5.4.1                h738cfa5_2    pkgw-forge
casacore                  2.4.2a0          py36he20af57_1    conda-forge
pkgw commented 5 years ago

@caseyjlaw Sorry this has been causing problems! But thanks for posting detailed information.

One issue at play is that I've only made builds for the most recent version of Python 3.x currently available. For the latest builds (CASA 5.4.1), that means Python 3.7. Since conda-forge moves so fast it's not surprising (although it's still frustrating) that the not-quite-so recent 3.6 packages have problems — I've been iterating towards trying to get the package specifications precise enough that you can't install mutually-incompatible versions of all of the dependent libraries, but that's still an ongoing task.

Should I interpret your earlier comment to mean that external factors are preventing you from upgrading to Python 3.7 right now? I can make fresh builds against 3.6, which should help ­— I just have to twiddle a few files and spend the CPU time.

caseyjlaw commented 5 years ago

I am a bit confused about what works. There are other conda issues that are making it hard to get this to work. For example, trying to start a new environment as a clone of my current one is not working. But yes, a new set of Python 3.6 builds would be really helpful. This is somewhat ironic, since I was requesting that you update to 3.7. Sorry about all this.

pkgw commented 5 years ago

Hey @caseyjlaw, I've uploaded a new casa-python. I managed to get a working environment, but it required a lot of manual futzing with package versions — hopefully that isn't generically true, but I think there's a bit of an issue since the base 3.6 packages were built with older C++ compilers, and the latest packages have shifted to newer ones. If that transition is done incompletely, you get library load failures.

Here's a listing of the package versions that worked for me:

asn1crypto                0.24.0                   py36_0  
blas                      1.1                    openblas    conda-forge
boost                     1.68.0          py36h8619c78_1001    conda-forge
boost-cpp                 1.68.0            h11c811c_1000    conda-forge
bzip2                     1.0.6                h14c3975_5  
ca-certificates           2019.3.9             hecc5488_0    conda-forge
cairo                     1.16.0            ha4e643d_1000    conda-forge
casa-python               5.4.1            py36hde29b37_2    file:///home/peter/sw/conda/pkgw-recipes/artifacts
casa-tools                5.4.1                h738cfa5_2    file:///home/peter/sw/conda/pkgw-recipes/artifacts
casacore                  2.4.2a0         py36hfc3e434_1004    conda-forge
certifi                   2019.3.9                 py36_0  
cffi                      1.11.4           py36h9745a5d_0  
cfitsio                   3.430             hc04bd9f_1001    conda-forge
chardet                   3.0.4            py36h0f667ec_1  
conda                     4.6.8                    py36_0    conda-forge
conda-env                 2.6.0                h36134e3_1  
cryptography              2.5              py36hb7f436b_1    conda-forge
curl                      7.61.0               h84994c4_0  
dbus                      1.13.0            h4e0c4b3_1000    conda-forge
dbus-cxx                  0.9.2             h956751d_1002    conda-forge
expat                     2.2.5                he0dffb1_0  
fftw                      3.3.8                h14c3975_0  
fontconfig                2.13.1            he4413a7_1000    conda-forge
freeglut                  3.0.0                hf484d3e_5  
freetype                  2.10.0               he983fc9_0    conda-forge
gettext                   0.19.8.1          h9745a5d_1001    conda-forge
glib                      2.58.3            hf63aee3_1001    conda-forge
graphite2                 1.3.11               h16798f4_2  
gsl                       2.4             blas_openblash5c7cb9b_1004  [blas_openblas]  conda-forge
harfbuzz                  2.3.1                h6824563_0    conda-forge
hdf5                      1.10.4          nompi_h11e915b_1105    conda-forge
icu                       58.2                 h9c2bf20_1  
idna                      2.6              py36h82fb2a8_1  
intel-openmp              2019.1                      144  
jpeg                      9c                h14c3975_1001    conda-forge
libcurl                   7.61.0               h1ad7b7a_0  
libedit                   3.1.20170329         h6b74fdf_2  
libffi                    3.2.1             hf484d3e_1005    conda-forge
libgcc-ng                 7.3.0                hdf63c60_0    conda-forge
libgfortran               3.0.0                         1  
libgfortran-ng            7.3.0                hdf63c60_0  
libiconv                  1.15              h14c3975_1004    conda-forge
libpng                    1.6.36            h84994c4_1000    conda-forge
libsakura                 3.0.0             hf484d3e_1000    conda-forge
libssh2                   1.8.0                h9cfc8f7_4  
libstdcxx-ng              8.2.0                hdf63c60_1  
libtiff                   4.0.9                he85c1e1_1  
libuuid                   2.32.1            h14c3975_1000    conda-forge
libxcb                    1.13                 h1bed415_1  
libxml2                   2.9.8                hf84eae3_0  
libxslt                   1.1.32               h1312cb7_0  
mkl                       2018.0.3                      1  
mkl_fft                   1.0.10                   py36_0    conda-forge
mkl_random                1.0.2                    py36_0    conda-forge
ncurses                   6.1                  hf484d3e_0  
numpy                     1.16.2          py36_blas_openblash1522bff_0  [blas_openblas]  conda-forge
openblas                  0.3.3             h9ac9557_1001    conda-forge
openssl                   1.0.2r               h14c3975_0    conda-forge
pango                     1.40.14           h4ea9474_1004    conda-forge
pcre                      8.42                 h439df22_0  
pgplot                    5.2.2             h3433452_1003    conda-forge
pip                       9.0.1            py36h6c6f9ce_4  
pixman                    0.34.0               hceecf20_3  
pycosat                   0.6.3           py36h14c3975_1001    conda-forge
pycparser                 2.18             py36hf9f622e_1  
pyopenssl                 17.5.0           py36h20ba746_0  
pysocks                   1.6.7            py36hd97a5b1_1  
python                    3.6.4                hc3d631a_1  
qt4                       4.8.7                hde88ee2_5    pkgw-forge
qwt5                      5.2.3                h6bb024c_6    file:///home/peter/sw/conda/pkgw-recipes/artifacts
readline                  7.0               hf8c457e_1001    conda-forge
requests                  2.18.4           py36he2e5f8d_1  
rpfits                    2.24              h9ac9557_1002    conda-forge
ruamel_yaml               0.15.35          py36h14c3975_1  
setuptools                38.4.0                   py36_0  
six                       1.11.0           py36h372c433_1  
sqlite                    3.22.0               h1bed415_0  
tk                        8.6.9             h84994c4_1000    conda-forge
urllib3                   1.22             py36hbe7ace6_0  
wcslib                    5.20              h9ac9557_1000    conda-forge
wheel                     0.30.0           py36hfd4bba0_1  
xerces-c28                2.8.0             h449daf6_1002    conda-forge
xorg-kbproto              1.0.7             h14c3975_1002    conda-forge
xorg-libice               1.0.9             h516909a_1004    conda-forge
xorg-libsm                1.2.3             h4937e3b_1000    conda-forge
xorg-libx11               1.6.7             h14c3975_1000    conda-forge
xorg-libxext              1.3.3             h516909a_1004    conda-forge
xorg-libxrender           0.9.10            h516909a_1002    conda-forge
xorg-renderproto          0.11.1            h14c3975_1002    conda-forge
xorg-xextproto            7.3.0             h14c3975_1002    conda-forge
xorg-xproto               7.0.31            h14c3975_1007    conda-forge
xz                        5.2.4             h14c3975_1001    conda-forge
yaml                      0.1.7             h14c3975_1001    conda-forge
zlib                      1.2.11            h14c3975_1004    conda-forge
caseyjlaw commented 5 years ago

Tricky! So I set up a script to install those versions, but I am still seeing the No module named '_atmosphere' error on import of casac.

From what I see, my versions are identical to yours, except for the ones from pkgw-forge. Mine come from the forge, while yours come from file:///home/peter/sw/conda/pkgw-recipes/artifacts. Are those two versions built identically?

pkgw commented 5 years ago

And here's the curated list with the packages that seem to be the most finicky:

boost                     1.68.0        py36h8619c78_1001    conda-forge
boost-cpp                 1.68.0            h11c811c_1000    conda-forge
casa-python               5.4.1            py36hde29b37_2    pkgw-forge
casa-tools                5.4.1                h738cfa5_2    pkgw-forge
casacore                  2.4.2a0       py36hfc3e434_1004    conda-forge
dbus-cxx                  0.9.2             h956751d_1002    conda-forge
qt4                       4.8.7                hde88ee2_5    pkgw-forge
qwt5                      5.2.3                h6bb024c_6    pkgw-forge
pkgw commented 5 years ago

@caseyjlaw Yes, the file:// ones are just the local copies that I test before I upload for general consumption.

So it is vexing that it isn't working! If you just run python -c "import casac", is there an ImportError line in the error output that gives a bit more information about a symbol not found or library not found?

caseyjlaw commented 5 years ago

It currently reports this: ImportError: libtinfo.so.6: cannot open shared object file: No such file or directory

This is different from the last error, so something changed at least.

pkgw commented 5 years ago

@caseyjlaw That file is associated with the ncurses package. Do you have exactly the right version of that one?

(edit: which I checked with grep libtinfo $(dirname $(which conda))/../conda-meta/*.json)

caseyjlaw commented 5 years ago

Here is what your grep finds:

grep libtinfo $(dirname $(which conda))/../conda-meta/*.json(deployment3) gygax$ grep libtinfo $(dirname $(which conda))/../conda-meta/*.json
/home/cbe-master/realfast/anaconda/envs/deployment3/bin/../conda-meta/ncurses-6.1-hf484d3e_0.json:    "lib/libtinfo.a",
/home/cbe-master/realfast/anaconda/envs/deployment3/bin/../conda-meta/ncurses-6.1-hf484d3e_0.json:    "lib/libtinfo.so",
/home/cbe-master/realfast/anaconda/envs/deployment3/bin/../conda-meta/ncurses-6.1-hf484d3e_0.json:    "lib/libtinfow.a",
/home/cbe-master/realfast/anaconda/envs/deployment3/bin/../conda-meta/ncurses-6.1-hf484d3e_0.json:    "lib/libtinfow.so",
/home/cbe-master/realfast/anaconda/envs/deployment3/bin/../conda-meta/ncurses-6.1-hf484d3e_0.json:    "lib/libtinfow.so.6",
/home/cbe-master/realfast/anaconda/envs/deployment3/bin/../conda-meta/ncurses-6.1-hf484d3e_0.json:    "lib/libtinfow.so.6.1",
/home/cbe-master/realfast/anaconda/envs/deployment3/bin/../conda-meta/ncurses-6.1-hf484d3e_0.json:        "_path": "lib/libtinfo.a",
/home/cbe-master/realfast/anaconda/envs/deployment3/bin/../conda-meta/ncurses-6.1-hf484d3e_0.json:        "_path": "lib/libtinfo.so",
/home/cbe-master/realfast/anaconda/envs/deployment3/bin/../conda-meta/ncurses-6.1-hf484d3e_0.json:        "_path": "lib/libtinfow.a",
/home/cbe-master/realfast/anaconda/envs/deployment3/bin/../conda-meta/ncurses-6.1-hf484d3e_0.json:        "_path": "lib/libtinfow.so",
/home/cbe-master/realfast/anaconda/envs/deployment3/bin/../conda-meta/ncurses-6.1-hf484d3e_0.json:        "_path": "lib/libtinfow.so.6",
/home/cbe-master/realfast/anaconda/envs/deployment3/bin/../conda-meta/ncurses-6.1-hf484d3e_0.json:        "_path": "lib/libtinfow.so.6.1",

So the library is indeed missing. Not sure why you have it!

If I update ncurses, it does this: pkgs/main::ncurses-6.1-hf484d3e_0 --> conda-forge::ncurses-6.1-hf484d3e_1002. And then I can import casac.