search

rdkit / conda-rdkit

Conda build recipe for the rdkit
50 stars 30 forks source link

Need some way to lock the version of boost to what we build against #71

Open greglandrum opened 6 years ago

greglandrum commented 6 years ago

Boost version numbers end up in the library names, so using DLLs from newer boost versions does not work.

This is connected to this rdkit issue. https://github.com/rdkit/rdkit/issues/1957

greglandrum commented 6 years ago

@rvianello : the immediately obvious way to do this would be to just directly match the version numbers in the host and run sections. Can you think of a better solution?

rvianello commented 6 years ago

Since a conda_build_config.yamlfile is already present for the rdkit recipe, maybe we could try using the pinning mechanism that is documented to work for build variants (it shouldn't necessarily imply building multiple rdkit packages).

The section with title "Pinning at the variant level" at https://conda.io/docs/user-guide/tasks/build-packages/variants.html provides an example that specifically addresses the boost case. The following section ("Pinning at the recipe level") illustrates how to obtain a similar result in the meta.yaml file alone.

markussitzmann commented 6 years ago

Does this already work? I merged all changes @greglandrum and @rvianello did in their forks and build in a fresh container, it build RDKit 2018.03.3, but at the end it is missing boost-1.65.1 and the container somehow has 1.67 installed:

`The following NEW packages will be INSTALLED:

blas:            1.0-mkl                         
bzip2:           1.0.6-h14c3975_5                
ca-certificates: 2018.03.07-0                    
cairo:           1.14.12-h8948797_3              
certifi:         2018.4.16-py36_0                
fontconfig:      2.13.0-h9420a91_0               
freetype:        2.9.1-h8a8886c_0                
glib:            2.56.1-h000015b_0               
icu:             58.2-h9c2bf20_1                 
intel-openmp:    2018.0.3-0                      
jpeg:            9b-h024ee3a_2                   
libboost:        1.67.0-h46d08c1_4               
libedit:         3.1.20170329-h6b74fdf_2         
libffi:          3.2.1-hd88cf55_4                
libgcc-ng:       7.2.0-hdf63c60_3                
libgfortran-ng:  7.2.0-hdf63c60_3                
libpng:          1.6.34-hb9fc6fc_0               
libstdcxx-ng:    7.2.0-hdf63c60_3                
libtiff:         4.0.9-he85c1e1_1                
libuuid:         1.0.3-h1bed415_2                
libxcb:          1.13-h1bed415_1                 
libxml2:         2.9.8-h26e45fe_1                
mkl:             2018.0.3-1                      
mkl_fft:         1.0.4-py36h4414c95_1            
mkl_random:      1.0.1-py36h4414c95_1            
ncurses:         6.1-hf484d3e_0                  
numpy:           1.15.0-py36h1b885b7_0           
numpy-base:      1.15.0-py36h3dfced4_0           
olefile:         0.45.1-py36_0                   
openssl:         1.0.2o-h14c3975_1               
pandas:          0.23.3-py36h04863e7_0           
pcre:            8.42-h439df22_0                 
pillow:          5.2.0-py36heded4f4_0            
pip:             10.0.1-py36_0                   
pixman:          0.34.0-hceecf20_3               
py-boost:        1.67.0-py36h04863e7_4           
python:          3.6.6-hc3d631a_0                
python-dateutil: 2.7.3-py36_0                    
pytz:            2018.5-py36_0                   
rdkit:           2018.03.3.0-py36h6bb024c_1 local
readline:        7.0-ha6073c6_4                  
setuptools:      39.2.0-py36_0                   
six:             1.11.0-py36_1                   
sqlite:          3.24.0-h84994c4_0               
tk:              8.6.7-hc745277_3                
wheel:           0.31.1-py36_0                   
xz:              5.2.4-h14c3975_4                
zlib:            1.2.11-ha838bed_2

Preparing transaction: ...working... done Verifying transaction: ...working... done Executing transaction: ...working... done import: 'rdkit' Traceback (most recent call last): File "/opt/conda/conda-bld/rdkit_1533517910182/test_tmp/run_test.py", line 2, in import rdkit File "/opt/conda/conda-bld/rdkit_1533517910182/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.6/site-packages/rdkit/init.py", line 2, in from .rdBase import rdkitVersion as version ImportError: libboost_python3.so.1.65.1: cannot open shared object file: No such file or directory Tests failed for rdkit-2018.03.3.0-py36h6bb024c_1.tar.bz2 - moving package to /opt/conda/conda-bld/broken WARNING:conda_build.build:Tests failed for rdkit-2018.03.3.0-py36h6bb024c_1.tar.bz2 - moving package to /opt/conda/conda-bld/broken removing: rdkit-2018.03.3.0-py36h6bb024c_1.tar.bz2 TESTS FAILED: rdkit-2018.03.3.0-py36h6bb024c_1.tar.bz2 ERROR: Service 'rdkit-conda-compile' failed to build: The command '/bin/sh -c CONDA_PY=36 conda install jinja2 conda-build --yes && CONDA_PY=36 conda build boost --quiet --no-anaconda-upload && CONDA_PY=36 conda build nox --quiet --no-anaconda-upload && CONDA_PY=36 conda build cairo_nox --quiet --no-anaconda-upload && CONDA_PY=36 conda build cairocffi --quiet --no-anaconda-upload && CONDA_PY=36 conda build eigen --quiet --no-anaconda-upload && CONDA_PY=36 CONDA_NPY=112 conda build rdkit --quiet --no-anaconda-upload' returned a non-zero code: 1`

greglandrum commented 6 years ago

@markussitzmann : when I do a fresh build using the development branch of this recipe everything works fine. Here's what gets installed for me on a centos6 container:

(py36_rdkit) [root@414215a97f78 conda-rdkit]# conda install --use-local rdkit
Solving environment: done

## Package Plan ##

  environment location: /opt/conda/envs/py36_rdkit

  added / updated specs: 
    - rdkit

The following NEW packages will be INSTALLED:

    blas:            1.0-mkl                         
    bzip2:           1.0.6-h14c3975_5                
    ca-certificates: 2018.03.07-0                    
    cairo:           1.14.12-h8948797_3              
    certifi:         2018.4.16-py36_0                
    fontconfig:      2.13.0-h9420a91_0               
    freetype:        2.9.1-h8a8886c_0                
    glib:            2.56.1-h000015b_0               
    icu:             58.2-h9c2bf20_1                 
    intel-openmp:    2018.0.3-0                      
    jpeg:            9b-h024ee3a_2                   
    libboost:        1.65.1-habcd387_4               
    libedit:         3.1.20170329-h6b74fdf_2         
    libffi:          3.2.1-hd88cf55_4                
    libgcc-ng:       7.2.0-hdf63c60_3                
    libgfortran-ng:  7.2.0-hdf63c60_3                
    libpng:          1.6.34-hb9fc6fc_0               
    libstdcxx-ng:    7.2.0-hdf63c60_3                
    libtiff:         4.0.9-he85c1e1_1                
    libuuid:         1.0.3-h1bed415_2                
    libxcb:          1.13-h1bed415_1                 
    libxml2:         2.9.8-h26e45fe_1                
    mkl:             2018.0.3-1                      
    mkl_fft:         1.0.4-py36h4414c95_1            
    mkl_random:      1.0.1-py36h4414c95_1            
    ncurses:         6.1-hf484d3e_0                  
    numpy:           1.15.0-py36h1b885b7_0           
    numpy-base:      1.15.0-py36h3dfced4_0           
    olefile:         0.45.1-py36_0                   
    openssl:         1.0.2o-h14c3975_1               
    pandas:          0.23.3-py36h04863e7_0           
    pcre:            8.42-h439df22_0                 
    pillow:          5.2.0-py36heded4f4_0            
    pip:             10.0.1-py36_0                   
    pixman:          0.34.0-hceecf20_3               
    py-boost:        1.65.1-py36hf484d3e_4           
    python:          3.6.6-hc3d631a_0                
    python-dateutil: 2.7.3-py36_0                    
    pytz:            2018.5-py36_0                   
    rdkit:           2018.03.3.0-py36h71b666b_1 local
    readline:        7.0-ha6073c6_4                  
    setuptools:      39.2.0-py36_0                   
    six:             1.11.0-py36_1                   (py36_rdkit) [root@414215a97f78 conda-rdkit]# conda install --use-local rdkit
Solving environment: done

## Package Plan ##

  environment location: /opt/conda/envs/py36_rdkit

  added / updated specs: 
    - rdkit

The following NEW packages will be INSTALLED:

    blas:            1.0-mkl                         
    bzip2:           1.0.6-h14c3975_5                
    ca-certificates: 2018.03.07-0                    
    cairo:           1.14.12-h8948797_3              
    certifi:         2018.4.16-py36_0                
    fontconfig:      2.13.0-h9420a91_0               
    freetype:        2.9.1-h8a8886c_0                
    glib:            2.56.1-h000015b_0               
    icu:             58.2-h9c2bf20_1                 
    intel-openmp:    2018.0.3-0                      
    jpeg:            9b-h024ee3a_2                   
    libboost:        1.65.1-habcd387_4               
    libedit:         3.1.20170329-h6b74fdf_2         
    libffi:          3.2.1-hd88cf55_4                
    libgcc-ng:       7.2.0-hdf63c60_3                
    libgfortran-ng:  7.2.0-hdf63c60_3                
    libpng:          1.6.34-hb9fc6fc_0               
    libstdcxx-ng:    7.2.0-hdf63c60_3                
    libtiff:         4.0.9-he85c1e1_1                
    libuuid:         1.0.3-h1bed415_2                
    libxcb:          1.13-h1bed415_1                 
    libxml2:         2.9.8-h26e45fe_1                
    mkl:             2018.0.3-1                      
    mkl_fft:         1.0.4-py36h4414c95_1            
    mkl_random:      1.0.1-py36h4414c95_1            
    ncurses:         6.1-hf484d3e_0                  
    numpy:           1.15.0-py36h1b885b7_0           
    numpy-base:      1.15.0-py36h3dfced4_0           
    olefile:         0.45.1-py36_0                   
    openssl:         1.0.2o-h14c3975_1               
    pandas:          0.23.3-py36h04863e7_0           
    pcre:            8.42-h439df22_0                 
    pillow:          5.2.0-py36heded4f4_0            
    pip:             10.0.1-py36_0                   
    pixman:          0.34.0-hceecf20_3               
    py-boost:        1.65.1-py36hf484d3e_4           
    python:          3.6.6-hc3d631a_0                
    python-dateutil: 2.7.3-py36_0                    
    pytz:            2018.5-py36_0                   
    rdkit:           2018.03.3.0-py36h71b666b_1 local
    readline:        7.0-ha6073c6_4                  
    setuptools:      39.2.0-py36_0                   
    six:             1.11.0-py36_1                   
    sqlite:          3.24.0-h84994c4_0               
    tk:              8.6.7-hc745277_3                
    wheel:           0.31.1-py36_0                   
    xz:              5.2.4-h14c3975_4                
    zlib:            1.2.11-ha838bed_2               

Proceed ([y]/n)? 

    sqlite:          3.24.0-h84994c4_0               
    tk:              8.6.7-hc745277_3                
    wheel:           0.31.1-py36_0                   
    xz:              5.2.4-h14c3975_4                
    zlib:            1.2.11-ha838bed_2               

Proceed ([y]/n)?
markussitzmann commented 6 years ago

@greglandrum Thanks for the testing and reply - although I hate it :-) ... I definitely have put boost-1.65.1 everywhere and diff'ed with yours and Riccardo's branches/forks. But somehow it ends up with boost-1.67.

I can retry only tonight. On which distro did you build? Centos? I build on debian:stretch

greglandrum commented 6 years ago

@markussitzmann : It also didn't work for me when I tried to force boost 1.65.1. The current branch is set to use 1.65, which grabs 1.65.1 by default.

I'm using centos, but that shouldn't make a difference.

Can you point me to the code you're using to do the build?

markussitzmann commented 6 years ago

While looking for the code I might have found the error I made but as I said I can only try tonight again (did the fixes in my repository but checked out the wrong branch during the Docker build - and my attempts were revolving around this 1.65 vs 1.65.1 thingy). Thanks again - I only will come back if I still struggle ;-).

markussitzmann commented 6 years ago

@greglandrum: hmm, when I run the development branch of this recipe here, the build works now, however now I receive boost 1.63 as built boost package and then starting rdkit fails because it is missing 1.65.1 ... a bit mysterious.

markussitzmann commented 6 years ago

I run this Dockerfile:

https://github.com/chembience/chembience/blob/master/context/base/rdkit-conda-compile/Dockerfile

after replacing line 44-46 with the standard conda-rdkit repository and branch development:

RUN git clone https://github.com/rdkit/conda-rdkit.git WORKDIR conda-rdkit RUN git checkout development

greglandrum commented 6 years ago

Unless you need it for something else, you can probably skip the boost build now (line 51 in your Dockerfile); that will use the boost build recipe from the RDKit channel, which will use 1.63. You almost certainly don't need to build eigen either.

I don't see anything else obviously wrong in the dockerfile.

When you get failures: do they happen during the build or later when you try and use the contents of the image?

greglandrum commented 6 years ago

FWIW: these two containers are what I am using to do conda RDKit builds: https://github.com/rdkit/rdkit_containers/tree/centos6_cleanup/docker/centos6_modern_bare https://github.com/rdkit/rdkit_containers/tree/centos6_cleanup/docker/centos6_modernconda

markussitzmann commented 6 years ago

@greglandrum Thanks for your hints. The builds fail in different places depending what I try. Sometimes during conda is packaging, sometimes when I start RDKit which then misses the right boost version (this happens with a subsequent image which copies all compiled packages from the image I build with the Dockerfile above).

Yep, and I need to give my Dockerfiles another clean up, they have collected old, overcautious or unnecessary stuff.

I am also considering to just switch to your already available anaconda packages; the reason for compiling RDKit myself was that very early last year, when I started to turn my preliminary works into something real, the anaconda RDKit packages still were on Python 3.5 and I had decided to go for Python 3.6 (BTW, Python 3.7 is out now for a few months already :-))

greglandrum commented 6 years ago

Yeah, assuming that there is an anaconda distrib that supports it by then, I will do Python 3.7 builds (and probably retire 3.5) for the 2018.09 release.

szcc commented 5 years ago

I have exactly same problem :ImportError: libboost_python3.so.1.65.1: cannot open shared object file: No such file or directory Tests failed for rdkit-2018.03.3.0-py36h6bb024c_1.tar.bz2 - moving package to /opt/conda/conda-bld/broken. Is there any solution for this?