GLIBC Hell

[kyleabeauchamp]

So I'm finding that the 14.0.1 release of ambermini no longer works on our cluster due to the usual GLIBC nonsense:

-bash-4.1$ conda install -c https://conda.binstar.org/omnia ambermini
The following NEW packages will be INSTALLED:

    ambermini: 14.0.1-0
-bash-4.1$ parmchk2
/cbio/jclab/home/kyleb/opt/bin/parmchk2_pvt: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /cbio/jclab/home/kyleb/opt/bin/parmchk2_pvt)

However, I find that using an older version from @pgrinaway seems to work:

-bash-4.1$ ls /cbio/jclab/home/pgrinaway/anaconda/pkgs/ambermini*.bz2
/cbio/jclab/home/pgrinaway/anaconda/pkgs/ambermini-14.0.1-0.tar.bz2  /cbio/jclab/home/pgrinaway/anaconda/pkgs/ambermini-14-0.tar.bz2

conda install /cbio/jclab/home/pgrinaway/anaconda/pkgs/ambermini-14-0.tar.bz2

-bash-4.1$ parmchk2 
AMBERHOME is not set!
-bash-4.1$ antechamber
Usage: antechamber -i   input file name
                   -fi  input file format

I'm trying to figure out what has changed since the last build. I also tried rebuilding using my Centos6.5 virtual machine, to no avail.

(Also, I had to add sudo yum install gcc-gfortran to the VM to make it work, which we should add to the VM once we figure out how to get this working).

[kyleabeauchamp]

The exact commands for the build were:

git clone git@github.com:omnia-md/virtual-machines.git
cd virtual-machines/linux/
vagrant up
vagrant ssh

export PATH=$HOME/miniconda/bin:$PATH
sudo yum install gcc-gfortran

git clone https://github.com/omnia-md/conda-recipes.git
cd conda-recipes
conda build ambermini

[rmcgibbo]

What version of gcc-gfortran did you install? On the VM, after building ambermini, try installing it and then running ldd on one of the shared object files. One of the lines listed should be libc.so.6, and then you can run readelf on that to see what version the symbols are:

e.g.

readelf -Ws /lib/x86_64-linux-gnu/libc.so.6 | grep GLIBC

[rmcgibbo]

Upgrading your compiler toolchain is going to upgrade your libc/libstdc++, which is probably not what you want, given that the point of these VMs is to link against older versions of the symbols.

[rmcgibbo]

The Version history table here http://en.wikipedia.org/wiki/GNU_C_Library is particularly useful, since it gives you the canonical glibc version for each rhel/debian release.

[kyleabeauchamp]

Yeah, I somehow thought I had already this VM set up to work properly...

[vagrant@localhost ~]$ gfortran  -v
Using built-in specs.
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk --disable-dssi --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre --enable-libgcj-multifile --enable-java-maintainer-mode --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libjava-multilib --with-ppl --with-cloog --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux
Thread model: posix
gcc version 4.4.7 20120313 (Red Hat 4.4.7-11) (GCC) 

[kyleabeauchamp]

OK, I think the issue is that my VM is now 6.6:

[vagrant@localhost ~]$ cat /etc/centos-release
CentOS release 6.6 (Final)

[rmcgibbo]

Can you post the output of readelf on the libc.so?

[kyleabeauchamp]

https://gist.github.com/kyleabeauchamp/dcd6c82b8ab2b006dc3b

[rmcgibbo]

Can you also post ldd --version?

[kyleabeauchamp]

[vagrant@localhost ~]$ ldd --version
ldd (GNU libc) 2.12
Copyright (C) 2010 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

[rmcgibbo]

And yet the binaries you made on this machine were actually crapping out, saying 'GLIBC_2.14' not found? That doesn't seem right.

[kyleabeauchamp]

My bad

[kyleabeauchamp]

I think it may have been due to binstar's refusal to overwrite packages during upload. Not sure. Regardless, fixed.

[kyleabeauchamp]

No, there actually is something messed up in the current conda package for omnia/ambermini:

cd omniaambermini
# https://binstar.org/omnia/ambermini/14.0.1/download/linux-64/ambermini-14.0.1-0.tar.bz2
tar -jxvf linux-64_ambermini-14.0.1-0.tar.bz2
readelf -Ws  ./bin/parmchk2_pvt |grep GLIBC|grep 14

cd ../kyleambermini
# https://binstar.org/kyleabeauchamp/ambermini/14.0.1/download/linux-64/ambermini-14.0.1-0.tar.bz2
tar -jxvf linux-64_ambermini-14.0.1-0.tar.bz2
readelf -Ws  ./bin/parmchk2_pvt |grep GLIBC|grep 14

kyleb@kyleb-DX4870:~/src/Software/omniambermini$ readelf -Ws  ./bin/parmchk2_pvt |grep GLIBC|grep 14
    14: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND modf@GLIBC_2.2.5 (3)
    29: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND memcpy@GLIBC_2.14 (5)
   140: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND fopen64@@GLIBC_2.2.5
   142: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND memcpy@@GLIBC_2.14
   214: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND fwrite@@GLIBC_2.2.5

kyleb@kyleb-DX4870:~/src/Software/kyleambermini$ readelf -Ws  ./bin/parmchk2_pvt |grep GLIBC|grep 14
    14: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND system@GLIBC_2.2.5 (3)
   141: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND fputs@@GLIBC_2.2.5
   147: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND atoi@@GLIBC_2.2.5

[kyleabeauchamp]

If someone else can verify that my new conda package works, I can upload it to the omnia conda repo.

[kyleabeauchamp]

Here are some more command lines if people want to do tests on a fresh miniconda:

MINICONDA=Miniconda-latest-Linux-x86_64.sh
wget http://repo.continuum.io/miniconda/$MINICONDA
bash Miniconda-latest-Linux-x86_64.sh
./miniconda/bin/conda install -c https://conda.binstar.org/omnia ambermini

MINICONDA=Miniconda-latest-Linux-x86_64.sh
wget http://repo.continuum.io/miniconda/$MINICONDA
bash Miniconda-latest-Linux-x86_64.sh
./miniconda/bin/conda install -c https://conda.binstar.org/kyleabeauchamp ambermini 

[bas-rustenburg]

Tried it.

The Omnia one is broken when I try it on a fresh miniconda install. Kyle's one does work. (Centos 6.4, hal.cbio.mskcc.org)

[jchodera]

Is this something I can try to automate the building of now?

[kyleabeauchamp]

Sure, I have my exact command lines listed near the top of this thread. The gcc-gfortran install should no longer be needed because it's part of the vagrant script now.

[kyleabeauchamp]

Any progress on automating this? If not, I will just upload the package that I already built.

[kyleabeauchamp]

OK, I uploaded my new package to omnia, so I'm closing the issue now.

https://binstar.org/omnia/ambermini/files

[jchodera]

@kyleabeauchamp : Can you show me how you built this today? Hoping to automate this too.

[kyleabeauchamp]

Sure. AFAIK, I believe this sequence of commands will finish the entire build chain:

git clone git@github.com:omnia-md/virtual-machines.git
cd virtual-machines/linux/
vagrant up
vagrant ssh

export PATH=$HOME/miniconda/bin:$PATH

git clone https://github.com/omnia-md/conda-recipes.git
cd conda-recipes
conda build ambermini

binstar upload -u omnia  PATH_TO_OUTPUT/ambermini-14.0.1-0.tar.bz2  --force
# You can get the path to the output from reading the output of the conda build.