search

EichlerLab / smrtsv2

Structural variant caller
MIT License
53 stars 6 forks source link

Unable to compile dependencies #3

Closed RSherman15 closed 5 years ago

RSherman15 commented 5 years ago

Hi, I'm trying to install smrtsv2, and I can't seem to get it to build the dependencies. I get the following error: 

File "/home-net/home-4/rsherma8@jhu.edu/bin/packages/smrtsv2/dep/conda/build/envs/python2/bin/pip", line 7, in <module>
    from pip._internal import main
ImportError: No module named _internal

which seems to be a problem with pip; however the issue is with the version of pip being installed by smrtsv -- I already have an installation of pip that works perfectly well. I'm also not clear on why make seems to be installing with both python2 and python3 -- are both needed? It seems to be doing an install with each environment, first successfully with python3:

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
date > build/install_flags/env_python3_install
Solving environment: done

and then with the error for python2:

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use:
# > source activate python2
#
# To deactivate an active environment, use:
# > source deactivate
#
date > build/install_flags/env_python2_deploy
Warning: 'defaults' already in 'channels' list, moving to the top
Warning: 'bioconda' already in 'channels' list, moving to the top
Warning: 'conda-forge' already in 'channels' list, moving to the top
Traceback (most recent call last):
  File "/home-net/home-4/rsherma8@jhu.edu/bin/packages/smrtsv2/dep/conda/build/envs/python2/bin/pip", line 7, in <module>
    from pip._internal import main
ImportError: No module named _internal
make[1]: *** [build/install_flags/env_python2_install] Error 1

I am using Linux CentOS 7. Both python 2.7 and 3.7 are installed, python points to 3.7 by default, but pointing it to 2.7 does not change the error. Any help would be appreciated.

paudano commented 5 years ago

Yes, it does look like a pip error. The build system installs specific versions of everything, so I should be seeing it too. I'll try to reproduce the problem today and see what happens.

You can try commenting pip install --upgrade pip in dep/conda/install_python2.sh and see if that changes anything.

Yes, SMRT-SV needs python2 and python3 environments because it has a mix of python2 and python3 code. Keeping them in their own environments mitigates dependency conflicts; that's why there are 4 independent conda environments. SMRT-SV knows which one to call as needed. Your system's python is never used.

paudano commented 5 years ago

I am still able to build the conda dependencies on multiple systems.

Try commenting that line (pip install --upgrade pip in dep/conda/install_python2.sh) and see if it gets past the error.

RSherman15 commented 5 years ago

Commenting out the pip upgrade, and doing the same thing in install_pacbio.sh as well (otherwise I just get the same error there too) got me past this error. However, I still cannot fully compile the dependencies, getting the following error:

# Flag complete
date > install_flags/dep_conda_link
mkdir -p install_flags
make -C smrtsvtools
make[1]: Entering directory `/home-net/home-4/rsherma8@jhu.edu/bin/packages/smrtsv2/dep/smrtsvtools'
mkdir -p build/
/home-net/home-4/rsherma8@jhu.edu/bin/packages/smrtsv2/dep/conda/build/envs/tools/bin/g++ -c  -I/home-net/home-4/rsherma8@jhu.edu/bin/packages/smrtsv2/dep/smrtsvtools/include -I/include -std=c++11 -g -o build/alignfixup.o src/alignfixup.cpp
In file included from /software/apps/compilers/gcc/5.5.0/lib/gcc/x86_64-redhat-linux/5.5.0/include/x86intrin.h:27:0,
                 from /home-net/home-4/rsherma8@jhu.edu/bin/packages/smrtsv2/dep/conda/build/envs/tools/gcc/include/c++/x86_64-unknown-linux-gnu/bits/opt_random.h:33,
                 from /home-net/home-4/rsherma8@jhu.edu/bin/packages/smrtsv2/dep/conda/build/envs/tools/gcc/include/c++/random:51,
                 from /home-net/home-4/rsherma8@jhu.edu/bin/packages/smrtsv2/dep/conda/build/envs/tools/gcc/include/c++/bits/stl_algo.h:65,
                 from /home-net/home-4/rsherma8@jhu.edu/bin/packages/smrtsv2/dep/conda/build/envs/tools/gcc/include/c++/algorithm:62,
                 from /software/centos7/usr/include/boost/any.hpp:13,
                 from /software/centos7/usr/include/boost/program_options/value_semantic.hpp:12,
                 from /software/centos7/usr/include/boost/program_options/options_description.hpp:13,
                 from /software/centos7/usr/include/boost/program_options.hpp:15,
                 from src/alignfixup.cpp:17:
/software/apps/compilers/gcc/5.5.0/lib/gcc/x86_64-redhat-linux/5.5.0/include/ia32intrin.h: In function ‘long long unsigned int __readeflags()’:
/software/apps/compilers/gcc/5.5.0/lib/gcc/x86_64-redhat-linux/5.5.0/include/ia32intrin.h:246:41: error: ‘__builtin_ia32_readeflags_u64’ was not declared in this scope
   return __builtin_ia32_readeflags_u64 ();
                                         ^
/software/apps/compilers/gcc/5.5.0/lib/gcc/x86_64-redhat-linux/5.5.0/include/ia32intrin.h: In function ‘void __writeeflags(long long unsigned int)’:
/software/apps/compilers/gcc/5.5.0/lib/gcc/x86_64-redhat-linux/5.5.0/include/ia32intrin.h:254:38: error: ‘__builtin_ia32_writeeflags_u64’ was not declared in this scope
   __builtin_ia32_writeeflags_u64 (__X);
                                      ^
In file included from /software/apps/compilers/gcc/5.5.0/lib/gcc/x86_64-redhat-linux/5.5.0/include/immintrin.h:45:0,
                 from /software/apps/compilers/gcc/5.5.0/lib/gcc/x86_64-redhat-linux/5.5.0/include/x86intrin.h:46,
                 from /home-net/home-4/rsherma8@jhu.edu/bin/packages/smrtsv2/dep/conda/build/envs/tools/gcc/include/c++/x86_64-unknown-linux-gnu/bits/opt_random.h:33,
                 from /home-net/home-4/rsherma8@jhu.edu/bin/packages/smrtsv2/dep/conda/build/envs/tools/gcc/include/c++/random:51,
                 from /home-net/home-4/rsherma8@jhu.edu/bin/packages/smrtsv2/dep/conda/build/envs/tools/gcc/include/c++/bits/stl_algo.h:65,
                 from /home-net/home-4/rsherma8@jhu.edu/bin/packages/smrtsv2/dep/conda/build/envs/tools/gcc/include/c++/algorithm:62,
                 from /software/centos7/usr/include/boost/any.hpp:13,
                 from /software/centos7/usr/include/boost/program_options/value_semantic.hpp:12,
                 from /software/centos7/usr/include/boost/program_options/options_description.hpp:13,
                 from /software/centos7/usr/include/boost/program_options.hpp:15,
                 from src/alignfixup.cpp:17:
/software/apps/compilers/gcc/5.5.0/lib/gcc/x86_64-redhat-linux/5.5.0/include/avx512fintrin.h: At global scope:
/software/apps/compilers/gcc/5.5.0/lib/gcc/x86_64-redhat-linux/5.5.0/include/avx512fintrin.h:33:29: error: attribute(target("avx512f")) is unknown
 #pragma GCC target("avx512f")
                             ^
/software/apps/compilers/gcc/5.5.0/lib/gcc/x86_64-redhat-linux/5.5.0/include/avx512fintrin.h:33: confused by earlier errors, bailing out
make[1]: *** [build/alignfixup.o] Error 1
make[1]: Leaving directory `/home-net/home-4/rsherma8@jhu.edu/bin/packages/smrtsv2/dep/smrtsvtools'
make: *** [install_flags/dep_smrtsvtools_build] Error 2

Might this be a gcc version issue? If you think that might be the issue, what version(s) have you successfully compiled with?

RSherman15 commented 5 years ago

Tried again with gcc 4.8.5 instead of 5.5.0 and was successful. You may want to add the gcc version(s) this has been tested with to the documentation.

paudano commented 5 years ago

It uses the version of GCC installed by conda. It does not use your system GCC. I compile it on a machine with GCC 8.2.1, and it works fine.

You can see this from the error message: /home-net/home-4/rsherma8@jhu.edu/bin/packages/smrtsv2/dep/conda/build/envs/tools/bin/g++ ...

I think what's happening is your module system is setting CXXFLAGS, and the Makefile for smrtsvtools is letting that through instead of overwriting it. If I fix the Makefile to ignore GCC environment variables, would you test it again?

RSherman15 commented 5 years ago

As I said I did successfully get it to build the dependencies, so I'm now running it (seems to be working fine). But sure; I'm happy to test it out for you if you update the Makefile.

wdecoster commented 5 years ago

FWIW I encountered the same issue with ImportError: No module named _internal, solved by commenting out both pip install --upgrade pip lines.

paudano commented 5 years ago

I removed the pip upgrade lines for all conda environments in dep. Hoping that doesn't break something else, but it looks like it was causing too many problems. Thanks @RSherman15 and @wdecoster for pointing out this problem.