Closed MarcusLCC closed 2 years ago
GWW
commented
2 years ago Hi Marcus,
Apologies for the issues compiling. When you installed conda did you rerun cmake? If you can try removing the build folder and running cmake again when your conda environment is active it should find the autotools commands.
The autotools part is to compile libhts (the samtools library) and If you can't get it to work I can try to find an alternative method to specify a path to a pre-existing install.
MarcusLCC
commented
2 years ago Hi GWW,
Thank you for the quick and kind reply. For reproducibility, I created a new conda environment and re-do the installation process as follows:
(base) -bash-4.2$conda create -n scsnv python=3.8
(base) -bash-4.2$conda activate scsnv
(scsnv) -bash-4.2$conda install -c conda-forge hdf5
(scsnv) -bash-4.2$conda install -c conda-forge cmake
(scsnv) -bash-4.2$conda install -c conda-forge cxx-compiler
(scsnv) -bash-4.2$conda install -c conda-forge autoconf
(scsnv) -bash-4.2$ cd /home/marcuslc/Tools/scsnv/
(scsnv) -bash-4.2$ mkdir build
(scsnv) -bash-4.2$ cd build/
(scsnv) -bash-4.2$ cmake -DCMAKE_BUILD_TYPE=Release ..
-- The C compiler identification is GNU 9.3.0
-- The CXX compiler identification is GNU 9.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /home/marcuslc/miniconda3/envs/scsnv/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/marcuslc/miniconda3/envs/scsnv/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- HDF5 C compiler wrapper is unable to compile a minimal HDF5 program.
-- HDF5 CXX compiler wrapper is unable to compile a minimal HDF5 program.
-- Found HDF5: /home/marcuslc/miniconda3/envs/scsnv/lib/libhdf5_cpp.so;/home/marcuslc/miniconda3/envs/scsnv/lib/libhdf5.so (found version "1.12.1") found components: C CXX HL
HDF5 LIBRARY: /home/marcuslc/miniconda3/envs/scsnv/lib/libhdf5_cpp.so;/home/marcuslc/miniconda3/envs/scsnv/lib/libhdf5.so
HDF5 LIBRARY CXX: /home/marcuslc/miniconda3/envs/scsnv/lib/libhdf5_cpp.so;/home/marcuslc/miniconda3/envs/scsnv/lib/libhdf5.so
HDF5 INCLUDE: /home/marcuslc/miniconda3/envs/scsnv/include
BUILD TYPE: Release
INSTALL PREFIX: /usr/local
-- Configuring done
-- Generating done
-- Build files have been written to: /home/marcuslc/Tools/scsnv/build
(scsnv) -bash-4.2$ make
Scanning dependencies of target libhts
[ 2%] Generating ../../external/htslib/libhts.a
autoreconf: 'configure.ac' or 'configure.in' is required
make[2]: *** [../external/htslib/libhts.a] Error 1
make[1]: *** [src/CMakeFiles/libhts.dir/all] Error 2
make: *** [all] Error 2The error still persisted.
And I also tried to install the htslib through conda conda install -c bioconda htslib, then re-create the build directory, which didn't work (with the same error).
I then checked the path of autoreconf, which seemed to be in the environment.
(scsnv) -bash-4.2$ which autoreconf
~/miniconda3/envs/scsnv/bin/autoreconfHere is the full list of the packages in this conda environment:
(scsnv) -bash-4.2$ conda list
# packages in environment at /home/marcuslc/miniconda3/envs/scsnv:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 4.5 1_gnu
autoconf 2.69 pl5320h36c2ea0_10 conda-forge
binutils 2.35.1 hdd6e379_2 conda-forge
binutils_impl_linux-64 2.35.1 h27ae35d_9
binutils_linux-64 2.35 h67ddf6f_30 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.17.1 h27cfd23_0
c-compiler 1.2.0 h7f98852_0 conda-forge
ca-certificates 2021.10.8 ha878542_0 conda-forge
certifi 2021.10.8 py38h06a4308_0
cmake 3.19.6 h973ab73_0
cxx-compiler 1.2.0 h4bd325d_0 conda-forge
expat 2.4.1 h9c3ff4c_0 conda-forge
gcc_impl_linux-64 9.3.0 h70c0ae5_19 conda-forge
gcc_linux-64 9.3.0 hf25ea35_30 conda-forge
gxx_impl_linux-64 9.3.0 hd87eabc_19 conda-forge
gxx_linux-64 9.3.0 h3fbe746_30 conda-forge
hdf5 1.12.1 nompi_h2750804_100 conda-forge
htslib 1.13 h9093b5e_0 bioconda
kernel-headers_linux-64 2.6.32 he073ed8_14 conda-forge
krb5 1.19.2 hcc1bbae_0 conda-forge
ld_impl_linux-64 2.35.1 h7274673_9
libcurl 7.78.0 h0b77cf5_0
libdeflate 1.7 h27cfd23_5
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libffi 3.3 he6710b0_2
libgcc-devel_linux-64 9.3.0 h7864c58_19 conda-forge
libgcc-ng 9.3.0 h5101ec6_17
libgfortran-ng 11.2.0 h69a702a_11 conda-forge
libgfortran5 11.2.0 h5c6108e_11 conda-forge
libgomp 9.3.0 h5101ec6_17
libnghttp2 1.43.0 h812cca2_0 conda-forge
libssh2 1.9.0 h1ba5d50_1
libstdcxx-devel_linux-64 9.3.0 hb016644_19 conda-forge
libstdcxx-ng 9.3.0 hd4cf53a_17
libuv 1.42.0 h7f98852_0 conda-forge
lz4-c 1.9.3 h9c3ff4c_1 conda-forge
m4 1.4.18 h516909a_1001 conda-forge
ncurses 6.2 he6710b0_1
openssl 1.1.1l h7f8727e_0
perl 5.32.1 0_h7f98852_perl5 conda-forge
pip 21.2.4 py38h06a4308_0
python 3.8.12 h12debd9_0
python_abi 3.8 2_cp38 conda-forge
readline 8.1 h27cfd23_0
rhash 1.4.1 h7f98852_0 conda-forge
setuptools 58.0.4 py38h06a4308_0
sqlite 3.36.0 hc218d9a_0
sysroot_linux-64 2.12 he073ed8_14 conda-forge
tk 8.6.11 h1ccaba5_0
wheel 0.37.0 pyhd3eb1b0_1
xz 5.2.5 h7b6447c_0
zlib 1.2.11 h7b6447c_3
zstd 1.4.9 ha95c52a_0 conda-forgeI guess it's still about environment setup, but I couldn't find it out. Could you please give me some suggestions? Thanks again for your help.
Regards, Marcus
GWW
commented
2 years ago Hi Marcus,
Thanks for all the details. I will try to reproduce exactly what you did to get to the bottom of the issue.
One thing you could try is to manually go into the external/htslib folder and just type make. If it compiles okay you can probably build scSNV after without needing to use autotools.
It will essentially skip cmake trying to auto compile htslib since you did it manually
MarcusLCC
commented
2 years ago Hi GWW,
Thank you for your suggestions. I found out I made a mistake. I downloaded the zip file from here and uploaded to the server since there's some issue about git clone on my server, and the subdirectories under 'external' were all empty.
I figured out another way to get the repository and re-run the installation. It went well for most of the part, but failed at the last 5%. The environment is the same as built before.
I'll paste the log (especially the part with warnings) below for your kind reference.
(scsnvClone) -bash-4.2$ cmake -DCMAKE_BUILD_TYPE=Release ..
-- The C compiler identification is GNU 9.3.0
-- The CXX compiler identification is GNU 9.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /home/marcuslc/miniconda3/envs/scsnvClone/bin/x86_64-conda-linux-gnu-cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/marcuslc/miniconda3/envs/scsnvClone/bin/x86_64-conda-linux-gnu-c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- HDF5 C compiler wrapper is unable to compile a minimal HDF5 program.
-- HDF5 CXX compiler wrapper is unable to compile a minimal HDF5 program.
-- Found HDF5: /home/marcuslc/miniconda3/envs/scsnvClone/lib/libhdf5_cpp.so;/home/marcuslc/miniconda3/envs/scsnvClone/lib/libhdf5.so (found version "1.12.1") found components: C CXX HL
HDF5 LIBRARY: /home/marcuslc/miniconda3/envs/scsnvClone/lib/libhdf5_cpp.so;/home/marcuslc/miniconda3/envs/scsnvClone/lib/libhdf5.so
HDF5 LIBRARY CXX: /home/marcuslc/miniconda3/envs/scsnvClone/lib/libhdf5_cpp.so;/home/marcuslc/miniconda3/envs/scsnvClone/lib/libhdf5.so
HDF5 INCLUDE: /home/marcuslc/miniconda3/envs/scsnvClone/include
BUILD TYPE: Release
INSTALL PREFIX: /usr/local
-- Configuring done
-- Generating done
-- Build files have been written to: /home/marcuslc/Tools/scsnv/build
(scsnvClone) -bash-4.2$ make
Scanning dependencies of target libhts
[ 2%] Generating ../../external/htslib/libhts.a
checking for x86_64-conda-linux-gnu-gcc... /home/marcuslc/miniconda3/envs/scsnvClone/bin/x86_64-conda-linux-gnu-cc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
...
...
checking for curl_easy_pause in -lcurl... no
checking for curl_easy_init in -lcurl... no
configure: WARNING: libcurl not enabled: library not found
configure: WARNING: GCS support not enabled: requires libcurl support
configure: WARNING: S3 support not enabled: requires libcurl support
checking whether PTHREAD_MUTEX_RECURSIVE is declared... yes
configure: creating ./config.status
...
...
[ 2%] Built target libhts
Scanning dependencies of target scsnvlib
[ 4%] Building CXX object src/CMakeFiles/scsnvlib.dir/aux.cpp.o
[ 6%] Building CXX object src/CMakeFiles/scsnvlib.dir/barcodes.cpp.o
[ 8%] Building CXX object src/CMakeFiles/scsnvlib.dir/build.cpp.o
...
...
[ 89%] Building C object src/CMakeFiles/scsnvlib.dir/bwa/bwtindex.c.o
[ 91%] Linking CXX static library libscsnvlib.a
[ 91%] Built target scsnvlib
Scanning dependencies of target rfilter
[ 93%] Building CXX object src/CMakeFiles/rfilter.dir/rfilter.cpp.o
[ 95%] Linking CXX executable rfilter
/home/marcuslc/miniconda3/envs/scsnvClone/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/marcuslc/miniconda3/envs/scsnvClone/lib/libstdc++.so: undefined reference to `aligned_alloc@GLIBC_2.16'
/home/marcuslc/miniconda3/envs/scsnvClone/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/marcuslc/miniconda3/envs/scsnvClone/lib/libstdc++.so: undefined reference to `clock_gettime@GLIBC_2.17'
collect2: error: ld returned 1 exit status
make[2]: *** [src/rfilter] Error 1
make[1]: *** [src/CMakeFiles/rfilter.dir/all] Error 2
make: *** [all] Error 2Thanks for your help.
Regards, Marcus
GWW
commented
2 years ago Hi Marcus,
Thanks for all the details and I am sorry this has been such a challenge. It's actually the part I find hardest about software development is making sure it's useable by others. I used git submodules to try to keep external packages up to date and avoid having users needing to install them. But I have also found that git doesn't include them with the package zip file. I'll look into explicitly including the external package code in the repository.
It seems like your conda environment may have outdated versions of gcc etc. I just tried creating a conda environment using mamba as below. I did it on a cluster that has outdated versions of everything. I use mamba because I have had better luck with installing compilers but conda should work too.
In particular, I think the paralell_hashmap library requires c++17 and some of its dependencies are missing from your version. If you have some issues upgrading the files I can look into building a binary that can be distributed.
mamba create -c conda-forge -c bioconda -n GWW_TEST gcc_linux-64 gxx_linux-64 gfortran_linux-64 hdf5 cmake
git clone --recurse-submodules https://github.com/GWW/scsnv.git
cd scsnv
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
makeThe conda package versions are:
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 1_gnu conda-forge
binutils_impl_linux-64 2.36.1 h193b22a_2 conda-forge
binutils_linux-64 2.36 hf3e587d_1 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.17.2 h7f98852_0 conda-forge
ca-certificates 2021.10.8 ha878542_0 conda-forge
cmake 3.21.3 h8897547_0 conda-forge
expat 2.4.1 h9c3ff4c_0 conda-forge
gcc_impl_linux-64 11.2.0 h82a94d6_11 conda-forge
gcc_linux-64 11.2.0 h39a9532_1 conda-forge
gfortran_impl_linux-64 11.2.0 h7a446d4_11 conda-forge
gfortran_linux-64 11.2.0 h777b47f_1 conda-forge
gxx_impl_linux-64 11.2.0 h82a94d6_11 conda-forge
gxx_linux-64 11.2.0 hacbe6df_1 conda-forge
hdf5 1.12.1 nompi_h7f166f4_101 conda-forge
kernel-headers_linux-64 2.6.32 he073ed8_14 conda-forge
krb5 1.19.2 h48eae69_2 conda-forge
ld_impl_linux-64 2.36.1 hea4e1c9_2 conda-forge
libcurl 7.79.1 h494985f_1 conda-forge
libedit 3.1.20210714 h7f8727e_0
libev 4.33 h516909a_1 conda-forge
libgcc-devel_linux-64 11.2.0 h0952999_11 conda-forge
libgcc-ng 11.2.0 h1d223b6_11 conda-forge
libgfortran-ng 11.2.0 h69a702a_11 conda-forge
libgfortran5 11.2.0 h5c6108e_11 conda-forge
libgomp 11.2.0 h1d223b6_11 conda-forge
libnghttp2 1.43.0 ha19adfc_1 conda-forge
libsanitizer 11.2.0 he4da1e4_11 conda-forge
libssh2 1.10.0 ha35d2d1_2 conda-forge
libstdcxx-devel_linux-64 11.2.0 h0952999_11 conda-forge
libstdcxx-ng 11.2.0 he4da1e4_11 conda-forge
libuv 1.42.0 h7f98852_0 conda-forge
libzlib 1.2.11 h36c2ea0_1013 conda-forge
lz4-c 1.9.3 h9c3ff4c_1 conda-forge
ncurses 6.2 h58526e2_4 conda-forge
openssl 3.0.0 h7f98852_1 conda-forge
rhash 1.4.1 h7f98852_0 conda-forge
sysroot_linux-64 2.12 he073ed8_14 conda-forge
tk 8.6.11 h27826a3_1 conda-forge
xz 5.2.5 h516909a_1 conda-forge
zlib 1.2.11 h36c2ea0_1013 conda-forge
zstd 1.5.0 ha95c52a_0 conda-forgeHi GWW,
Thank you so much. This is more than helpful.
I created a new conda environment from your list and re-installed the package, it worked!
For others who might have difficulties installing, I'll attach the conda list for creating environment below (For the python install step, pacakges such as numpy and scipy will be needed, which are also in this env list).
Download the list, and use conda create -n scsnv -c conda-forge --file conda_env_scsnv.txt to make a new conda environment for scSNV. In addition, one might need to pip install 'cyvcf2' and 'NCLS' in the new environment, which will be needed too.
And then follow the installation step such as cmake, make, and python3 setup.py install.
Thank you GWW for helping me out. I'll try to run the pipeline next :)
Regards, Marcus
GWW
commented
2 years ago My pleasure. I am glad it's working. Please let me know if you have any other issues.
Hi. Thanks for creating this exciting tool.
I have some scRNAseq data and want to try it out. However I encountered some problems when installing the software.
Following the README instruction, I've gone though the 'cmake' step successfully, but failed at the 'make' step. The full command prompts are like this.
`-bash-4.2$ cmake -DCMAKE_BUILD_TYPE=Release .. -- The C compiler identification is GNU 4.8.5 -- The CXX compiler identification is GNU 9.1.0 -- Check for working C compiler: /bin/cc -- Check for working C compiler: /bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXX compiler: /software/gcc/9.1.0/bin/c++ -- Check for working CXX compiler: /software/gcc/9.1.0/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- HDF5: Using hdf5 compiler wrapper to determine C configuration -- HDF5: Using hdf5 compiler wrapper to determine CXX configuration -- Found HDF5: /software/hdf5/1.10.2/lib/libhdf5_cpp.so;/software/hdf5/1.10.2/lib/libhdf5.so;/usr/lib64/librt.so;/usr/lib64/libpthread.so;/software/hdf5/1.10.2/lib/libz.so;/usr/lib64/libdl.so;/usr/lib64/libm.so (found version "1.10.2") found components: C CXX HL HDF5 LIBRARY: /software/hdf5/1.10.2/lib/libhdf5_cpp.so;/software/hdf5/1.10.2/lib/libhdf5.so;/usr/lib64/librt.so;/usr/lib64/libpthread.so;/software/hdf5/1.10.2/lib/libz.so;/usr/lib64/libdl.so;/usr/lib64/libm.so HDF5 LIBRARY CXX: /software/hdf5/1.10.2/lib/libhdf5_cpp.so;/software/hdf5/1.10.2/lib/libhdf5.so;/usr/lib64/librt.so;/usr/lib64/libpthread.so;/software/hdf5/1.10.2/lib/libz.so;/usr/lib64/libdl.so;/usr/lib64/libm.so HDF5 INCLUDE: /software/hdf5/1.10.2/include BUILD TYPE: Release INSTALL PREFIX: /usr/local -- Configuring done -- Generating done -- Build files have been written to: /home/marcuslc/Tools/scsnv-master/build
-bash-4.2$ make Scanning dependencies of target libhts [ 2%] Generating ../../external/htslib/libhts.a autoreconf: 'configure.ac' or 'configure.in' is required make[2]: [../external/htslib/libhts.a] Error 1 make[1]: [src/CMakeFiles/libhts.dir/all] Error 2 make: *** [all] Error 2`
I've also tried using conda environment to reinstall autoreconf, cmake, gcc v9.1 or v11.2, but the same error still occured. I have little knowledge about C++ and also haven't found useful solution on google. Could you please give me some suggestions how I should solve this error?
Many thanks! Marcus