Unable to install optimotu

[LukeLikesDirt]

Hi brendaf

I am very keen to try out your new clustering approach. However, I cannot install your package from GitHub and it also fails to install when I try to build it locally. I am on a Mac with an M2 chip and running MacOS Ventura 13.5. Do you know if anyone else is having issues installing your package? Below is the output when attempting to build the package locally.

Cheers Luke

── R CMD build ─────────────────────────────────────────────────────────────────────────────────────────────────────────── ✔ checking for file ‘/private/var/folders/g3/jfhf6nxx2ybfzplbmt_03y0r0000gn/T/RtmpvfCEE0/remotes8b14e6d6312/optimotu/DESCRIPTION’ ... ─ preparing ‘optimotu’: ✔ checking DESCRIPTION meta-information ─ cleaning src ─ running ‘cleanup’ ─ checking for LF line-endings in source and make files and shell scripts ─ checking for empty or unneeded directories ─ building ‘optimotu_0.7.0.tar.gz’

• installing source package ‘optimotu’ ... using staged installation ls: SneakySnake/*.c: No such file or directory ls: WFA2-lib/*/.c: No such file or directory configure: creating ./config.status config.status: creating src/Makevars libs using C compiler: ‘Apple clang version 14.0.3 (clang-1403.0.22.14.1)’ using C++ compiler: ‘Apple clang version 14.0.3 (clang-1403.0.22.14.1)’ using C++14 using SDK: ‘MacOSX13.3.sdk’ clang++ -arch arm64 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DOPTIMOTU_R -I./WFA2-lib -I./SneakySnake -I./edlib -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/Rcpp/include' -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppParallel/include' -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppThread/include' -I/opt/R/arm64/include -fPIC -falign-functions=64 -Wall -g -O2 -c AlignClusterWorker.cpp -o AlignClusterWorker.o clang++ -arch arm64 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DOPTIMOTU_R -I./WFA2-lib -I./SneakySnake -I./edlib -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/Rcpp/include' -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppParallel/include' -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppThread/include' -I/opt/R/arm64/include -fPIC -falign-functions=64 -Wall -g -O2 -c AllPairGenerator.cpp -o AllPairGenerator.o In file included from AllPairGenerator.cpp:1: In file included from ./AllPairGenerator.h:4: In file included from ./PairGenerator.h:4: In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/memory:860: In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/memory/shared_ptr.h:28: /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__memory/unique_ptr.h:48:5: warning: delete called on non-final 'AllPairGenerator' that has virtual functions but non-virtual destructor [-Wdelete-non-abstract-non-virtual-dtor] delete ptr; ^ /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/memory/unique_ptr.h:305:7: note: in instantiation of member function 'std::default_delete::operator()' requested here ptr_.second()(tmp); ^ /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/memory/unique_ptr.h:259:19: note: in instantiation of member function 'std::unique_ptr::reset' requested here ~unique_ptr() { reset(); } ^ AllPairGenerator.cpp:7:10: note: in instantiation of member function 'std::unique_ptr::~unique_ptr' requested here return std::make_unique(begin, end); ^ In file included from AllPairGenerator.cpp:1: In file included from ./AllPairGenerator.h:4: In file included from ./PairGenerator.h:4: In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/memory:860: In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/memory/shared_ptr.h:28: /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__memory/unique_ptr.h:48:5: warning: delete called on 'PairGenerator' that is abstract but has non-virtual destructor [-Wdelete-abstract-non-virtual-dtor] delete ptr; ^ /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/memory/unique_ptr.h:305:7: note: in instantiation of member function 'std::default_delete::operator()' requested here ptr_.second()(tmp); ^ /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/memory/unique_ptr.h:259:19: note: in instantiation of member function 'std::unique_ptr::reset' requested here ~unique_ptr() { reset(); } ^ AllPairGenerator.cpp:7:10: note: in instantiation of member function 'std::unique_ptr::~unique_ptr' requested here return std::make_unique(begin, end); ^ 2 warnings generated. clang++ -arch arm64 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DOPTIMOTU_R -I./WFA2-lib -I./SneakySnake -I./edlib -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/Rcpp/include' -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppParallel/include' -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppThread/include' -I/opt/R/arm64/include -fPIC -falign-functions=64 -Wall -g -O2 -c ClusterAlgorithmFactory.cpp -o ClusterAlgorithmFactory.o clang++ -arch arm64 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DOPTIMOTU_R -I./WFA2-lib -I./SneakySnake -I./edlib -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/Rcpp/include' -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppParallel/include' -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppThread/include' -I/opt/R/arm64/include -fPIC -falign-functions=64 -Wall -g -O2 -c ClusterIndexedMatrix.cpp -o ClusterIndexedMatrix.o clang++ -arch arm64 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DOPTIMOTU_R -I./WFA2-lib -I./SneakySnake -I./edlib -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/Rcpp/include' -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppParallel/include' -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppThread/include' -I/opt/R/arm64/include -fPIC -falign-functions=64 -Wall -g -O2 -c ClusterMatrix.cpp -o ClusterMatrix.o clang++ -arch arm64 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DOPTIMOTU_R -I./WFA2-lib -I./SneakySnake -I./edlib -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/Rcpp/include' -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppParallel/include' -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppThread/include' -I/opt/R/arm64/include -fPIC -falign-functions=64 -Wall -g -O2 -c ClusterSLINK.cpp -o ClusterSLINK.o ClusterSLINK.cpp:143:10: warning: variable 'jp' set but not used [-Wunused-but-set-variable] j_t j, jp; ^ 1 warning generated. clang++ -arch arm64 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DOPTIMOTU_R -I./WFA2-lib -I./SneakySnake -I./edlib -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/Rcpp/include' -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppParallel/include' -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppThread/include' -I/opt/R/arm64/include -fPIC -falign-functions=64 -Wall -g -O2 -c ClusterTree.cpp -o ClusterTree.o clang++ -arch arm64 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DOPTIMOTU_R -I./WFA2-lib -I./SneakySnake -I./edlib -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/Rcpp/include' -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppParallel/include' -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppThread/include' -I/opt/R/arm64/include -fPIC -falign-functions=64 -Wall -g -O2 -c ClusterWorker.cpp -o ClusterWorker.o clang++ -arch arm64 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DOPTIMOTU_R -I./WFA2-lib -I./SneakySnake -I./edlib -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/Rcpp/include' -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppParallel/include' -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppThread/include' -I/opt/R/arm64/include -fPIC -falign-functions=64 -Wall -g -O2 -c DistanceConverter.cpp -o DistanceConverter.o clang++ -arch arm64 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DOPTIMOTU_R -I./WFA2-lib -I./SneakySnake -I./edlib -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/Rcpp/include' -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppParallel/include' -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppThread/include' -I/opt/R/arm64/include -fPIC -falign-functions=64 -Wall -g -O2 -c EdlibClusterWorker.cpp -o EdlibClusterWorker.o In file included from EdlibClusterWorker.cpp:3: ./pairwise_alignment.h:7:10: fatal error: 'bindings/cpp/WFAligner.hpp' file not found

  include <bindings/cpp/WFAligner.hpp>

   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~

  1 error generated. make: *** [EdlibClusterWorker.o] Error 1 ERROR: compilation failed for package ‘optimotu’

• removing ‘/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/optimotu’ Warning message: In i.p(...) : installation of package ‘/var/folders/g3/jfhf6nxx2ybfzplbmt_03y0r0000gn/T//RtmpvfCEE0/file8b16f835af7/optimotu_0.7.0.tar.gz’ had non-zero exit status

[brendanf]

Hi LukeLikesDirt, It looks like the submodules (In this case WPA2-lib) have not been downloaded. How are you initializing the github installation? (e.g., via renv::install, remotes::install_github, manually cloning the repository, etc. )

[brendanf]

Incidentally if you don't need the new goodies which are included in the latest development version (and which are not documented in the README at all) then installing 0.6.5 will get all of the functionality which is used in, e.g. GSSP, without the complication of any submodules.

[brendanf]

Looking in the automated build logs, it looks like install on MacOS is failing on compiling WPA2lib even when it is present. So maybe installing 0.6.5 will be the best option for you. That one installed successfully on all tested platforms.

[brendanf]

If you'd be interested in helping me test to resolve these issues sometime, that could be helpful, since I don't have a Mac to test on locally.

[LukeLikesDirt]

Hi brendanf

Thanks for your quick response! v0.6.5 installed with no issues.

When trying the installation of the latest version using "remotes::install_github("brendanf/optimotu)" the installation would freeze. So I cloned the repository and attempted the installation as follows:

devtools::build("optimotu") devtools::install_local("optimotu_0.7.0.tar.gz")

Happy to help and test when needed, just let me know.

Cheers Luke

[brendanf]

Ominous that remotes::install_github() freezes. But if you have manually cloned the repository, then you can fetch the submodules with git submodule update --init --recursive and try devtools::build() again.

I expect this will fail because clang tries to include the WFA2-lib VERSION file (containing only the text "v2.3") as a c++ header.

[brendanf]

Apparently this is because it thinks it is the c++20 standard library header (and it is not case sensitive?).

[brendanf]

I've pushed a potential fix in 96a194280

[brendanf]

After solving a few more errors because clang is more strict than gcc, 9bb6ce2 now seems to install on MacOS. (There are still warnings from R CMD check which cause the git workflow to return "fail", but the package probably works.)

[LukeLikesDirt]

Hi brendanf

Unfortunately, I am still unable to install v0.7.0, even when updating the submodules.

When runninggit submodule update --init --recursive after cloning, I get this error for "WFA2-lib.git", "edlib.git" and "SneakySnake.git":

git@github.com: Permission denied (publickey) fatal: Could not read from remote repository

I am still keen to use v0.6.5 but I am struggling to get USEARCH configured on my Mac. I need to run it via docker as the 32-bit version of USEARCH is not compatible with newer Macs. Because Docker is new to me I'm facing an uphill battle. Have you considered implementing VSEARCH to circumvent USEARCH compatibility issues?

Cheers Luke

[brendanf]

This is probably because you don't have SSL configured for git (as most people don't, I imagine). I've changed the submodule addresses to use https instead in e65f78e.

For implementing VSEARCH -- if you mean using VSEARCH as an external distance matrix generator, then no, because VSEARCH has not implemented the --distmx command. If you mean implementing the USEARCH/VSEARCH algorithm internally in optimotu, then yes, that is in my plans. The existing options in 7.0.0 are strictly superior to that in terms of results, but are also quite a bit slower.

[LukeLikesDirt]

Hi brendaf

Thanks, that worked a treat, despite a bunch of warning messages.

In case anyone else has the same issue, here a recap of how I installed the latest version:

Clone optimotu

git clone https://github.com/brendanf/optimotu.git

Update the submodules

cd optimotu git submodule update --init --recursive

Install optimotu in R

devtools::build("optimotu") devtools::install_local("optimotu_0.7.0.tar.gz")

[LukeLikesDirt]

Hi brendaf

I want to create an environment on an HPC cluster to run optimotu. When installing directly from GitHub I run into similar issues as before so I am cloning the repository and installing locally as above. However, there are some issues and the installation fails. I have attached the slurm output with details. Could you please help me troubleshoot this? create_env.sh.328315.out.txt

[brendanf]

If you are making a conda environment anyway, then the easiest thing to do is probably to install from conda. I see that I have only built up to v0.6.5, I'll give it a shot now with 0.7.0.

[brendanf]

Which version of R are you using?

[brendanf]

I'll give it a shot now with 0.7.0.

This proved not to work. I'm learning so many ways in which my compiler is overly forgiving!

[brendanf]

OK, 0.7.0 is now available from my personal conda channel, with builds for R 4.2 and 4.3. If you want a different version, let me know.

47bbfaf should also have fixed the particular error you got, so your create_env.sh may work now.

[LukeLikesDirt]

Hi brendanf

Thanks for your help!

I am using R 4.3.

I didn't realise optimotu was available on conda. I can now successfully create environments on the cluster using both installation methods, via conda and GitHub.

Cheers Luke