Make proper release and add LICENSE file for conda

[jfy133]

Hi Gabriel,

At the request of a collegue, I've trying making a conda recipe for Gargammel for bioconda.

I've been able to get it to compile (albiet without the ms2chromosome.py functionality) - and will ask my collegue to test it a bit more fully.

However, following the bioconda guidelines, they request that the tarball comes from a 'stable' URL with a version, and also that the tarball includes a LICENSE file (https://bioconda.github.io/contributor/guidelines.html#stable-url).

I was wondering if you would be willing to add the corresponding License file to the repository (as I see in your paper you release it under GPL - although not sure which version) , and make a proper release of gargammel here on github? Then I could submit the recipe to bioconda (if you're interested in that, of course).

Cheers,

[grenaud]

Hi James, I have since added a license file and I have created a release for this. Let me know if there's anything missing. By the way, I was having trouble with submitting to bioconda due to bamtools, how did you circumvent this dependency?

Thank you very much for your interest in your bug report!

Gabriel

[jfy133]

Hi Gabriel,

Thanks! Once I've got my colleague to check it actually works I'll finalize the recipe and try to submit it to bioconda.

For bamtools, I can't tell you much. The compilation itself worked out of the box for me with your own instructions. Do you remember what the error you had?

Cheers James

[jfy133]

Just to be clear, by compile I mean I don't get any errors and can get a help message without an error. I don't know the functionality of the tool is working (What my colleague will test)

[grenaud]

It Alex Pelzter (you might know him, he used to work with Johannes K) that commented that another repo of mine: https://github.com/grenaud/schmutzi did not work with bioconda. I think the issue was that bioconda did not like bamtools. Could you try to package it in bioconda if you have a bit of time and let me know.

[jfy133]

Ah ok. I'll have a look at the issue on Schmutzi. Unfortunately while gargammel could compile in a conda build test on my laptop OK, when I made the PR there is an issue where the CI environment doesn't pick up g++ correctly. I'm not sure why yet but will try to investigate and see if it's the same issue as Alex reported.

EDIT: Looks like the same issue... if Alex can't get it to work easily I likely won't be able to. I guess it's something to do with the 'custom' compiler variables not being passed down inside the make file? That you're compiling a dependency within a compilation (so to say), becomes very difficult unfortunately I think, at least with conda.

[grenaud]

Forgive my ignorance, what is PR?

[jfy133]

PR: Pull request

[grenaud]

Ok, it seems a general problem with bamtools, I am slowly removing it from all my programs and replacing it with samtools/htslib. bamtools is no longer being maintained it seems.

[jfy133]

Ah ok. Then I'll keep an eye on this repository, and can update the conda recipe accordingly when replaced. I'll close this issue as the release/LICENSE is updated. Thanks for that anyway!

[jfy133]

@grenaud looking a bit closer, it's not bamtools that is the issue but rather libgab. So bamtools does compile correctly, but libgab doesn't inherit the environment variable for g++ in this case it seems.

https://app.circleci.com/jobs/github/bioconda/bioconda-recipes/97057

13:31:29 BIOCONDA INFO (OUT) [100%] Built target bamtools_cmd
13:31:29 BIOCONDA INFO (OUT) make[2]: Leaving directory `$SRC_DIR/bamtools/build'
13:31:29 BIOCONDA INFO (OUT) make[1]: Leaving directory `$SRC_DIR/bamtools/build'
13:31:29 BIOCONDA INFO (OUT) rm -rf libgab/
13:31:29 BIOCONDA INFO (OUT) git clone --depth 1 --recursive https://github.com/grenaud/libgab.git
13:31:29 BIOCONDA INFO (OUT) Cloning into 'libgab'...
13:31:29 BIOCONDA INFO (OUT) remote: Enumerating objects: 35, done.
remote: Counting objects: 100% (35/35), done.nting objects:   2% (1/35)        
remote: Compressing objects: 100% (30/30), done.ssing objects:   3% (1/30)        
remote: Total 35 (delta 1), reused 23 (delta 1), pack-reused 05)   
Unpacking objects: 100% (35/35), done.cking objects:  74% (26/35)   
13:31:29 BIOCONDA INFO (OUT) make -C libgab
13:31:29 BIOCONDA INFO (OUT) Makefile:59: warning: overriding commands for target `-'make[1]: Entering directory `$SRC_DIR/libgab'
13:31:29 BIOCONDA INFO (OUT) 
13:31:29 BIOCONDA INFO (OUT) Makefile:55: warning: ignoring old commands for target `-'
13:31:29 BIOCONDA INFO (OUT) bamtools found
13:31:29 BIOCONDA INFO (OUT) Makefile:59: warning: overriding commands for target `-'
13:31:29 BIOCONDA INFO (OUT) Makefile:55: warning: ignoring old commands for target `-'
13:31:29 BIOCONDA INFO (OUT) make[2]: Entering directory `$SRC_DIR/libgab'
13:31:29 BIOCONDA INFO (OUT) g++  -Wall  -lm -O3 -I. -Igzstream/ -I$SRC_DIR/bamtools/src/ -I$SRC_DIR/bamtools/src/  -c ReconsReferenceBAM.cpp -o ReconsReferenceBAM.o
13:31:29 BIOCONDA INFO (OUT) make[2]: g++: Command not found
13:31:29 BIOCONDA INFO (OUT) make[2]: *** [ReconsReferenceBAM.o] Error 127
13:31:29 BIOCONDA INFO (OUT) make[2]: Leaving directory `$SRC_DIR/libgab'
13:31:29 BIOCONDA INFO (OUT) make[1]: *** [/opt/conda/conda-bld/gargammel_1583069300258/work/bamtools/src/-/opt/conda/conda-bld/gargammel_1583069300258/work/bamtools/src] Error 2
13:31:29 BIOCONDA INFO (OUT) make[1]: Leaving directory `$SRC_DIR/libgab'
13:31:29 BIOCONDA INFO (OUT) make: *** [libgab/utils.o] Error 2
13:31:31 BIOCONDA INFO (OUT) Traceback (most recent call last):
13:31:31 BIOCONDA INFO (OUT)   File "/opt/conda/bin/conda-build", line 11, in <module>
13:31:31 BIOCONDA INFO (OUT)     sys.exit(main())
13:31:31 BIOCONDA INFO (OUT)   File "/opt/conda/lib/python3.7/site-packages/conda_build/cli/main_build.py", line 445, in main
13:31:31 BIOCONDA INFO (OUT)     execute(sys.argv[1:])
13:31:31 BIOCONDA INFO (OUT)   File "/opt/conda/lib/python3.7/site-packages/conda_build/cli/main_build.py", line 436, in execute
13:31:31 BIOCONDA INFO (OUT)     verify=args.verify, variants=args.variants)
13:31:31 BIOCONDA INFO (OUT)   File "/opt/conda/lib/python3.7/site-packages/conda_build/api.py", line 209, in build
13:31:31 BIOCONDA INFO (OUT)     notest=notest, need_source_download=need_source_download, variants=variants)
13:31:31 BIOCONDA INFO (OUT)   File "/opt/conda/lib/python3.7/site-packages/conda_build/build.py", line 2343, in build_tree
13:31:31 BIOCONDA INFO (OUT)     notest=notest,
13:31:31 BIOCONDA INFO (OUT)   File "/opt/conda/lib/python3.7/site-packages/conda_build/build.py", line 1491, in build
13:31:31 BIOCONDA INFO (OUT)     cwd=src_dir, stats=build_stats)
13:31:31 BIOCONDA INFO (OUT)   File "/opt/conda/lib/python3.7/site-packages/conda_build/utils.py", line 397, in check_call_env
13:31:31 BIOCONDA INFO (OUT)     return _func_defaulting_env_to_os_environ('call', *popenargs, **kwargs)
13:31:31 BIOCONDA INFO (OUT)   File "/opt/conda/lib/python3.7/site-packages/conda_build/utils.py", line 377, in _func_defaulting_env_to_os_environ
13:31:31 BIOCONDA INFO (OUT)     raise subprocess.CalledProcessError(proc.returncode, _args)
13:31:31 BIOCONDA INFO (OUT) subprocess.CalledProcessError: Command '['/bin/bash', '-e', '/opt/conda/conda-bld/gargammel_1583069300258/work/conda_build.sh']' returned non-zero exit status 2.
13:31:33 BIOCONDA ERROR COMMAND FAILED (exited with 1): docker run -t --net host --rm -v /tmp/tmpv_g65uvu/build_script.bash:/opt/build_script.bash -v /home/circleci/project/miniconda/conda-bld/:/opt/host-conda-bld -v /home/circleci/project/recipes/gargammel:/opt/recipe -e LANG=en_US.UTF-8 -e HOST_USER_ID=1001 bioconda/bioconda-utils-build-env:latest /bin/bash /opt/build_script.bash

13:31:33 BIOCONDA ERROR BUILD FAILED recipes/gargammel
13:31:33 BIOCONDA INFO (COMMAND) conda build purge
13:31:34 BIOCONDA ERROR BUILD SUMMARY: of 1 recipes, 1 failed and 0 were skipped. Details of recipes and environments follow.
13:31:34 BIOCONDA ERROR BUILD SUMMARY: FAILED recipe recipes/gargammel

Exited with code exit status 1
CircleCI received exit code 1

[grenaud]

g++: Command not found ?

What can be done to fix this?

On Mon, Mar 2, 2020 at 1:39 PM James A. Fellows Yates < notifications@github.com> wrote:

@grenaud https://github.com/grenaud looking a bit closer, it's not bamtools that is the issue but rather libgab. So bamtools does compile correctly

https://app.circleci.com/jobs/github/bioconda/bioconda-recipes/97057

13:31:29 BIOCONDA INFO (OUT) [100%] Built target bamtools_cmd 13:31:29 BIOCONDA INFO (OUT) make[2]: Leaving directory $SRC_DIR/bamtools/build' 13:31:29 BIOCONDA INFO (OUT) make[1]: Leaving directory$SRC_DIR/bamtools/build' 13:31:29 BIOCONDA INFO (OUT) rm -rf libgab/ 13:31:29 BIOCONDA INFO (OUT) git clone --depth 1 --recursive https://github.com/grenaud/libgab.git 13:31:29 BIOCONDA INFO (OUT) Cloning into 'libgab'... 13:31:29 BIOCONDA INFO (OUT) remote: Enumerating objects: 35, done. remote: Counting objects: 100% (35/35), done.nting objects: 2% (1/35) remote: Compressing objects: 100% (30/30), done.ssing objects: 3% (1/30) remote: Total 35 (delta 1), reused 23 (delta 1), pack-reused 05) Unpacking objects: 100% (35/35), done.cking objects: 74% (26/35) 13:31:29 BIOCONDA INFO (OUT) make -C libgab 13:31:29 BIOCONDA INFO (OUT) Makefile:59: warning: overriding commands for target -'make[1]: Entering directory$SRC_DIR/libgab' 13:31:29 BIOCONDA INFO (OUT) 13:31:29 BIOCONDA INFO (OUT) Makefile:55: warning: ignoring old commands for target -' 13:31:29 BIOCONDA INFO (OUT) bamtools found 13:31:29 BIOCONDA INFO (OUT) Makefile:59: warning: overriding commands for target-' 13:31:29 BIOCONDA INFO (OUT) Makefile:55: warning: ignoring old commands for target -' 13:31:29 BIOCONDA INFO (OUT) make[2]: Entering directory$SRC_DIR/libgab' 13:31:29 BIOCONDA INFO (OUT) g++ -Wall -lm -O3 -I. -Igzstream/ -I$SRC_DIR/bamtools/src/ -I$SRC_DIR/bamtools/src/ -c ReconsReferenceBAM.cpp -o ReconsReferenceBAM.o 13:31:29 BIOCONDA INFO (OUT) make[2]: g++: Command not found 13:31:29 BIOCONDA INFO (OUT) make[2]: [ReconsReferenceBAM.o] Error 127 13:31:29 BIOCONDA INFO (OUT) make[2]: Leaving directory `$SRC_DIR/libgab' 13:31:29 BIOCONDA INFO (OUT) make[1]: [/opt/conda/conda-bld/gargammel_1583069300258/work/bamtools/src/-/opt/conda/conda-bld/gargammel_1583069300258/work/bamtools/src] Error 2 13:31:29 BIOCONDA INFO (OUT) make[1]: Leaving directory `$SRC_DIR/libgab' 13:31:29 BIOCONDA INFO (OUT) make: [libgab/utils.o] Error 2 13:31:31 BIOCONDA INFO (OUT) Traceback (most recent call last): 13:31:31 BIOCONDA INFO (OUT) File "/opt/conda/bin/conda-build", line 11, in 13:31:31 BIOCONDA INFO (OUT) sys.exit(main()) 13:31:31 BIOCONDA INFO (OUT) File "/opt/conda/lib/python3.7/site-packages/conda_build/cli/main_build.py", line 445, in main 13:31:31 BIOCONDA INFO (OUT) execute(sys.argv[1:]) 13:31:31 BIOCONDA INFO (OUT) File "/opt/conda/lib/python3.7/site-packages/conda_build/cli/main_build.py", line 436, in execute 13:31:31 BIOCONDA INFO (OUT) verify=args.verify, variants=args.variants) 13:31:31 BIOCONDA INFO (OUT) File "/opt/conda/lib/python3.7/site-packages/conda_build/api.py", line 209, in build 13:31:31 BIOCONDA INFO (OUT) notest=notest, need_source_download=need_source_download, variants=variants) 13:31:31 BIOCONDA INFO (OUT) File "/opt/conda/lib/python3.7/site-packages/conda_build/build.py", line 2343, in build_tree 13:31:31 BIOCONDA INFO (OUT) notest=notest, 13:31:31 BIOCONDA INFO (OUT) File "/opt/conda/lib/python3.7/site-packages/conda_build/build.py", line 1491, in build 13:31:31 BIOCONDA INFO (OUT) cwd=src_dir, stats=build_stats) 13:31:31 BIOCONDA INFO (OUT) File "/opt/conda/lib/python3.7/site-packages/conda_build/utils.py", line 397, in check_call_env 13:31:31 BIOCONDA INFO (OUT) return _func_defaulting_env_to_os_environ('call', popenargs, kwargs) 13:31:31 BIOCONDA INFO (OUT) File "/opt/conda/lib/python3.7/site-packages/conda_build/utils.py", line 377, in _func_defaulting_env_to_os_environ 13:31:31 BIOCONDA INFO (OUT) raise subprocess.CalledProcessError(proc.returncode, _args) 13:31:31 BIOCONDA INFO (OUT) subprocess.CalledProcessError: Command '['/bin/bash', '-e', '/opt/conda/conda-bld/gargammel_1583069300258/work/conda_build.sh']' returned non-zero exit status 2. 13:31:33 BIOCONDA ERROR COMMAND FAILED (exited with 1): docker run -t --net host --rm -v /tmp/tmpv_g65uvu/build_script.bash:/opt/build_script.bash -v /home/circleci/project/miniconda/conda-bld/:/opt/host-conda-bld -v /home/circleci/project/recipes/gargammel:/opt/recipe -e LANG=en_US.UTF-8 -e HOST_USER_ID=1001 bioconda/bioconda-utils-build-env:latest /bin/bash /opt/build_script.bash

13:31:33 BIOCONDA ERROR BUILD FAILED recipes/gargammel 13:31:33 BIOCONDA INFO (COMMAND) conda build purge 13:31:34 BIOCONDA ERROR BUILD SUMMARY: of 1 recipes, 1 failed and 0 were skipped. Details of recipes and environments follow. 13:31:34 BIOCONDA ERROR BUILD SUMMARY: FAILED recipe recipes/gargammel

Exited with code exit status 1 CircleCI received exit code 1

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/grenaud/gargammel/issues/7?email_source=notifications&email_token=AAQRNI2RXC5VKI7BYKX5F3DRFOSJPA5CNFSM4K5J52RKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENPEXGY#issuecomment-593382299, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAQRNI4UIXDWTT4M6A57DZTRFOSJPANCNFSM4K5J52RA .

[jfy133]

https://bioconda.github.io/contributor/troubleshooting.html#g-or-gcc-not-found

This is what the bioconda teams suggest for top level compilation (i.e. not stuff being compiled within the main compilation). This would therefore require modifying the cloned make file within libgab compilation to something like make CC=$CXX, during the conda build process. I don't have experience with make though.

Edit, I think the problem could be the hardcoding here:

https://github.com/grenaud/libgab/blob/73968766daf3d0815737ed63806791b1cad5c879/Makefile#L22

Which could be overrided when compiling with gargammel here? https://github.com/grenaud/gargammel/blob/5314b666c753affd0fb0fee988fd2917a086a23d/Makefile#L25 to adding 'make -C libgab CC=$CXX` for bioconda?

[grenaud]

I will mark this email as unread and get back to you :-) Thank you for finding this so quickly!

On Mon, Mar 2, 2020 at 1:44 PM James A. Fellows Yates notifications@github.com wrote:

https://bioconda.github.io/contributor/troubleshooting.html#g-or-gcc-not-found

This is what the bioconda teams suggest for top level compilation (i.e. not stuff being compiled within the main compilation). This would therefore require modifying the cloned make file, during the conda build process. I don't have experience with make though.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[grenaud]

Hi James, Sorry for the delay, I had grants to write and got sick on top of that.

I am not sure I fully understand their recommendation but I did the following:

• CXXFLAGS and LDLIBS are now with an append "+=" rather than a "="
• I pass the compiler variable to a subdirectory that I compile as such "CXX=${CXX}"

In the main Makefile for gargammel, I have added "CC=${CXX}" to each subdirectory: make CC=${CXX} -C libgab make CC=${CXX} -C src

Are these changes sufficient? Does it work now?

[jfy133]

Hey Gabriel,

No problem! It will also be a while until I get back to this as COVID childcare means I'm barely getting through my essential tasks. Will let you know once I've tried it out. However I don't see any changes in the repository? Or do you mean for myself to modify accordingly and test?

[grenaud]

Hi James, Don't worry about it, we will all take a collective productivity dive for 2020 :-)

I modified the Makefile in libgab, I thought that was sufficient, should I set "make CXX=${CXX}" for every sub makefile? Because currently, the Makefile from the top directory calls the makefile in src/ without anything. Should I change it to "make CXX=${CXX}"?

cheers,

Gabriel

On Fri, May 1, 2020 at 7:47 AM James A. Fellows Yates < notifications@github.com> wrote:

Hey Gabriel,

No problem! It will also be a while until I get back to this as COVID childcare means I'm barely getting through my essential tasks. Will let you know once I've tried it out. However I don't see any changes in the repository? Or do you mean for myself to modify accordingly and test?

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/grenaud/gargammel/issues/7#issuecomment-622259639, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAQRNI4IILA3LRCKKD657GLRPJO67ANCNFSM4K5J52RA .

[jfy133]

Just to let you know: had a chance to attempt to re-compile, and got a bit further:

https://app.circleci.com/pipelines/github/bioconda/bioconda-recipes/25325/workflows/eb823132-d4a1-453f-a002-96833e379e13/jobs/106789

18:03:13 BIOCONDA INFO (OUT) make[2]: Entering directory `$SRC_DIR/libgab'

18:03:13 BIOCONDA INFO (OUT) $BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-c++ -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem $PREFIX/include -fdebug-prefix-map=$SRC_DIR=/usr/local/src/conda/gargammel-1.1.2 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -Wall  -lm -O3 -I. -Igzstream/ -I$SRC_DIR/bamtools/src/ -I$SRC_DIR/bamtools/src/  -Wall  -lm -O3 -I. -Igzstream/ -I$SRC_DIR/bamtools/src/ -I$SRC_DIR/bamtools/src/  -c utils.cpp -o utils.o

18:03:14 BIOCONDA INFO (OUT) make CXX=$BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-c++ -C gzstream/

18:03:14 BIOCONDA INFO (OUT) make[3]: Entering directory `$SRC_DIR/libgab/gzstream'

18:03:14 BIOCONDA INFO (OUT) $BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-c++ -I. -O -c -o gzstream.o gzstream.C

18:03:14 BIOCONDA INFO (OUT) In file included from gzstream.C:29:0:

18:03:14 BIOCONDA INFO (OUT) ./gzstream.h:35:10: fatal error: zlib.h: No such file or directory

18:03:14 BIOCONDA INFO (OUT)  #include <zlib.h>

18:03:14 BIOCONDA INFO (OUT)           ^~~~~~~~

18:03:14 BIOCONDA INFO (OUT) compilation terminated.

18:03:14 BIOCONDA INFO (OUT) make[3]: *** [gzstream.o] Error 1

18:03:14 BIOCONDA INFO (OUT) make[3]: Leaving directory `$SRC_DIR/libgab/gzstream'

18:03:14 BIOCONDA INFO (OUT) make[2]: *** [gzstream/libgzstream.a] Error 2

18:03:14 BIOCONDA INFO (OUT) make[2]: Leaving directory `$SRC_DIR/libgab'

18:03:14 BIOCONDA INFO (OUT) make[1]: *** [/opt/conda/conda-bld/gargammel_1588528794116/work/bamtools/src/-/opt/conda/conda-bld/gargammel_1588528794116/work/bamtools/src] Error 2

18:03:14 BIOCONDA INFO (OUT) make[1]: Leaving directory `$SRC_DIR/libgab'

18:03:14 BIOCONDA INFO (OUT) make: *** [libgab/utils.o] Error 2

I will try this out: https://bioconda.github.io/contributor/troubleshooting.html#zlib

[jfy133]

Ok! I think we are now one step away from everything compling!

https://app.circleci.com/pipelines/github/bioconda/bioconda-recipes/25329/workflows/e9a3f839-b906-4e5a-86fa-5c0414b20b9c/jobs/106798

Maybe you can help here? Looks like it's fragSim failing, with the -lz flag, which if I understand correctly comes from complication of libbamtools? Could it be the LDFLAGs thing needs to be applied that there too? (If not already). Could that be right?

20:11:16 BIOCONDA INFO (OUT) make[1]: Leaving directory `$SRC_DIR/libgab'

20:11:16 BIOCONDA INFO (OUT) make -C src CXX=$BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-c++

20:11:16 BIOCONDA INFO (OUT) make[1]: Entering directory `$SRC_DIR/src'

20:11:16 BIOCONDA INFO (OUT) $BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-c++ -std=c++0x -Wall -lm -O3 -lz -I$SRC_DIR/libgab -I$SRC_DIR/libgab/gzstream/ -I$SRC_DIR/bamtools/src/  -c fragSim.cpp

20:11:20 BIOCONDA INFO (OUT) $BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-c++ -o fragSim fragSim.o  $SRC_DIR/libgab/FastQObj.o  $SRC_DIR/libgab/FastQParser.o $SRC_DIR/libgab/PutProgramInHeader.o $SRC_DIR/bamtools/build/src/api/libbamtools.a  $SRC_DIR/libgab/utils.o $SRC_DIR/libgab/gzstream/libgzstream.a -lm -lz

20:11:20 BIOCONDA INFO (OUT) /opt/conda/conda-bld/gargammel_1588536473007/_build_env/bin/../lib/gcc/x86_64-conda_cos6-linux-gnu/7.3.0/../../../../x86_64-conda_cos6-linux-gnu/bin/ld: cannot find -lz

20:11:20 BIOCONDA INFO (OUT) collect2: error: ld returned 1 exit status

20:11:20 BIOCONDA INFO (OUT) make[1]: *** [fragSim] Error 1

[grenaud]

Hello! I was reading the following: https://stackoverflow.com/questions/51017495/zlib-not-found-when-installing-shortread https://gitmemory.com/marcelm

As far as I can tell, it seems a problem with the LDFLAGS. I have added a += instead of a = it seems that conda already said that variable and I seem to be overriding it. I committed a code to github, could you try to give it another spin?

Cheers!

On Sun, May 3, 2020 at 10:18 PM James A. Fellows Yates notifications@github.com wrote:

Ok! I think we are now one step away from everything compling!

https://app.circleci.com/pipelines/github/bioconda/bioconda-recipes/25329/workflows/e9a3f839-b906-4e5a-86fa-5c0414b20b9c/jobs/106798

Maybe you can help here? Looks like it's fragSim failing, with the -lz flag, which if I understand correctly comes from complication of libbamtools? Could it be the LDFLAGs thing needs to be applied that there too? (If not already). Could that be right?

20:11:16 BIOCONDA INFO (OUT) make[1]: Leaving directory `$SRC_DIR/libgab'

20:11:16 BIOCONDA INFO (OUT) make -C src CXX=$BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-c++

20:11:16 BIOCONDA INFO (OUT) make[1]: Entering directory `$SRC_DIR/src'

20:11:16 BIOCONDA INFO (OUT) $BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-c++ -std=c++0x -Wall -lm -O3 -lz -I$SRC_DIR/libgab -I$SRC_DIR/libgab/gzstream/ -I$SRC_DIR/bamtools/src/ -c fragSim.cpp

20:11:20 BIOCONDA INFO (OUT) $BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-c++ -o fragSim fragSim.o $SRC_DIR/libgab/FastQObj.o $SRC_DIR/libgab/FastQParser.o $SRC_DIR/libgab/PutProgramInHeader.o $SRC_DIR/bamtools/build/src/api/libbamtools.a $SRC_DIR/libgab/utils.o $SRC_DIR/libgab/gzstream/libgzstream.a -lm -lz

20:11:20 BIOCONDA INFO (OUT) /opt/conda/conda-bld/gargammel_1588536473007/_build_env/bin/../lib/gcc/x86_64-conda_cos6-linux-gnu/7.3.0/../../../../x86_64-conda_cos6-linux-gnu/bin/ld: cannot find -lz

20:11:20 BIOCONDA INFO (OUT) collect2: error: ld returned 1 exit status

20:11:20 BIOCONDA INFO (OUT) make[1]: *** [fragSim] Error 1

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or unsubscribe.

[jfy133]

:tada: When I copy your edits, the (almost) package compiles! It crashes but at I think at a bioconda error (I think at the .dynamic) bit.

I'll try and ask the bioconda team some time this week if they can help, and if I solve it I'll let you know and you can make the proper release (so I don't have to do the patching in the bioconda build.sh script.

19:50:54 BIOCONDA INFO (OUT) WARNING :: `patchelf --print-rpath` failed for /opt/conda/conda-bld/gargammel_1588621571732/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/share/gargammel-1.1.2-0/src/misincorporation2prof.o, will proceed with LIEF (was None)

19:50:54 BIOCONDA INFO (OUT) patchelf: cannot find section '.dynamic'. The input file is most likely statically linked

19:50:54 BIOCONDA INFO (OUT) WARNING :: `patchelf --print-rpath` failed for /opt/conda/conda-bld/gargammel_1588621571732/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/share/gargammel-1.1.2-0/bamtools/tools/Linux/clang-format, will proceed with LIEF (was None)

19:50:54 BIOCONDA INFO (OUT) Traceback (most recent call last):

19:50:54 BIOCONDA INFO (OUT)   File "/opt/conda/bin/conda-build", line 11, in <module>

19:50:54 BIOCONDA INFO (OUT)     sys.exit(main())

https://app.circleci.com/pipelines/github/bioconda/bioconda-recipes/25400/workflows/fc53711f-9471-40ef-af28-47395d67d589/jobs/106970/steps

It might be due to the internally compiled binaries: https://github.com/bioconda/bioconda-recipes/pull/14233/files

EDIT: I just realised I never shared with you the (draft) conda recipe: https://github.com/bioconda/bioconda-recipes/pull/20645

[jfy133]

Hi Gabriel,

We now have a compiling recipe thanks to some help from the community! https://github.com/bioconda/bioconda-recipes/pull/20645. It was indeed the dependencies were not transferred properly.

I'm now trying to build the recipe locally and check all the commands work.

However, when I went to test it properly, I realised half the the examples here on your github required the additional python scripts, which I'm now having a few problems with executing.

Could you list what version of python the scripts were running in, and also what additional libraries the python scripts require?

Also I'm not sure I will be able to get the bacterialex working in the recipe, unless you inbuild the download into the perl script (for example), because there isn't a clean way to specify the makefile that it couldn't conflict with others.

Once Iv'e got (most) of your test commands working, I'll get the conda package integrated into bioconda, you can make a new release with the changes you've made here, and I can update the recipe patches I currently do in the conda build.sh

Cheers,

[grenaud]

Hi James, This is great! Thank you for letting me know! Feel free to close the issue whenever everything works out.

Gabriel

On Wed, May 6, 2020 at 9:59 AM James A. Fellows Yates notifications@github.com wrote:

Hi Gabriel,

We now have a compiling recipe thanks to some help from the community! bioconda/bioconda-recipes#20645. It was indeed the dependencies were not transferred properly.

I'm now trying to build the recipe locally and check all the commands work.

However, when I went to test it properly, I realised half the the examples here on your github required the additional python scripts, which I'm now having a few problems with executing.

Could you list what version of python the scripts were running in, and also what additional libraries the python scripts require?

Also I'm not sure I will be able to get the bacterialex working in the recipe, unless you inbuild the download into the perl script (for example), because there isn't a clean way to specify the makefile that it couldn't conflict with others.

Once Iv'e got (most) of your test commands working, I'll get the conda package integrated into bioconda, you can make a new release with the changes you've made here, and I can update the recipe patches I currently do in the conda build.sh

Cheers,

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or unsubscribe.

[jfy133]

Will do! Just to recap my questins from before:

Could you list what version of python the scripts were running in, and also what additional libraries the python scripts require?

Also I'm not sure I will be able to get the bacterialex working in the recipe, unless you inbuild the download into the perl script (for example), because there isn't a clean way to specify the makefile that it couldn't conflict with others. Do you think that would be OK?

[jfy133]

@grenaud - @marcelm is kindly helping me with with getting the gargammel bioconda scripts running.

He's making some modifications to the python scripts (at least being patched) so they work with Python 3, given 2 is dead.

I'm posting this here so Marcel can track when you reply with the library versions for the python scripts, and also if you have any feeling about the bacterialex stuff.

[grenaud]

Hi James, The only Python used is: ./src/microbial_fetcher/src/select_random_speciesftp.py Is a member we coded it using python 2.7 Requires the following imports which should be pretty standard: import fileinput import sys import re import os import gzip import logging from random import randint

Are we required to build bacterialex? It's more for users as an example. It is not software per se.

Gabriel

On Wed, May 6, 2020 at 4:43 PM James A. Fellows Yates < notifications@github.com> wrote:

Will do! Just to recap my questins from before:

Could you list what version of python the scripts were running in, and also what additional libraries the python scripts require?

Also I'm not sure I will be able to get the bacterialex working in the recipe, unless you inbuild the download into the perl script (for example), because there isn't a clean way to specify the makefile that it couldn't conflict with others. Do you think that would be OK?

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/grenaud/gargammel/issues/7#issuecomment-624691307, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAQRNIZAPI7AIIMRVNEJ7S3RQFZRNANCNFSM4K5J52RA .

[jfy133]

For bacterialex - fair enough, we can indeed make that easier. I was just trying to follow as much functionality described as possible on the github README (I've not actually used gargammel myself yet), so I user could still actually run what is described there even if they don't have to compile themselves.

For Python, Ok, but I'm confused as you also have in the repository ms2chromosomes.py and msprime_chromosomes.py , the first one which you specify being a 'requirement'. Am I misunderstanding here?

[grenaud]

You can ignore ms2chromosome.py as being a requirement. msprime solved the problem of producing segregating sites on a chromosome level.

On Thu, May 7, 2020 at 1:09 PM James A. Fellows Yates < notifications@github.com> wrote:

For bacterialex - fair enough, we can indeed make that easier. I was just trying to follow as much functionality described as possible on the github README (I've not actually used gargammel myself yet), so I user could still actually run what is described there even if they don't have to compile themselves.

For Python, Ok, but I'm confused as you also have in the repository ms2chromosomes.py and msprime_chromosomes.py , the first one which you specify being a 'requirement'. Am I misunderstanding here?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/grenaud/gargammel/issues/7#issuecomment-625188188, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAQRNI4BOCHPTJWPCOQDKZTRQKJFNANCNFSM4K5J52RA .

[jfy133]

Ah ok. Then I will ignore both ms2chromosome and msprime python scripts. We are good to go. I've pushed the last commit rolling back trying to add those, so once tests pass we will integrate into bioconda :+1:

[grenaud]

Dear James, thank you very much for your hard work! Can we close this issue? Thanks again!

[jfy133]

Oops yes! Amongst all the other PRs etc. I forgot