Closed RichardCorbett closed 4 years ago
RichardCorbett
commented
4 years ago Looks like this could be related to https://github.com/heathsc/gemBS/issues/55
which found a fix for a possibly related bcftools_plugins problem, though the testing was done on the code, not through the container. The last release (3.3.0) predates the results of issue 55.
heathsc
commented
4 years ago Thanks for the report - I will look into this. bcftools and samtools are included in the container so normally the pipeline should use the ‘internal’ container versions rather than trying to find an installation on the host. I will see if I can replicate the error.
Thanks, Simon
On 17 Oct 2019, at 19:48, Richard Corbett notifications@github.com wrote:
Looks like this could be related to
55 https://github.com/heathsc/gemBS/issues/55
which found a fix for a possibly related bcftools_plugins problem, though the testing was done on the code, not through the container. The last release (3.3.0) predates the results of issue 55.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/heathsc/gemBS/issues/63?email_source=notifications&email_token=AAY4653DJNC5GQKROSMNKMTQPCQQXA5CNFSM4JBOF5B2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBQ6SGA#issuecomment-543287576, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAY4653A3HQTKBPMVC3FDPDQPCQQXANCNFSM4JBOF5BQ.
RichardCorbett
commented
4 years ago Thanks @heathsc ,
In case it helps, the container does seem to have access to bcftools and samtools binaries during the map and call steps which appear to work successfully. Those are at these locations in the container:
Singularity gembs_latest.sif:/projects/rcorbettprj2/gemBS> ls -ltr /usr/local/lib/python3.5/dist-packages/gemBS/bin/
total 19814
-rwxr-xr-x 1 root root 2927952 Oct 11 22:27 tabix
-rwxr-xr-x 1 root root 5307080 Oct 11 22:27 samtools
-rwxr-xr-x 1 root root 2859000 Oct 11 22:27 htsfile
-rwxr-xr-x 1 root root 2438192 Oct 11 22:27 bgzip
-rwxr-xr-x 1 root root 6755664 Oct 11 22:27 bcftools
Also, I do see some bcftools plugin objects at this location:
ls -la /usr/local/lib/python3.5/dist-packages/gemBS/libexec/bcftools
which seems to be correct in the code: https://www.bcgsc.ca/wiki/display/HL/WGBS-seq+Data+Analysis#WGBS-seqDataAnalysis-gembs
heathsc
commented
4 years ago And you have no other installation of samtools / bcftools on the host machine?
Simon
On 17 Oct 2019, at 20:02, Richard Corbett notifications@github.com wrote:
Thanks @heathsc https://github.com/heathsc ,
In case it helps, the container does seem to have access to bcftools and samtools binaries during the map and call steps which appear to work successfully. Those are at these locations in the container:
Singularity gembs_latest.sif:/projects/rcorbettprj2/gemBS> ls -ltr /usr/local/lib/python3.5/dist-packages/gemBS/bin/ total 19814 -rwxr-xr-x 1 root root 2927952 Oct 11 22:27 tabix -rwxr-xr-x 1 root root 5307080 Oct 11 22:27 samtools -rwxr-xr-x 1 root root 2859000 Oct 11 22:27 htsfile -rwxr-xr-x 1 root root 2438192 Oct 11 22:27 bgzip -rwxr-xr-x 1 root root 6755664 Oct 11 22:27 bcftools
Also, I do see some bcftools plugin objects at this location: ls -la /usr/local/lib/python3.5/dist-packages/gemBS/libexec/bcftools
which seems to be correct in the code: https://www.bcgsc.ca/wiki/display/HL/WGBS-seq+Data+Analysis#WGBS-seqDataAnalysis-gembs https://www.bcgsc.ca/wiki/display/HL/WGBS-seq+Data+Analysis#WGBS-seqDataAnalysis-gembs — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/heathsc/gemBS/issues/63?email_source=notifications&email_token=AAY465ZY3JRUPHEIVTMAM7TQPCSEBA5CNFSM4JBOF5B2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBQ73TI#issuecomment-543292877, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAY4656MTF3XGKCKGHEL6ZDQPCSEBANCNFSM4JBOF5BQ.
RichardCorbett
commented
4 years ago Not locally on the machine, but I do have a samtools in my path generally. When I use the container I see the following:
# gphost14 /projects/rcorbettprj2/gemBS $
singularity shell ./gembs_latest.sif
Singularity gembs_latest.sif:/projects/rcorbettprj2/gemBS> which samtools #returns nothing
Singularity gembs_latest.sif:/projects/rcorbettprj2/gemBS> which bcftools #returns nothing
Singularity gembs_latest.sif:/projects/rcorbettprj2/gemBS> whereis samtools
samtools:
Singularity gembs_latest.sif:/projects/rcorbettprj2/gemBS> whereis bcftools
bcftools:Now I’ve had the time to look at the earlier mails (!) It looks like it is finding the executables but for some reason the BCFTOOLS_PLUGIN variable is not being set. I don’t think this has anything to do with the problem, but what is your login shell ?
Cheers, Simon
On 17 Oct 2019, at 23:03, Richard Corbett notifications@github.com wrote:
Not locally on the machine, but I do have a samtools in my path generally. When I use the container I see the following:
gphost14 /projects/rcorbettprj2/gemBS $
singularity shell ./gembs_latest.sif Singularity gembs_latest.sif:/projects/rcorbettprj2/gemBS> which samtools #returns nothing Singularity gembs_latest.sif:/projects/rcorbettprj2/gemBS> which bcftools #returns nothing — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/heathsc/gemBS/issues/63?email_source=notifications&email_token=AAY465422YIHPF6Z4NBNMYLQPDHKRA5CNFSM4JBOF5B2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBRQP2A#issuecomment-543361000, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAY465Y5ESV7XPV4QMAMIQLQPDHKRANCNFSM4JBOF5BQ.
RichardCorbett
commented
4 years ago My login shell and related information:
echo $0
-bash
uname -a
Linux gphost14.bcgsc.ca 3.10.0-693.5.2.el7.x86_64 #1 SMP Fri Oct 20 20:32:50 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
bash --version
GNU bash, version 4.2.46(2)-release (x86_64-redhat-linux-gnu)
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Tried running some human samples just to confirm that the error above wasn't specific to the yeast test data. As before, I did not see any error messages until the extract step where I got the following output.
So no BCFTOOLS_PLUGINS error this time, but similar "unrecognized line 1 of /dev/stdin" error:
Singularity gembs_latest.sif:/projects/rcorbettprj2/gemBS/MCF> gemBS --loglevel debug extract
:
: Command extract started at 2019-10-17 21:56:24.932548
:
: Methylation Extraction...
2019-10-17 21:56:24,984 DEBUG: Using bundled binary : /usr/local/lib/python3.5/dist-packages/gemBS/bin/bcftools
2019-10-17 21:56:24,984 DEBUG: Using bundled binary : /usr/local/lib/python3.5/dist-packages/gemBS/bin/bcftools
2019-10-17 21:56:24,985 DEBUG: Using bundled binary : /usr/local/lib/python3.5/dist-packages/gemBS/gemBSbinaries/wigToBigWig
2019-10-17 21:56:24,985 INFO: Starting:
/usr/local/lib/python3.5/dist-packages/gemBS/bin/bcftools view -R ./extract/A34002/A34002_contig_list.bed -Ou ./calls/A34002/A34002.bcf | /usr/local/lib/python3.5/dist-packages/gemBS/bin/bcftools +mextr -- -z -o ./extract/A34002/A34002_cpg.txt --noncpgfile ./extract/A34002/A34002_non_cpg.txt --min-nc 1 -b ./extract/A34002/A34002 -w - --inform 1 --threshold 10 --mode strand-specific | /usr/local/lib/python3.5/dist-packages/gemBS/gemBSbinaries/wigToBigWig /dev/stdin indexes/hg38_no_alt.contig.sizes ./extract/A34002/A34002.bw
2019-10-17 21:56:24,985 DEBUG: Setting process log file to ./extract/A34002/mextr_A34002.err
2019-10-17 21:56:24,994 DEBUG: Starting subprocess
2019-10-17 21:56:24,999 DEBUG: Setting process log file to ./extract/A34002/mextr_A34002.err
2019-10-17 21:56:25,014 DEBUG: Setting process input to parent output
2019-10-17 21:56:25,014 DEBUG: Starting subprocess
2019-10-17 21:56:25,018 DEBUG: Setting process log file to ./extract/A34002/mextr_A34002.err
2019-10-17 21:56:25,020 DEBUG: Setting process input to parent output
2019-10-17 21:56:25,021 DEBUG: Starting subprocess
2019-10-17 21:56:25,197 DEBUG: Process '/usr/local/lib/python3.5/dist-packages/gemBS/gemBSbinaries/wigToBigWig' finished with 255
2019-10-17 21:56:25,197 ERROR: Process '/usr/local/lib/python3.5/dist-packages/gemBS/gemBSbinaries/wigToBigWig' finished with 255
2019-10-17 21:56:25,198 ERROR: Unrecognized line 1 of /dev/stdin:
2019-10-17 21:56:25,198 ERROR: 10584 0
2019-10-17 21:56:25,198 ERROR:
Traceback (most recent call last):
File "/usr/local/bin/gemBS", line 11, in <module>
load_entry_point('gemBS==3.3.1', 'console_scripts', 'gemBS')()
File "/usr/local/lib/python3.5/dist-packages/gemBS/commands.py", line 159, in gemBS_main
instances[args.command].run(args)
File "/usr/local/lib/python3.5/dist-packages/gemBS/production.py", line 1406, in run
self.do_filter(v)
File "/usr/local/lib/python3.5/dist-packages/gemBS/production.py", line 1500, in do_filter
snps=snps,snp_list=self.snp_list,snp_db=self.snp_db)
File "/usr/local/lib/python3.5/dist-packages/gemBS/__init__.py", line 1179, in methylationFiltering
raise ValueError("Error while extracting methylation calls.")
ValueError: Error while extracting methylation calls.
Thanks - can you confirm the version of singularity you are using?
Simon
On 18 Oct 2019, at 00:00, Richard Corbett notifications@github.com wrote:
Tried running some human samples just to confirm that the error above wasn't specific to the yeast test data. As before, I did not see any error messages until the extract step where I got the following output.
So no BCFTOOLS_PLUGINS error this time, but similar "unrecognized line 1 of /dev/stdin" error:
Singularity gembs_latest.sif:/projects/rcorbettprj2/gemBS/MCF> gemBS --loglevel debug extract : : Command extract started at 2019-10-17 21:56:24.932548 : : Methylation Extraction... 2019-10-17 21:56:24,984 DEBUG: Using bundled binary : /usr/local/lib/python3.5/dist-packages/gemBS/bin/bcftools 2019-10-17 21:56:24,984 DEBUG: Using bundled binary : /usr/local/lib/python3.5/dist-packages/gemBS/bin/bcftools 2019-10-17 21:56:24,985 DEBUG: Using bundled binary : /usr/local/lib/python3.5/dist-packages/gemBS/gemBSbinaries/wigToBigWig 2019-10-17 21:56:24,985 INFO: Starting: /usr/local/lib/python3.5/dist-packages/gemBS/bin/bcftools view -R ./extract/A34002/A34002_contig_list.bed -Ou ./calls/A34002/A34002.bcf | /usr/local/lib/python3.5/dist-packages/gemBS/bin/bcftools +mextr -- -z -o ./extract/A34002/A34002_cpg.txt --noncpgfile ./extract/A34002/A34002_non_cpg.txt --min-nc 1 -b ./extract/A34002/A34002 -w - --inform 1 --threshold 10 --mode strand-specific | /usr/local/lib/python3.5/dist-packages/gemBS/gemBSbinaries/wigToBigWig /dev/stdin indexes/hg38_no_alt.contig.sizes ./extract/A34002/A34002.bw 2019-10-17 21:56:24,985 DEBUG: Setting process log file to ./extract/A34002/mextr_A34002.err 2019-10-17 21:56:24,994 DEBUG: Starting subprocess 2019-10-17 21:56:24,999 DEBUG: Setting process log file to ./extract/A34002/mextr_A34002.err 2019-10-17 21:56:25,014 DEBUG: Setting process input to parent output 2019-10-17 21:56:25,014 DEBUG: Starting subprocess 2019-10-17 21:56:25,018 DEBUG: Setting process log file to ./extract/A34002/mextr_A34002.err 2019-10-17 21:56:25,020 DEBUG: Setting process input to parent output 2019-10-17 21:56:25,021 DEBUG: Starting subprocess 2019-10-17 21:56:25,197 DEBUG: Process '/usr/local/lib/python3.5/dist-packages/gemBS/gemBSbinaries/wigToBigWig' finished with 255 2019-10-17 21:56:25,197 ERROR: Process '/usr/local/lib/python3.5/dist-packages/gemBS/gemBSbinaries/wigToBigWig' finished with 255 2019-10-17 21:56:25,198 ERROR: Unrecognized line 1 of /dev/stdin: 2019-10-17 21:56:25,198 ERROR: 10584 0 2019-10-17 21:56:25,198 ERROR: Traceback (most recent call last): File "/usr/local/bin/gemBS", line 11, in
load_entry_point('gemBS==3.3.1', 'console_scripts', 'gemBS')() File "/usr/local/lib/python3.5/dist-packages/gemBS/commands.py", line 159, in gemBS_main instances[args.command].run(args) File "/usr/local/lib/python3.5/dist-packages/gemBS/production.py", line 1406, in run self.do_filter(v) File "/usr/local/lib/python3.5/dist-packages/gemBS/production.py", line 1500, in do_filter snps=snps,snp_list=self.snp_list,snp_db=self.snp_db) File "/usr/local/lib/python3.5/dist-packages/gemBS/init.py", line 1179, in methylationFiltering raise ValueError("Error while extracting methylation calls.") ValueError: Error while extracting methylation calls. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/heathsc/gemBS/issues/63?email_source=notifications&email_token=AAY4654KOVKWXWIB7ME2KKTQPDN6ZA5CNFSM4JBOF5B2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBRVAUY#issuecomment-543379539, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAY4655NN4FX4OISFAPP3MLQPDN6ZANCNFSM4JBOF5BQ.
heathsc
commented
4 years ago Hi Richard,
Tracked this down to a bug in the output of mextr that was in some situations not printing the header lines for the wig files. This is now fixed in gemBS 3.3.2 and tested with the new singularity image on Singularity Hub.
Thanks for the report. Simon
On 18 Oct 2019, at 00:00, Richard Corbett notifications@github.com wrote:
Tried running some human samples just to confirm that the error above wasn't specific to the yeast test data. As before, I did not see any error messages until the extract step where I got the following output.
So no BCFTOOLS_PLUGINS error this time, but similar "unrecognized line 1 of /dev/stdin" error:
Singularity gembs_latest.sif:/projects/rcorbettprj2/gemBS/MCF> gemBS --loglevel debug extract : : Command extract started at 2019-10-17 21:56:24.932548 : : Methylation Extraction... 2019-10-17 21:56:24,984 DEBUG: Using bundled binary : /usr/local/lib/python3.5/dist-packages/gemBS/bin/bcftools 2019-10-17 21:56:24,984 DEBUG: Using bundled binary : /usr/local/lib/python3.5/dist-packages/gemBS/bin/bcftools 2019-10-17 21:56:24,985 DEBUG: Using bundled binary : /usr/local/lib/python3.5/dist-packages/gemBS/gemBSbinaries/wigToBigWig 2019-10-17 21:56:24,985 INFO: Starting: /usr/local/lib/python3.5/dist-packages/gemBS/bin/bcftools view -R ./extract/A34002/A34002_contig_list.bed -Ou ./calls/A34002/A34002.bcf | /usr/local/lib/python3.5/dist-packages/gemBS/bin/bcftools +mextr -- -z -o ./extract/A34002/A34002_cpg.txt --noncpgfile ./extract/A34002/A34002_non_cpg.txt --min-nc 1 -b ./extract/A34002/A34002 -w - --inform 1 --threshold 10 --mode strand-specific | /usr/local/lib/python3.5/dist-packages/gemBS/gemBSbinaries/wigToBigWig /dev/stdin indexes/hg38_no_alt.contig.sizes ./extract/A34002/A34002.bw 2019-10-17 21:56:24,985 DEBUG: Setting process log file to ./extract/A34002/mextr_A34002.err 2019-10-17 21:56:24,994 DEBUG: Starting subprocess 2019-10-17 21:56:24,999 DEBUG: Setting process log file to ./extract/A34002/mextr_A34002.err 2019-10-17 21:56:25,014 DEBUG: Setting process input to parent output 2019-10-17 21:56:25,014 DEBUG: Starting subprocess 2019-10-17 21:56:25,018 DEBUG: Setting process log file to ./extract/A34002/mextr_A34002.err 2019-10-17 21:56:25,020 DEBUG: Setting process input to parent output 2019-10-17 21:56:25,021 DEBUG: Starting subprocess 2019-10-17 21:56:25,197 DEBUG: Process '/usr/local/lib/python3.5/dist-packages/gemBS/gemBSbinaries/wigToBigWig' finished with 255 2019-10-17 21:56:25,197 ERROR: Process '/usr/local/lib/python3.5/dist-packages/gemBS/gemBSbinaries/wigToBigWig' finished with 255 2019-10-17 21:56:25,198 ERROR: Unrecognized line 1 of /dev/stdin: 2019-10-17 21:56:25,198 ERROR: 10584 0 2019-10-17 21:56:25,198 ERROR: Traceback (most recent call last): File "/usr/local/bin/gemBS", line 11, in
load_entry_point('gemBS==3.3.1', 'console_scripts', 'gemBS')() File "/usr/local/lib/python3.5/dist-packages/gemBS/commands.py", line 159, in gemBS_main instances[args.command].run(args) File "/usr/local/lib/python3.5/dist-packages/gemBS/production.py", line 1406, in run self.do_filter(v) File "/usr/local/lib/python3.5/dist-packages/gemBS/production.py", line 1500, in do_filter snps=snps,snp_list=self.snp_list,snp_db=self.snp_db) File "/usr/local/lib/python3.5/dist-packages/gemBS/init.py", line 1179, in methylationFiltering raise ValueError("Error while extracting methylation calls.") ValueError: Error while extracting methylation calls. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/heathsc/gemBS/issues/63?email_source=notifications&email_token=AAY4654KOVKWXWIB7ME2KKTQPDN6ZA5CNFSM4JBOF5B2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBRVAUY#issuecomment-543379539, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAY4655NN4FX4OISFAPP3MLQPDN6ZANCNFSM4JBOF5BQ.
RichardCorbett
commented
4 years ago Thanks @heathsc My singularity version:
singularity --version
singularity version 3.0.1-104.g4feafc42.dirty.el7I'll test out the new container today and report back here.
RichardCorbett
commented
4 years ago I ran the new container and now all the steps complete successfully (thank you and well done!).
My first test was on the demo yeast data and I found that the extract results don't match the extract results provided with the test data. Is this expected?
diff --suppress-common-lines --side-by-side ./extract_ref/A001XS1/A001XS1.wig ./extract/A001XS1/A001XS1.wig | head -n 15
970 3.03 | 970 2.941
1439 18.75 | 1439 17.65
2747 14.81 | 2747 14.29
2909 20 | 2909 19.05
3475 75 | 3475 80
> 5292 0
> 5325 0
> variableStep chrom=chrI span=1
variableStep chrom=chrI span=1 <
> 8670 0
10284 4.545 | 10284 8.696
> variableStep chrom=chrI span=1
variableStep chrom=chrI span=1 <
12443 17.65 | 12443 16.67
14282 0 | 14282 14.29
Yes, I have to remake the 'standards'...
On Fri, 18 Oct 2019, 18:04 Richard Corbett, notifications@github.com wrote:
I ran the new container and now all the steps complete successfully.
My first test was on the demo yeast data and I found that the extract results don't match the extract results provided with the test data. Is this expected?
diff --suppress-common-lines --side-by-side ./extract_ref/A001XS1/A001XS1.wig ``` ./extract/A001XS1/A001XS1.wig | head -n 15 970 3.03 | 970 2.941 1439 18.75 | 1439 17.65 2747 14.81 | 2747 14.29 2909 20 | 2909 19.05 3475 75 | 3475 80
5292 0 5325 0 variableStep chrom=chrI span=1 variableStep chrom=chrI span=1 < 8670 0 10284 4.545 | 10284 8.696 variableStep chrom=chrI span=1 variableStep chrom=chrI span=1 < 12443 17.65 | 12443 16.67 14282 0 | 14282 14.29
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/heathsc/gemBS/issues/63?email_source=notifications&email_token=AAY465ZIXWUCCOJNHQVM4ILQPHNBRA5CNFSM4JBOF5B2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBU6TZI#issuecomment-543812069, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAY4655GBRAH5E3KQFI6JW3QPHNBRANCNFSM4JBOF5BQ .
RichardCorbett
commented
4 years ago Perfect. Thanks!
Hi folks,
I'm trying out the example yeast data from here: http://statgen.cnag.cat/GEMBS/UserGuide/downloads/gemBS_example.tar.gz
using the container here:
singularity pull docker://heathsc/gembs:latestI get no errors until I get to the extract step of the example. Here are my commands:
When I go into the container and run in debug mode I see the following messages:
gemBS --loglevel debug extractAfter seeing the above error I tried installing bcftools and plugins into the same folder as the test data and added the path to the BCFTOOLS_PLUGINS variable, but got the same error as above.
thanks Richard