search

galaxyproject / tools-iuc

Tool Shed repositories maintained by the Intergalactic Utilities Commission
https://galaxyproject.org/iuc
MIT License
160 stars 424 forks source link

macs2 fails when producing 'html' summary output #1691

Closed wkopp closed 6 years ago

wkopp commented 6 years ago

Dear IUC team,

I've encountered an issue when running the latest version of macs2 (revision acbd3fb47f90) on paired-end bam-files when requesting the 'html' output summary.

The Error produced is: Fatal error: Exit code 2 ()

However, the output of the tool does not indicate any errors. It seems that it successfully finished the job:

# Command line: callpeak --name MACS2 -t /data/galaxy/database/files/071/dataset_71461.dat -c /data/galaxy/database/files/071/dataset_71462.dat --format BAMPE --gsize 2700000000 --keep-dup 1 --qvalue 0.05 --mfold 5 50 --bw 300
# ARGUMENTS LIST:
# name = MACS2
# format = BAMPE
# ChIP-seq file = ['/data/galaxy/database/files/071/dataset_71461.dat']
# control file = ['/data/galaxy/database/files/071/dataset_71462.dat']
# effective genome size = 2.70e+09
# band width = 300
# model fold = [5, 50]
# qvalue cutoff = 5.00e-02
# Larger dataset will be scaled towards smaller dataset.
# Range for calculating regional lambda is: 1000 bps and 10000 bps
# Broad region calling is off
# Paired-End mode is on

INFO  @ Mon, 29 Jan 2018 10:42:20: #1 read fragment files... 
INFO  @ Mon, 29 Jan 2018 10:42:20: #1 read treatment fragments... 
INFO  @ Mon, 29 Jan 2018 10:42:30:  1000000 
INFO  @ Mon, 29 Jan 2018 10:42:40:  2000000 
INFO  @ Mon, 29 Jan 2018 10:42:49:  3000000 
:
:
:
INFO  @ Mon, 29 Jan 2018 10:53:00:  37000000 
INFO  @ Mon, 29 Jan 2018 10:53:09:  38000000 
INFO  @ Mon, 29 Jan 2018 10:53:18:  39000000 
INFO  @ Mon, 29 Jan 2018 10:53:27:  40000000 
INFO  @ Mon, 29 Jan 2018 10:53:36:  41000000 
INFO  @ Mon, 29 Jan 2018 10:54:03: #1 mean fragment size is determined as 193 bp from treatment 
INFO  @ Mon, 29 Jan 2018 10:54:03: #1 note: mean fragment size in control is 209 bp -- value ignored 
INFO  @ Mon, 29 Jan 2018 10:54:03: #1 fragment size = 193 
INFO  @ Mon, 29 Jan 2018 10:54:03: #1  total fragments in treatment: 31619050 
INFO  @ Mon, 29 Jan 2018 10:54:03: #1 user defined the maximum fragments... 
INFO  @ Mon, 29 Jan 2018 10:54:03: #1 filter out redundant fragments by allowing at most 1 identical fragment(s) 
INFO  @ Mon, 29 Jan 2018 10:55:06: #1  fragments after filtering in treatment: 30858378 
INFO  @ Mon, 29 Jan 2018 10:55:06: #1  Redundant rate of treatment: 0.02 
INFO  @ Mon, 29 Jan 2018 10:55:06: #1  total fragments in control: 41074688 
INFO  @ Mon, 29 Jan 2018 10:55:06: #1 user defined the maximum fragments... 
INFO  @ Mon, 29 Jan 2018 10:55:06: #1 filter out redundant fragments by allowing at most 1 identical fragment(s) 
INFO  @ Mon, 29 Jan 2018 10:56:27: #1  fragments after filtering in control: 39822728 
INFO  @ Mon, 29 Jan 2018 10:56:27: #1  Redundant rate of control: 0.03 
INFO  @ Mon, 29 Jan 2018 10:56:27: #1 finished! 
INFO  @ Mon, 29 Jan 2018 10:56:27: #2 Build Peak Model... 
INFO  @ Mon, 29 Jan 2018 10:56:27: #2 Skipped... 
INFO  @ Mon, 29 Jan 2018 10:56:27: #2 Use 193 as fragment length 
INFO  @ Mon, 29 Jan 2018 10:56:27: #3 Call peaks... 
INFO  @ Mon, 29 Jan 2018 10:56:27: #3 Pre-compute pvalue-qvalue table... 
INFO  @ Mon, 29 Jan 2018 11:01:18: #3 Call peaks for each chromosome... 
INFO  @ Mon, 29 Jan 2018 11:03:47: #4 Write output xls file... MACS2_peaks.xls 
INFO  @ Mon, 29 Jan 2018 11:03:47: #4 Write peak in narrowPeak format file... MACS2_peaks.narrowPeak 
INFO  @ Mon, 29 Jan 2018 11:03:47: #4 Write summits bed file... MACS2_summits.bed 
INFO  @ Mon, 29 Jan 2018 11:03:47: Done!

The command that was invoked by Galaxy was the following: export PYTHON_EGG_CACHE=pwd&& (macs2 callpeak --name 'MACS2' -t '/data/galaxy/database/files/071/dataset_71461.dat' -c '/data/galaxy/database/files/071/dataset_71462.dat' --format BAMPE --gsize '2700000000' --keep-dup '1' --qvalue '0.05' --mfold '5' '50' --bw '300' 2>&1 > macs2_stderr) && Rscript MACS2_model.r > MACS2_model.r.log && ( count=ls -1 MACS2 2>/dev/null | wc -l`; if [ $count != 0 ]; then mkdir '/data/galaxy/database/job_working_directory/040/40397/dataset_71874_files' && cp -r MACS2 '/data/galaxy/database/job_working_directory/040/40397/dataset_71874_files' && python '/data/galaxy/shed_tools/toolshed.g2.bx.psu.edu/repos/iuc/macs2/acbd3fb47f90/macs2/dir2html.py' '/data/galaxy/database/job_working_directory/040/40397/dataset_71874_files' macs2_stderr > '/data/galaxy/database/files/071/dataset_71874.dat'; fi; ) && exit_code_for_galaxy=$? && cat macs2_stderr 2>&1 && (exit $exit_code_for_galaxy)`

I have tried to reproduce the error by running the above command on the command line. I found that macs2 indeed produces the output files correctly, but afterwards mkdir '/data/galaxy/database/job_working_directory/040/40397/dataset_71874_files' fails, because it cannot recursively create the output directory.

I believe that mkdir -p <path> should be used there or am I mistaken?

I have tried this locally by changing the command in the macs2_callpeak.xml. Now the output is created and visible in View data in Galaxy after the job was finished. However, macs2 still fails with Fatal error: Exit code 2 () even though on the command line I cannot reproduce the error anymore.

Thank you for your help! Best, Wolfgang

bgruening commented 6 years ago

@wkopp is this the latest MACS2 version from the toolshed? If you like, can you try to reproduce this error on http://galaxy.uni-freiburg.de.

wkopp commented 6 years ago

Yes, this occurred with the latest version, Revision 9:acbd3fb47f90 (2018-01-25). I'll try to reproduce it on http://galaxy.uni-freiburg.de.

wkopp commented 6 years ago

I have reproduced the issue on http://galaxy.uni-freiburg.de. You'll find the history here: https://galaxy.uni-freiburg.de/u/wkopp/h/macs2-issue

bgruening commented 6 years ago

@wkopp what I figured out until now is that it works if you don't tick the PDF output. So its seems to be related to the PDF.

bgruening commented 6 years ago

@wkopp in BAMPE mode MACS2 does not produce any PDF output. This potential fix should make it more obvious: https://github.com/galaxyproject/tools-iuc/compare/master...1691?quick_pull=1

Thanks to @mblue9 for debugging help this late :)

bgruening commented 6 years ago

@wkopp if you like please test: https://github.com/galaxyproject/tools-iuc/pull/1698 It should fix our issue, by simply not generating the PDF if you are running in BAMPE mode.

wkopp commented 6 years ago

@bgruening There has been a new release for macs2 (revision 10:f0b351e734b8 from 2018-01-31) which contains the fix. I saw that you already installed it on galaxy.uni-freiburg.de. So I just tried to test it again on the same data. Unfortunately, it still fails. See: https://galaxy.uni-freiburg.de/u/wkopp/h/macs2-issue. This time for a different reason:

Fatal error: Exit code 1 ()
cat: MACS2_model.r: No such file or directory

Thank you for the help!

bgruening commented 6 years ago

Hi Wolfgang, sorry I did not update the wrapper in time. I did now and also fixed the older version. Should work now.