Closed zhenyisong closed 7 years ago
ewels
commented
7 years ago Hi @zhenyisong,
This looks like a bug in the pipeline code - it's telling Picard to deduplicate your reads with 0 Gb of memory.. I'll see if we can fix this next week for you.
Phil
ewels
commented
7 years ago Hi @zhenyisong,
I've just pushed a change which should fix your problem - it checks the value returned by task.memory now and defaults to 3GB if it's 0. Also, if it's less than 250MB, it defaults to 250MB.
Please let me know if this fixes the problem for you :)
Phil
ewels
commented
7 years ago ps. I'm not clear on why the command is getting 0GB back as the process memory when you're setting it to 40GB in your config file.
ewels
commented
7 years ago Apologies - fix is in ewels/NGI-ChIPseq and not in this repo yet. Will be after a PR + merge soon.
zhenyisong
commented
7 years ago Same error. the bwa dir in the results contains no output. I used the top command to monitor the process, it seemed that the bwa process was performed but no outpout (sam or bam files were missed), which I think led to the error in Picard and other steps.
chuan-wang
commented
7 years ago Hi @zhenyisong , Can you get into the working directory for the bwa process and check the following two things? 1) Are there fq files from trim_galore? 2) What's the error message in .command.err? Best, Chuan
zhenyisong
commented
7 years ago @chuan-wang the running dir does contain the trimmed fq from upstream. And I used samtools view -H to check the bam output, it did show that the bwa completed its task. (@PG ID:bwa PN:bwa VN:0.7.15-r1140 CL:bwa mem -M BWAIndex/genome.fa SRR1029877_trimmed.fq.gz ). Head the .command.err: [samfaipath] build FASTA index... [fai_build] fail to open the FASTA file BWAIndex [samfaipath] fail to build FASTA index. [M::bwa_idx_load_from_disk] read 0 ALT contigs [M::process] read 202044 sequences (10000070 bp)... [M::process] read 202040 sequences (10000073 bp)... [M::mem_process_seqs] Processed 202044 reads in 55.884 CPU sec, 56.103 real sec
PS. results dir has an empty bwa dir. (no bam files in it, however)
chuan-wang
commented
7 years ago @zhenyisong Do you mean that there were .bam output files in the working folder? From the error message it seems that the index files are missing. Which version of samtools are you using?
ewels
commented
7 years ago Hi both,
This is the expected behaviour. See --saveTrimmed and --saveAlignedIntermediates.
The files will be generated and present in the work directory and used by downstream processes, but to save space for routine runs we don't copy these files to results by default. There should be BAM files saved to results after deduplication and sorting though.
I am surprised that the bwa directory is present and empty though. My latest changes should have prevented that from being created if it has no files. Also they should have fixed the Picard errors. This makes me wonder whether you are definitely running the pipeline with the latest version of the code @zhenyisong?
If you're using NextFlow to get the pipeline (eg. nextflow run SciLifeLab/NGI-ChIPseq), note that NF doesn't update the pipeline every time. It just uses the same code that it used when you first run that command. You need to run nextflow pull SciLifeLab/NGI-ChIPseq to get the latest version.
Phil
ewels
commented
7 years ago And just to clarify..
it seemed that the bwa process was performed but no outpout (sam or bam files were missed), which I think led to the error in Picard and other steps.
This is not the case, no. The bwa process did produce output, it just didn't copy it to the results directory (see NF publishDir directive).
The reported Picard error is due to this line of old code which I updated. It was fetching the amount of memory available for the task and telling Picard to use this.
In your error log you can see that the Picard command is getting the following string: java -Xmx0g. This is telling Java to use 0 GB of memory - nothing. That is why the process fails.
My fix checks for this and enforces a minimum amount of memory, defaulting to 3 GB.
Phil
zhenyisong
commented
7 years ago @ewels @chuan-wang thanks. I updated the Github source and re-run it and I will check the result tomorrow.
zhenyisong
commented
7 years ago @chuan-wang hi, I re-ran the pileline and found masc2 did not proceed its task. Here is the excerpt from nextflow.log.
Caused by:
Process `macs (SRR1029876.dedup.sorted)` terminated with an error exit status (1)
Command executed:
macs2 callpeak \
-t SRR1029880.fastq.dedup.sorted.bam \
-c SRR1029881.fastq.dedup.sorted.bam \
-f BAM \
-g mm \
-n heart_17_5 \
-q 0.01
Command exit status:
1
Command output:
(empty)
Command error:
INFO @ Tue, 13 Jun 2017 16:49:34:
# Command line: callpeak -t SRR1029880.fastq.dedup.sorted.bam -c SRR1029881.fastq.dedup.sorted.bam -f BAM -g mm -n heart_17_5 -q 0.01
# ARGUMENTS LIST:
# name = heart_17_5
# format = BAM
# ChIP-seq file = ['SRR1029880.fastq.dedup.sorted.bam']
# control file = ['SRR1029881.fastq.dedup.sorted.bam']
# effective genome size = 1.87e+09
# band width = 300
# model fold = [5, 50]
# qvalue cutoff = 1.00e-02
# Larger dataset will be scaled towards smaller dataset.
# Range for calculating regional lambda is: 1000 bps and 10000 bps
# Broad region calling is off
# Paired-End mode is offmm not specified in MACS2 commandline? By the way, is there any param to open the broad model? Or support epic (fast version of Sicer)? And for QC metrics, do you have any plan to bundle together with ChiLin?
chuan-wang
commented
7 years ago @zhenyisong Hi, mm is a specified option in MACS2, and the it has been processed correctly by the scripts as we can see that # effective genome size = 1.87e+09 which is exactly the size for mm. Have you double checked the names of bam files from picard? How did you set up the sample names in the macsconfig file?
Currently we don't have plans to add other modules, since we are using this pipleline for general QC of different projects from external users.
zhenyisong
commented
7 years ago More from the log:
Command error:
INFO @ Tue, 13 Jun 2017 16:49:34:
# Command line: callpeak -t SRR1029880.fastq.dedup.sorted.bam -c SRR1029881.fastq.dedup.sorted.bam -f BAM -g mm -n heart_17_5 -q 0.01
# ARGUMENTS LIST:
# name = heart_17_5
# format = BAM
# ChIP-seq file = ['SRR1029880.fastq.dedup.sorted.bam']
# control file = ['SRR1029881.fastq.dedup.sorted.bam']
# effective genome size = 1.87e+09
# band width = 300
# model fold = [5, 50]
# qvalue cutoff = 1.00e-02
# Larger dataset will be scaled towards smaller dataset.
# Range for calculating regional lambda is: 1000 bps and 10000 bps
# Broad region calling is off
# Paired-End mode is off
INFO @ Tue, 13 Jun 2017 16:49:34: #1 read tag files...
INFO @ Tue, 13 Jun 2017 16:49:34: #1 read treatment tags...
Traceback (most recent call last):
File "/usr/local/bin/macs2", line 617, in <module>
main()
File "/usr/local/bin/macs2", line 57, in main
run( args )
File "/usr/local/lib/python2.7/dist-packages/MACS2/callpeak_cmd.py", line 73, in run
else: (treat, control) = load_tag_files_options (options)
File "/usr/local/lib/python2.7/dist-packages/MACS2/callpeak_cmd.py", line 395, in load_tag_files_options
tp = options.parser(options.tfile[0], buffer_size=options.buffer_size)
File "MACS2/IO/Parser.pyx", line 774, in MACS2.IO.Parser.BAMParser.__init__ (MACS2/IO/Parser.c:11146)
File "/usr/lib/python2.7/gzip.py", line 34, in open
return GzipFile(filename, mode, compresslevel)
File "/usr/lib/python2.7/gzip.py", line 94, in __init__
fileobj = self.myfileobj = __builtin__.open(filename, mode or 'rb')
IOError: [Errno 2] No such file or directory: 'SRR1029880.fastq.dedup.sorted.bam'I think it is the file naming problem. In the soft link dir, there did exit the file:
SRR1029876.dedup.sorted.bamIn macs2 command line, the pipeline tried to call this file instead:
SRR1029876.fastq.dedup.sorted.bam @zhenyisong OK, that's exactly what I have suspected. In the macsconfig file, the names of test and control samples should be the base name without any .fastq or .fastq.gz postfix.
zhenyisong
commented
7 years ago @chuan-wang thanks. I corrected the macs.config. Futher, how do I open the broad mode in macs2 setting? I mean, my ChIP-seq data was from H3K27me3 or similar epigenetic protein family. I reviewed the ChIP-seq manual, but failed to find any parameter to specify this.
chuan-wang
commented
7 years ago @zhenyisong The easiest way is to modify your local version of main.nf. You can find it in ~/.nextflow/assets/SciLifeLab/NGI-ChIPseq/.
zhenyisong
commented
7 years ago @chuan-wang Thanks. This is a choice and will influence the global setting. However, the script code will be not well maintained. I will change MACS params manually if I transfer the model (broad or narrow).
ewels
commented
7 years ago I've updated the docs a little to make it more clear about the filename thing, see #34
How much needs to be changed to run in broad-peak mode? If it's just a single flag then we could add that as an optional pipeline flag, I think it would be useful.
ewels
commented
7 years ago Pipeine --broad flag added, which sets the --broad flag for MACS2. See updated docs
zhenyisong
commented
7 years ago still error:
[4a/fb6265] Submitted process > macs (SRR1029876.dedup.sorted)
ERROR ~ Error executing process > 'ngsplot (SRR1029876.dedup.sorted)'
Caused by:
Process `ngsplot (SRR1029876.dedup.sorted)` terminated with an error exit status (1)
Command executed:
ngs_config_generate.r SRR1029876.dedup.sorted.bam SRR1029884.dedup.sorted.bam SRR1029888.dedup.sorted.bam SRR1029877.dedup.sorted.bam SRR1029886.dedup.sorted.bam SRR1029878.
dedup.sorted.bam SRR1029881.dedup.sorted.bam SRR1029889.dedup.sorted.bam SRR1029879.dedup.sorted.bam SRR1029882.dedup.sorted.bam SRR1029880.dedup.sorted.bam SRR1029887.dedup.sorted.bam SRR1029883.dedup.sorted.bam SRR1029885.dedup.sorted.bam
ngs.plot.r \
-G mm10 \
-R genebody \
-C ngsplot_config \
-O Genebody \
-D ensembl \
-FL 300
ngs.plot.r \
-G mm10 \
-R tss \
-C ngsplot_config \
-O TSS \
-FL 300
Command exit status:
1
Command output:
Configuring variables...
Command error:
Error in CheckRegionAllowed(reg2plot, default.tbl) :
Unknown region specified. Must be one of: bed
Execution halted
Work dir:
/mnt/date/biodata/data/cellChIP/heart/work/35/230ff3607e63888b7fa0a9587d2bfc
Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`
-- Check '.nextflow.log' file for details
WARN: Killing pending tasks (10)
The parameter for ngs.plot is confusing. The "mm10'" is default naming for UCSC assemble, the input and accepted naming for this pipeline is NCBI GRCm38.
And the MACS2 uses the default q = 0.05 to filter out the non-significant peaks, and I think we should use the q = 1 instead thus guaranteed to keep the whole output peaks and leave it to end user to make decision.
And one more, current setting is that even pipeline is broken down in some steps, the final output still can be viewed in the results dir, which can lead to confusion. My suggestion is that only pipeline successfully is completed, then the result dir will appear. Otherwise, I have to check the .nextflow.log to make sure all is right. My first time experience is that when I checked the results dir, I am confused about the empty bwa dir which arose me worry about the final result. If the BWA dir was at that time not empty, I would like to think the whole task is finished without any complaint.
I am still looking for the specific QC metrics for broad setting. Recent review in QC and pipeline management is reviewed here. Chilin is not well maintained, at least no response from their Github account.
Thanks for your effort to maintain such pipeline.
chuan-wang
commented
7 years ago @zhenyisong Hi, do you have both the .bam and .bai files in the working dir? What does the ngsplot_config look like?
mm10 is a pre-defined paramter by ngs-plot, and we can do nothing about it.
For the pipeline running issue, we only tolerate error messages from QC steps but not from the core data processing steps. This is for the purpose of minimizing time consumption since QC steps can be easily redone manually. As pointed by @ewels, additional parameter --saveAlignedIntermediates is required for saving bwa alignment results.
We will investigate your suggestion for improving this pipeline. Thanks!
ewels
commented
7 years ago To clarify - results are saved in the results directory (or whatever you specify with --outdir) as the pipeline goes, irrespective of failures (@chuan-wang - yes we do tolerate some errors, but even if there are errors that halt the pipeline there will still be files in the results directory). This is core NextFlow behaviour and in my opinion a useful thing.
There are four different ways to track pipeline success:
.nextflow.log fileDocumentation/pipeline_report.html file--email is set)The last two specifically highlight in big red text if something has gone wrong. I think that this is sufficient to notice if the pipeline has had an error.
We could add an option to allow custom specification of the q threshold for MACS2, but I do not think that we should return all peaks by default. Most users will not want to have to select peaks themselves, though I agree that it is a useful for some users to be able to do such downstream processing. @chuan-wang - perhaps we can add params.macs_q = 0.5 to the pipeline instead of hardcoding the value?
Thanks for the manuscript link, looks useful 👍
Phil
ewels
commented
7 years ago Regarding the error you pasted about from ngsplot; please can you paste the output from the following command?
ls -la /mnt/date/biodata/data/cellChIP/heart/work/35/230ff3607e63888b7fa0a9587d2bfcThis is so that we can see all files in the work directory for that process to help with debugging.
zhenyisong
commented
7 years ago Here is the command output:
$ ls -la /mnt/date/biodata/data/cellChIP/heart/work/35/230ff3607e63888b7fa0a9587d2bfc
total 40
drwxrwxr-x 2 zhenyisong zhenyisong 4096 Jun 15 18:15 .
drwxrwxr-x 4 zhenyisong zhenyisong 80 Jun 15 18:15 ..
-rw-rw-r-- 1 zhenyisong zhenyisong 0 Jun 15 18:15 .command.begin
-rw-rw-r-- 1 zhenyisong zhenyisong 81 Jun 15 18:15 .command.env
-rw-rw-r-- 1 zhenyisong zhenyisong 120 Jun 15 18:15 .command.err
-rw-rw-r-- 1 zhenyisong zhenyisong 144 Jun 15 18:15 .command.log
-rw-rw-r-- 1 zhenyisong zhenyisong 24 Jun 15 18:15 .command.out
-rw-rw-r-- 1 zhenyisong zhenyisong 7335 Jun 15 18:15 .command.run
-rw-rw-r-- 1 zhenyisong zhenyisong 637 Jun 15 18:15 .command.sh
-rw-rw-r-- 1 zhenyisong zhenyisong 1 Jun 15 18:15 .exitcode
-rw-r--r-- 1 root root 616 Jun 15 18:15 ngsplot_config
lrwxrwxrwx 1 zhenyisong zhenyisong 104 Jun 15 18:15 SRR1029876.dedup.sorted.bam -> /mnt/date/biodata/data/cellChIP/heart/work/ec/18a7e53e8118ac8d54d3b2f9022767/SRR1029876.dedup.sorted.bam
lrwxrwxrwx 1 zhenyisong zhenyisong 108 Jun 15 18:15 SRR1029876.dedup.sorted.bam.bai -> /mnt/date/biodata/data/cellChIP/heart/work/ec/18a7e53e8118ac8d54d3b2f9022767/SRR1029876.dedup.sorted.bam.bai
lrwxrwxrwx 1 zhenyisong zhenyisong 104 Jun 15 18:15 SRR1029877.dedup.sorted.bam -> /mnt/date/biodata/data/cellChIP/heart/work/e1/ace6525722b409e5df4d0618c9ff65/SRR1029877.dedup.sorted.bam
lrwxrwxrwx 1 zhenyisong zhenyisong 108 Jun 15 18:15 SRR1029877.dedup.sorted.bam.bai -> /mnt/date/biodata/data/cellChIP/heart/work/e1/ace6525722b409e5df4d0618c9ff65/SRR1029877.dedup.sorted.bam.bai
lrwxrwxrwx 1 zhenyisong zhenyisong 104 Jun 15 18:15 SRR1029878.dedup.sorted.bam -> /mnt/date/biodata/data/cellChIP/heart/work/49/f0bb43e38961fe416eb75d86b34a72/SRR1029878.dedup.sorted.bam
lrwxrwxrwx 1 zhenyisong zhenyisong 108 Jun 15 18:15 SRR1029878.dedup.sorted.bam.bai -> /mnt/date/biodata/data/cellChIP/heart/work/49/f0bb43e38961fe416eb75d86b34a72/SRR1029878.dedup.sorted.bam.bai
lrwxrwxrwx 1 zhenyisong zhenyisong 104 Jun 15 18:15 SRR1029879.dedup.sorted.bam -> /mnt/date/biodata/data/cellChIP/heart/work/f4/1a4a17813e45a48151db4c8624e496/SRR1029879.dedup.sorted.bam
lrwxrwxrwx 1 zhenyisong zhenyisong 108 Jun 15 18:15 SRR1029879.dedup.sorted.bam.bai -> /mnt/date/biodata/data/cellChIP/heart/work/f4/1a4a17813e45a48151db4c8624e496/SRR1029879.dedup.sorted.bam.bai
lrwxrwxrwx 1 zhenyisong zhenyisong 104 Jun 15 18:15 SRR1029880.dedup.sorted.bam -> /mnt/date/biodata/data/cellChIP/heart/work/1d/48299d3930ff5e739cd99c7e68b26f/SRR1029880.dedup.sorted.bam
lrwxrwxrwx 1 zhenyisong zhenyisong 108 Jun 15 18:15 SRR1029880.dedup.sorted.bam.bai -> /mnt/date/biodata/data/cellChIP/heart/work/1d/48299d3930ff5e739cd99c7e68b26f/SRR1029880.dedup.sorted.bam.bai
lrwxrwxrwx 1 zhenyisong zhenyisong 104 Jun 15 18:15 SRR1029881.dedup.sorted.bam -> /mnt/date/biodata/data/cellChIP/heart/work/17/d564a92b445ce0c63a5ba2481f234b/SRR1029881.dedup.sorted.bam
lrwxrwxrwx 1 zhenyisong zhenyisong 108 Jun 15 18:15 SRR1029881.dedup.sorted.bam.bai -> /mnt/date/biodata/data/cellChIP/heart/work/17/d564a92b445ce0c63a5ba2481f234b/SRR1029881.dedup.sorted.bam.bai
lrwxrwxrwx 1 zhenyisong zhenyisong 104 Jun 15 18:15 SRR1029882.dedup.sorted.bam -> /mnt/date/biodata/data/cellChIP/heart/work/cf/2e24cf9d4a3a142dfe9746f4cf67d4/SRR1029882.dedup.sorted.bam
lrwxrwxrwx 1 zhenyisong zhenyisong 108 Jun 15 18:15 SRR1029882.dedup.sorted.bam.bai -> /mnt/date/biodata/data/cellChIP/heart/work/cf/2e24cf9d4a3a142dfe9746f4cf67d4/SRR1029882.dedup.sorted.bam.bai
lrwxrwxrwx 1 zhenyisong zhenyisong 104 Jun 15 18:15 SRR1029883.dedup.sorted.bam -> /mnt/date/biodata/data/cellChIP/heart/work/7d/34d10c48d7ad330841e2176d6a8360/SRR1029883.dedup.sorted.bam
lrwxrwxrwx 1 zhenyisong zhenyisong 108 Jun 15 18:15 SRR1029883.dedup.sorted.bam.bai -> /mnt/date/biodata/data/cellChIP/heart/work/7d/34d10c48d7ad330841e2176d6a8360/SRR1029883.dedup.sorted.bam.bai
lrwxrwxrwx 1 zhenyisong zhenyisong 104 Jun 15 18:15 SRR1029884.dedup.sorted.bam -> /mnt/date/biodata/data/cellChIP/heart/work/38/67b3b699fc7b25b267a07a6a460584/SRR1029884.dedup.sorted.bam
lrwxrwxrwx 1 zhenyisong zhenyisong 108 Jun 15 18:15 SRR1029884.dedup.sorted.bam.bai -> /mnt/date/biodata/data/cellChIP/heart/work/38/67b3b699fc7b25b267a07a6a460584/SRR1029884.dedup.sorted.bam.bai
lrwxrwxrwx 1 zhenyisong zhenyisong 104 Jun 15 18:15 SRR1029885.dedup.sorted.bam -> /mnt/date/biodata/data/cellChIP/heart/work/20/c6ab710025fbc2fcc53b15b83d1496/SRR1029885.dedup.sorted.bam
lrwxrwxrwx 1 zhenyisong zhenyisong 108 Jun 15 18:15 SRR1029885.dedup.sorted.bam.bai -> /mnt/date/biodata/data/cellChIP/heart/work/20/c6ab710025fbc2fcc53b15b83d1496/SRR1029885.dedup.sorted.bam.bai
lrwxrwxrwx 1 zhenyisong zhenyisong 104 Jun 15 18:15 SRR1029886.dedup.sorted.bam -> /mnt/date/biodata/data/cellChIP/heart/work/7f/231acf85d13a19bfe87f38f80f8611/SRR1029886.dedup.sorted.bam
lrwxrwxrwx 1 zhenyisong zhenyisong 108 Jun 15 18:15 SRR1029886.dedup.sorted.bam.bai -> /mnt/date/biodata/data/cellChIP/heart/work/7f/231acf85d13a19bfe87f38f80f8611/SRR1029886.dedup.sorted.bam.bai
lrwxrwxrwx 1 zhenyisong zhenyisong 104 Jun 15 18:15 SRR1029887.dedup.sorted.bam -> /mnt/date/biodata/data/cellChIP/heart/work/25/7e9fa035918486efbad8d01e29102e/SRR1029887.dedup.sorted.bam
lrwxrwxrwx 1 zhenyisong zhenyisong 108 Jun 15 18:15 SRR1029887.dedup.sorted.bam.bai -> /mnt/date/biodata/data/cellChIP/heart/work/25/7e9fa035918486efbad8d01e29102e/SRR1029887.dedup.sorted.bam.bai
lrwxrwxrwx 1 zhenyisong zhenyisong 104 Jun 15 18:15 SRR1029888.dedup.sorted.bam -> /mnt/date/biodata/data/cellChIP/heart/work/28/9d32219b97125637d9d2216de7ea37/SRR1029888.dedup.sorted.bam
lrwxrwxrwx 1 zhenyisong zhenyisong 108 Jun 15 18:15 SRR1029888.dedup.sorted.bam.bai -> /mnt/date/biodata/data/cellChIP/heart/work/28/9d32219b97125637d9d2216de7ea37/SRR1029888.dedup.sorted.bam.bai
lrwxrwxrwx 1 zhenyisong zhenyisong 104 Jun 15 18:15 SRR1029889.dedup.sorted.bam -> /mnt/date/biodata/data/cellChIP/heart/work/6d/873f23be4080d30a637c2a3b31af13/SRR1029889.dedup.sorted.bam
lrwxrwxrwx 1 zhenyisong zhenyisong 108 Jun 15 18:15 SRR1029889.dedup.sorted.bam.bai -> /mnt/date/biodata/data/cellChIP/heart/work/6d/873f23be4080d30a637c2a3b31af13/SRR1029889.dedup.sorted.bam.baiAnd this is the ngsplot_config:
more ngs*
SRR1029876.dedup.sorted.bam -1 "SRR1029876"
SRR1029877.dedup.sorted.bam -1 "SRR1029877"
SRR1029878.dedup.sorted.bam -1 "SRR1029878"
SRR1029879.dedup.sorted.bam -1 "SRR1029879"
SRR1029880.dedup.sorted.bam -1 "SRR1029880"
SRR1029881.dedup.sorted.bam -1 "SRR1029881"
SRR1029882.dedup.sorted.bam -1 "SRR1029882"
SRR1029883.dedup.sorted.bam -1 "SRR1029883"
SRR1029884.dedup.sorted.bam -1 "SRR1029884"
SRR1029885.dedup.sorted.bam -1 "SRR1029885"
SRR1029886.dedup.sorted.bam -1 "SRR1029886"
SRR1029887.dedup.sorted.bam -1 "SRR1029887"
SRR1029888.dedup.sorted.bam -1 "SRR1029888"
SRR1029889.dedup.sorted.bam -1 "SRR1029889"Z
chuan-wang
commented
7 years ago @zhenyisong Hi, there is nothing wrong with the files. From the error message "Unknown region specified. Must be one of: bed" I'm 99% sure it is due to the ngsplot itself since a correct error message should be like this: "Unknown region specified. Must be one of: cgi exon genebody tss tes bed". Could you try to install the latest version of ngsplot and rerun the scripts in the working directory?
zhenyisong
commented
7 years ago @chuan-wang I used the Docker configuration. Or does this mean I should provide extra annotation file for this program (ngsplot) outside of the Docker environment?
chuan-wang
commented
7 years ago @zhenyisong Hi, I finally found that in this version of ngsplot the genome files are not included by default and user needs to install them manually. I have fixed it and a PR has been merged to SciLifeLab/NGI-ChIPseq. Please re-build the docker image and try the scripts. Thank you for pinpointing this bug!
ewels
commented
7 years ago No need to build the docker image, just pull the latest version from dockerhub: https://hub.docker.com/r/scilifelab/ngi-chipseq/builds/bmjrz25ebuub34o2w8rfyuu/
ewels
commented
7 years ago Ok great - is this all of your issues now resolved @zhenyisong?
zhenyisong
commented
7 years ago @chuan-wang & @ewels
yet another package should be imaged in the docker:
Command output:
Configuring variables...
Using database:
/opt/ngsplot-2.61//database/mm10/mm10.ensembl.genebody.protein_coding.RData
Done
Loading R libraries.
Command error:
Bioconductor version 3.4 (BiocInstaller 1.24.0), ?biocLite for help
A new version of Bioconductor is available after installing the most recent
version of R; see http://bioconductor.org/install
Error in match(x, table, nomatch = 0L) : object 'BSgenome' not found
Calls: biocLite -> %in% -> match
In addition: Warning message:
In library(package, lib.loc = lib.loc, character.only = TRUE, logical.return = TRUE, :
there is no package called ‘BSgenome’
Execution halted
Work dir:
/mnt/date/biodata/data/cellChIP/heart/work/ab/8520f278bfeec7508ae8b9ab85c626
Tip: when you have fixed the problem you can continue the execution appending to the nextflow command line the option `-resume`Hi, there is an installation of the R package BSgenome in the scripts. Maybe an R version issue. I will look into it. Best, chuan
chuan-wang
commented
7 years ago @zhenyisong Hi, I found the bug. libxml2-dev is required for the installation of BSgenome. @ewels, can you add it somewhere between line 10 and 26 in the Dockerfile in your latest PR? Thanks!
ewels
commented
7 years ago Added in #37
zhenyisong
commented
7 years ago @chuan-wang One R package (markdown?) is missed?
NGI-ChIPseq execution completed unsuccessfully!
The exit status of the task that caused the workflow execution to fail was: 1.
The full error message was:
Error executing process > 'output_documentation (SRR1029885)'
Caused by:
Process `output_documentation (SRR1029885)` terminated with an error exit status (1)
Command executed:
markdown_to_html.r /home/zhenyisong/.nextflow/assets/SciLifeLab/NGI-ChIPseq/docs/output.md results_description.html
Command exit status:
1
Command output:
g++ -std=c++11 -I/usr/local/lib/R/include -DNDEBUG -I. -UDEBUG -DNDEBUG -DU_HAVE_ELF_H -I/usr/local/include -fpic -fpic -g -O2 -c stri_search_regex_detect.cpp -o stri_search_regex_detect.o
g++ -std=c++11 -I/usr/local/lib/R/include -DNDEBUG -I. -UDEBUG -DNDEBUG -DU_HAVE_ELF_H -I/usr/local/include -fpic -fpic -g -O2 -c stri_search_regex_extract.cpp -o stri_search_regex_extract.o
g++ -std=c++11 -I/usr/local/lib/R/include -DNDEBUG -I. -UDEBUG -DNDEBUG -DU_HAVE_ELF_H -I/usr/local/include -fpic -fpic -g -O2 -c stri_search_regex_locate.cpp -o stri_search_regex_locate.o
g++ -std=c++11 -I/usr/local/lib/R/include -DNDEBUG -I. -UDEBUG -DNDEBUG -DU_HAVE_ELF_H -I/usr/local/include -fpic -fpic -g -O2 -c stri_search_regex_match.cpp -o stri_search_regex_match.o
g++ -std=c++11 -I/usr/local/lib/R/include -DNDEBUG -I. -UDEBUG -DNDEBUG -DU_HAVE_ELF_H -I/usr/local/include -fpic -fpic -g -O2 -c stri_search_regex_replace.cpp -o stri_search_regex_replace.o
g++ -std=c++11 -I/usr/local/lib/R/include -DNDEBUG -I. -UDEBUG -DNDEBUG -DU_HAVE_ELF_H -I/usr/local/include -fpic -fpic -g -O2 -c stri_search_regex_split.cpp -o stri_search_regex_split.o
g++ -std=c++11 -I/usr/local/lib/R/include -DNDEBUG -I. -UDEBUG -DNDEBUG -DU_HAVE_ELF_H -I/usr/local/include -fpic -fpic -g -O2 -c stri_search_regex_subset.cpp -o stri_search_regex_subset.o
g++ -std=c++11 -I/usr/local/lib/R/include -DNDEBUG -I. -UDEBUG -DNDEBUG -DU_HAVE_ELF_H -I/usr/local/include -fpic -fpic -g -O2 -c stri_sort.cpp -o stri_sort.o
g++ -std=c++11 -I/usr/local/lib/R/include -DNDEBUG -I. -UDEBUG -DNDEBUG -DU_HAVE_ELF_H -I/usr/local/include -fpic -fpic -g -O2 -c stri_stats.cpp -o stri_stats.o
g++ -std=c++11 -I/usr/local/lib/R/include -DNDEBUG -I. -UDEBUG -DNDEBUG -DU_HAVE_ELF_H -I/usr/local/include -fpic -fpic -g -O2 -c stri_stringi.cpp -o stri_stringi.o
g++ -std=c++11 -I/usr/local/lib/R/include -DNDEBUG -I. -UDEBUG -DNDEBUG -DU_HAVE_ELF_H -I/usr/local/include -fpic -fpic -g -O2 -c stri_sub.cpp -o stri_sub.o
g++ -std=c++11 -I/usr/local/lib/R/include -DNDEBUG -I. -UDEBUG -DNDEBUG -DU_HAVE_ELF_H -I/usr/local/include -fpic -fpic -g -O2 -c stri_test.cpp -o stri_test.o
g++ -std=c++11 -I/usr/local/lib/R/include -DNDEBUG -I. -UDEBUG -DNDEBUG -DU_HAVE_ELF_H -I/usr/local/include -fpic -fpic -g -O2 -c stri_time_zone.cpp -o stri_time_zone.o
g++ -std=c++11 -I/usr/local/lib/R/include -DNDEBUG -I. -UDEBUG -DNDEBUG -DU_HAVE_ELF_H -I/usr/local/include -fpic -fpic -g -O2 -c stri_time_calendar.cpp -o stri_time_calendar.o
g++ -std=c++11 -I/usr/local/lib/R/include -DNDEBUG -I. -UDEBUG -DNDEBUG -DU_HAVE_ELF_H -I/usr/local/include -fpic -fpic -g -O2 -c stri_time_symbols.cpp -o stri_time_symbols.o
g++ -std=c++11 -I/usr/local/lib/R/include -DNDEBUG -I. -UDEBUG -DNDEBUG -DU_HAVE_ELF_H -I/usr/local/include -fpic -fpic -g -O2 -c stri_time_format.cpp -o stri_time_format.o
g++ -std=c++11 -I/usr/local/lib/R/include -DNDEBUG -I. -UDEBUG -DNDEBUG -DU_HAVE_ELF_H -I/usr/local/include -fpic -fpic -g -O2 -c stri_trans_casemap.cpp -o stri_trans_casemap.o
g++ -std=c++11 -I/usr/local/lib/R/include -DNDEBUG -I. -UDEBUG -DNDEBUG -DU_HAVE_ELF_H -I/usr/local/include -fpic -fpic -g -O2 -c stri_trans_other.cpp -o stri_trans_other.o
g++ -std=c++11 -I/usr/local/lib/R/include -DNDEBUG -I. -UDEBUG -DNDEBUG -DU_HAVE_ELF_H -I/usr/local/include -fpic -fpic -g -O2 -c stri_trans_normalization.cpp -o stri_trans_normalization.o
g++ -std=c++11 -I/usr/local/lib/R/include -DNDEBUG -I. -UDEBUG -DNDEBUG -DU_HAVE_ELF_H -I/usr/local/include -fpic -fpic -g -O2 -c stri_trans_transliterate.cpp -o stri_trans_transliterate.o
g++ -std=c++11 -I/usr/local/lib/R/include -DNDEBUG -I. -UDEBUG -DNDEBUG -DU_HAVE_ELF_H -I/usr/local/include -fpic -fpic -g -O2 -c stri_ucnv.cpp -o stri_ucnv.o
g++ -std=c++11 -I/usr/local/lib/R/include -DNDEBUG -I. -UDEBUG -DNDEBUG -DU_HAVE_ELF_H -I/usr/local/include -fpic -fpic -g -O2 -c stri_uloc.cpp -o stri_uloc.o
g++ -std=c++11 -I/usr/local/lib/R/include -DNDEBUG -I. -UDEBUG -DNDEBUG -DU_HAVE_ELF_H -I/usr/local/include -fpic -fpic -g -O2 -c stri_utils.cpp -o stri_utils.o
g++ -std=c++11 -I/usr/local/lib/R/include -DNDEBUG -I. -UDEBUG -DNDEBUG -DU_HAVE_ELF_H -I/usr/local/include -fpic -fpic -g -O2 -c stri_wrap.cpp -o stri_wrap.o
g++ -std=c++11 -shared -L/usr/local/lib -o stringi.so stri_brkiter.o stri_collator.o stri_common.o stri_compare.o stri_container_base.o stri_container_bytesearch.o stri_container_listint.o stri_container_listraw.o stri_container_listutf8.o stri_container_regex.o stri_container_usearch.o stri_container_utf16.o stri_container_utf8.o stri_container_utf8_indexable.o stri_encoding_conversion.o stri_encoding_detection.o stri_encoding_management.o stri_escape.o stri_exception.o stri_ICU_settings.o stri_join.o stri_length.o stri_pad.o stri_prepare_arg.o stri_random.o stri_reverse.o stri_search_class_count.o stri_search_class_detect.o stri_search_class_extract.o stri_search_class_locate.o stri_search_class_replace.o stri_search_class_split.o stri_search_class_startsendswith.o stri_search_class_subset.o stri_search_class_trim.o stri_search_common.o stri_search_coll_count.o stri_search_coll_detect.o stri_search_coll_extract.o stri_search_coll_locate.o stri_search_coll_replace.o stri_search_coll_split.o stri_search_coll_startsendswith.o stri_search_coll_subset.o stri_search_boundaries_count.o stri_search_boundaries_extract.o stri_search_boundaries_locate.o stri_search_boundaries_split.o stri_search_fixed_count.o stri_search_fixed_detect.o stri_search_fixed_extract.o stri_search_fixed_locate.o stri_search_fixed_replace.o stri_search_fixed_split.o stri_search_fixed_subset.o stri_search_fixed_startsendswith.o stri_search_in.o stri_search_other_replace.o stri_search_other_split.o stri_search_regex_count.o stri_search_regex_detect.o stri_search_regex_extract.o stri_search_regex_locate.o stri_search_regex_match.o stri_search_regex_replace.o stri_search_regex_split.o stri_search_regex_subset.o stri_sort.o stri_stats.o stri_stringi.o stri_sub.o stri_test.o stri_time_zone.o stri_time_calendar.o stri_time_symbols.o stri_time_format.o stri_trans_casemap.o stri_trans_other.o stri_trans_normalization.o stri_trans_transliterate.o stri_ucnv.o stri_uloc.o stri_utils.o stri_wrap.o -licui18n -licuuc -licudata
gcc -std=gnu99 -I/usr/local/lib/R/include -DNDEBUG -I. -DNDEBUG -I/usr/local/include -fpic -g -O2 -c api.c -o api.o
gcc -std=gnu99 -I/usr/local/lib/R/include -DNDEBUG -I. -DNDEBUG -I/usr/local/include -fpic -g -O2 -c dumper.c -o dumper.o
gcc -std=gnu99 -I/usr/local/lib/R/include -DNDEBUG -I. -DNDEBUG -I/usr/local/include -fpic -g -O2 -c emitter.c -o emitter.o
gcc -std=gnu99 -I/usr/local/lib/R/include -DNDEBUG -I. -DNDEBUG -I/usr/local/include -fpic -g -O2 -c implicit.c -o implicit.o
gcc -std=gnu99 -I/usr/local/lib/R/include -DNDEBUG -I. -DNDEBUG -I/usr/local/include -fpic -g -O2 -c loader.c -o loader.o
gcc -std=gnu99 -I/usr/local/lib/R/include -DNDEBUG -I. -DNDEBUG -I/usr/local/include -fpic -g -O2 -c parser.c -o parser.o
gcc -std=gnu99 -I/usr/local/lib/R/include -DNDEBUG -I. -DNDEBUG -I/usr/local/include -fpic -g -O2 -c r-ext.c -o r-ext.o
gcc -std=gnu99 -I/usr/local/lib/R/include -DNDEBUG -I. -DNDEBUG -I/usr/local/include -fpic -g -O2 -c reader.c -o reader.o
gcc -std=gnu99 -I/usr/local/lib/R/include -DNDEBUG -I. -DNDEBUG -I/usr/local/include -fpic -g -O2 -c scanner.c -o scanner.o
gcc -std=gnu99 -I/usr/local/lib/R/include -DNDEBUG -I. -DNDEBUG -I/usr/local/include -fpic -g -O2 -c writer.c -o writer.o
gcc -std=gnu99 -shared -L/usr/local/lib -o yaml.so api.o dumper.o emitter.o implicit.o loader.o parser.o r-ext.o reader.o scanner.o writer.o
gcc -std=gnu99 -I/usr/local/lib/R/include -DNDEBUG -I/usr/local/include -fpic -g -O2 -c rawmatch.c -o rawmatch.o
gcc -std=gnu99 -shared -L/usr/local/lib -o mime.so rawmatch.o
gcc -std=gnu99 -I/usr/local/lib/R/include -DNDEBUG -I/usr/local/include -fpic -g -O2 -c Rbase64.c -o Rbase64.o
gcc -std=gnu99 -I/usr/local/lib/R/include -DNDEBUG -I/usr/local/include -fpic -g -O2 -c Rinit.c -o Rinit.o
gcc -std=gnu99 -I/usr/local/lib/R/include -DNDEBUG -I/usr/local/include -fpic -g -O2 -c Rmarkdown.c -o Rmarkdown.o
gcc -std=gnu99 -I/usr/local/lib/R/include -DNDEBUG -I/usr/local/include -fpic -g -O2 -c autolink.c -o autolink.o
gcc -std=gnu99 -I/usr/local/lib/R/include -DNDEBUG -I/usr/local/include -fpic -g -O2 -c buffer.c -o buffer.o
gcc -std=gnu99 -I/usr/local/lib/R/include -DNDEBUG -I/usr/local/include -fpic -g -O2 -c houdini_href_e.c -o houdini_href_e.o
gcc -std=gnu99 -I/usr/local/lib/R/include -DNDEBUG -I/usr/local/include -fpic -g -O2 -c houdini_html_e.c -o houdini_html_e.o
gcc -std=gnu99 -I/usr/local/lib/R/include -DNDEBUG -I/usr/local/include -fpic -g -O2 -c html.c -o html.o
gcc -std=gnu99 -I/usr/local/lib/R/include -DNDEBUG -I/usr/local/include -fpic -g -O2 -c html_smartypants.c -o html_smartypants.o
gcc -std=gnu99 -I/usr/local/lib/R/include -DNDEBUG -I/usr/local/include -fpic -g -O2 -c markdown.c -o markdown.o
gcc -std=gnu99 -I/usr/local/lib/R/include -DNDEBUG -I/usr/local/include -fpic -g -O2 -c stack.c -o stack.o
gcc -std=gnu99 -shared -L/usr/local/lib -o markdown.so Rbase64.o Rinit.o Rmarkdown.o autolink.o buffer.o houdini_href_e.o houdini_html_e.o html.o html_smartypants.o markdown.o stack.o
Command error:
** installing vignettes
** testing if installed package can be loaded
* DONE (stringr)
* installing *source* package ‘markdown’ ...
** package ‘markdown’ successfully unpacked and MD5 sums checked
** libs
installing to /usr/local/lib/R/site-library/markdown/libs
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
* DONE (markdown)
* installing *source* package ‘evaluate’ ...
** package ‘evaluate’ successfully unpacked and MD5 sums checked
** R
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (evaluate)
* installing *source* package ‘knitr’ ...
** package ‘knitr’ successfully unpacked and MD5 sums checked
** R
** demo
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
* DONE (knitr)
The downloaded source packages are in
‘/tmp/RtmpnHFkY7/downloaded_packages’
Warning message:
In library(package, lib.loc = lib.loc, character.only = TRUE, logical.return = TRUE, :
there is no package called ‘markdown’
Error in readLines(con) : cannot open the connection
Calls: markdownToHTML ... renderMarkdown -> tryCatch -> tryCatchList -> readLines
In addition: Warning message:
In readLines(con) :
cannot open file '/home/zhenyisong/.nextflow/assets/SciLifeLab/NGI-ChIPseq/docs/output.md': No such file or directory
Execution halted
.command.run.1: line 104: 13 Terminated nxf_trace "$pid" .command.trace
Work dir:
/mnt/date/biodata/data/cellChIP/heart/work/cf/43dd86bf62477c2ad0710be2bf910d
Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`You're right, it does seem to be missing. @chuan-wang - could you add this to the docker file please? I don't really understand why you're seeing all of these errors when the automated tests pass every time.. Are you running on a server without an internet connection?
chuan-wang
commented
7 years ago Installation of required R packages in PR #40. /c
On Tue, Jul 4, 2017 at 2:58 PM, Phil Ewels notifications@github.com wrote:
You're right, it does seem to be missing. @chuan-wang https://github.com/chuan-wang - could you add this to the docker file please? I don't really understand why you're seeing all of these errors when the automated tests https://travis-ci.org/SciLifeLab/NGI-ChIPseq pass every time.. Are you running on a server without an internet connection?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/SciLifeLab/NGI-ChIPseq/issues/32#issuecomment-312871609, or mute the thread https://github.com/notifications/unsubscribe-auth/ARdKNJkE2C4mHbcJ-on2jWGMT-M-dh0wks5sKjbugaJpZM4N08pR .
zhenyisong
commented
7 years ago I chekced the 'output.md', this file did exist.
[ -e /home/zhenyisong/.nextflow/assets/SciLifeLab/NGI-ChIPseq/docs/output.md ] && echo 'found' || echo 'not found'
foundI am not familiar with the grammar of Markdown R package.
Documentations in results dir.
NGI-ChIPseq execution completed unsuccessfully!
The exit status of the task that caused the workflow execution to fail was: 1.
The full error message was:
Error executing process > 'output_documentation (SRR1029886)'
Caused by:
Process `output_documentation (SRR1029886)` terminated with an error exit status (1)
Command executed:
markdown_to_html.r /home/zhenyisong/.nextflow/assets/SciLifeLab/NGI-ChIPseq/docs/output.md results_description.html
Command exit status:
1
Command output:
(empty)
Command error:
Loading required package: markdown
Error in readLines(con) : cannot open the connection
Calls: markdownToHTML ... renderMarkdown -> tryCatch -> tryCatchList -> readLines
In addition: Warning message:
In readLines(con) :
cannot open file '/home/zhenyisong/.nextflow/assets/SciLifeLab/NGI-ChIPseq/docs/output.md': No such file or directory
Execution halted
.command.run.1: line 104: 13 Terminated nxf_trace "$pid" .command.trace
Work dir:
/mnt/date/biodata/data/cellChIP/heart/work/d6/383ec681abf0a7bc8eeecac6451e18
Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`Thanks @zhenyisong - what happens if you try to run the process manually?
cd /mnt/date/biodata/data/cellChIP/heart/work/d6/383ec681abf0a7bc8eeecac6451e18
bash .command.shHmm, I think I may know what is causing this. I think it's a docker mount point issue. I'll submit a PR to see if it fixes it..
zhenyisong
commented
7 years ago @ewels
.command.sh: line 2: markdown_to_html.r: command not foundCompleted successfully in the newest commit. Thanks.
ewels
commented
7 years ago Fantastic! Got there at last... 🎉
Thanks for the testing and your persistence @zhenyisong! Just adding more proof to the old adage - you never know how good your code is until someone else tries to run it...
The results only included fastqc and trim_galore. They seemed to be finished. The bwa dir was empty and left tasks were undone.
Here is my config for docker version according my poor understanding:
Here is the log output: