wdecoster / NanoPlot

Plotting scripts for long read sequencing data
http://nanoplot.bioinf.be
MIT License
429 stars 47 forks source link

NanoPlot error #100

Closed rmsaha closed 6 years ago

rmsaha commented 6 years ago

Thanks for the NanoPlot tool. It is very helpful! I am trying to QC a PromethION run with NanoPlot using summary files as input. It keeps crashing for some reason. It is working for other runs.

Below is the logfile

2018-10-09 16:45:27,043 NanoPlot 1.14.1 started with arguments Namespace(N50=False, alength=False, bam=None, barcoded=False, color='#4CB391', cram=None, downsample=None, drop_outliers=False, fasta=None, fastq=None, fastq_minimal=None, fastq_rich=None, format='png', listcolors=False, loglength=True, maxlength=None, minlength=None, minqual=None, no_N50=False, outdir='summary-plots-unfiltered', percentqual=False, pickle=None, plots=['kde', 'dot'], prefix='', raw=False, readtype='1D', runtime_until=None, store=False, summary=['merged_seq_summary_PROM14.txt'], threads=4, title=None, verbose=False) 2018-10-09 16:45:27,043 Python version is: 3.6.2 (default, Sep 20 2017, 14:18:06) [GCC 4.8.5] 2018-10-09 16:45:27,046 Nanoplotter: valid output format png 2018-10-09 16:45:27,059 Nanoget: Staring to collect statistics from summary file merged_seq_summary_PROM14.txt 2018-10-09 16:45:27,060 Nanoget: Collecting statistics for 1D sequencing 2018-10-09 16:47:25,721 Nanoget: Finished collecting statistics from summary file merged_seq_summary_PROM14.txt 2018-10-09 16:47:47,229 A process in the process pool was terminated abruptly while the future was running or pending. Traceback (most recent call last): File "/home/dnat/tools/nanoplot/nanoplot/NanoPlot.py", line 63, in main barcoded=args.barcoded) File "/home/dnat/tools/nanoplot/nanoget/nanoget.py", line 74, in get_input dfs=[out for out in executor.map(extration_function, files)], File "/home/dnat/tools/nanoplot/nanoget/nanoget.py", line 74, in dfs=[out for out in executor.map(extration_function, files)], File "/software/python/3.6.2/x86_64-linux-ubuntu14.04/lib/python3.6/concurrent/futures/_base.py", line 556, in result_iterator yield future.result() File "/software/python/3.6.2/x86_64-linux-ubuntu14.04/lib/python3.6/concurrent/futures/_base.py", line 405, in result return self.__get_result() File "/software/python/3.6.2/x86_64-linux-ubuntu14.04/lib/python3.6/concurrent/futures/_base.py", line 357, in __get_result raise self._exception concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Thank you for your help!

wdecoster commented 6 years ago

Hi,

Did you get any more error messages in the terminal while this happened? Could you update NanoPlot and try again, and if that doesn't work do you think you can share your input file with me?

Cheers, Wouter

rmsaha commented 6 years ago

Thanks Wouter for your quick reply.

This is the message I get in the terminal

Process Process-1: Traceback (most recent call last): File "/software/python/3.6.2/x86_64-linux-ubuntu14.04/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap self.run() File "/software/python/3.6.2/x86_64-linux-ubuntu14.04/lib/python3.6/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) File "/software/python/3.6.2/x86_64-linux-ubuntu14.04/lib/python3.6/concurrent/futures/process.py", line 181, in _process_worker result=r)) File "/software/python/3.6.2/x86_64-linux-ubuntu14.04/lib/python3.6/multiprocessing/queues.py", line 354, in put self._writer.send_bytes(obj) File "/software/python/3.6.2/x86_64-linux-ubuntu14.04/lib/python3.6/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/software/python/3.6.2/x86_64-linux-ubuntu14.04/lib/python3.6/multiprocessing/connection.py", line 393, in _send_bytes header = struct.pack("!i", n) struct.error: 'i' format requires -2147483648 <= number <= 2147483647

If you read this then NanoPlot has crashed :-( Please report this issue at https://github.com/wdecoster/NanoPlot/issues If you include the log file that would be really helpful. Thanks!

Traceback (most recent call last): File "/home/dnat/tools/nanoplot/nanoplot/NanoPlot.py", line 490, in main() File "/home/dnat/tools/nanoplot/nanoplot/NanoPlot.py", line 63, in main barcoded=args.barcoded) File "/home/dnat/tools/nanoplot/nanoget/nanoget.py", line 74, in get_input dfs=[out for out in executor.map(extration_function, files)], File "/home/dnat/tools/nanoplot/nanoget/nanoget.py", line 74, in dfs=[out for out in executor.map(extration_function, files)], File "/software/python/3.6.2/x86_64-linux-ubuntu14.04/lib/python3.6/concurrent/futures/_base.py", line 556, in result_iterator yield future.result() File "/software/python/3.6.2/x86_64-linux-ubuntu14.04/lib/python3.6/concurrent/futures/_base.py", line 405, in result return self.__get_result() File "/software/python/3.6.2/x86_64-linux-ubuntu14.04/lib/python3.6/concurrent/futures/_base.py", line 357, in __get_result raise self._exception concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

I will update NanoPlot and try to run it again.

Thanks! Ruta

wdecoster commented 6 years ago

I suspect that your summary file is huge, therefore crashing NanoPlot. I have made changes recently that could solve this issue.

biowackysci commented 6 years ago

If you include the log file that would be really helpful. Thanks!

concurrent.futures.process._RemoteTraceback: """ Traceback (most recent call last): File "/usr/local/python/3.6.2/lib/python3.6/concurrent/futures/process.py", line 175, in _process_worker r = call_item.fn(*call_item.args, *call_item.kwargs) File "/usr/local/python/3.6.2/lib/python3.6/concurrent/futures/process.py", line 153, in _process_chunk return [fn(args) for args in chunk] File "/usr/local/python/3.6.2/lib/python3.6/concurrent/futures/process.py", line 153, in return [fn(*args) for args in chunk] File "/usr/local/python/3.6.2/lib/python3.6/site-packages/nanoget/nanoget.py", line 398, in process_fastq_plain data=[res for res in extract_from_fastq(inputfastq) if res], File "/usr/local/python/3.6.2/lib/python3.6/site-packages/nanoget/nanoget.py", line 398, in data=[res for res in extract_from_fastq(inputfastq) if res], File "/usr/local/python/3.6.2/lib/python3.6/site-packages/nanoget/nanoget.py", line 408, in extract_from_fastq for rec in SeqIO.parse(fq, "fastq"): File "/usr/local/python/3.6.2/lib/python3.6/site-packages/Bio/SeqIO/init.py", line 611, in parse for r in i: File "/usr/local/python/3.6.2/lib/python3.6/site-packages/Bio/SeqIO/QualityIO.py", line 1033, in FastqPhredIterator for title_line, seq_string, quality_string in FastqGeneralIterator(handle): File "/usr/local/python/3.6.2/lib/python3.6/site-packages/Bio/SeqIO/QualityIO.py", line 897, in FastqGeneralIterator line = handle_readline() File "/usr/local/python/3.6.2/lib/python3.6/codecs.py", line 321, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte """

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/local/python/3.6.2/bin/NanoPlot", line 11, in load_entry_point('NanoPlot==1.13.0', 'console_scripts', 'NanoPlot')() File "/usr/local/python/3.6.2/lib/python3.6/site-packages/nanoplot/NanoPlot.py", line 62, in main barcoded=args.barcoded) File "/usr/local/python/3.6.2/lib/python3.6/site-packages/nanoget/nanoget.py", line 74, in get_input dfs=[out for out in executor.map(extration_function, files)], File "/usr/local/python/3.6.2/lib/python3.6/site-packages/nanoget/nanoget.py", line 74, in dfs=[out for out in executor.map(extration_function, files)], File "/usr/local/python/3.6.2/lib/python3.6/concurrent/futures/_base.py", line 556, in result_iterator yield future.result() File "/usr/local/python/3.6.2/lib/python3.6/concurrent/futures/_base.py", line 405, in result return self.get_result() File "/usr/local/python/3.6.2/lib/python3.6/concurrent/futures/_base.py", line 357, in get_result raise self._exception UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

I am trying to do a QC plot with fastq files and I get this. So this the command that i use NanoPlot --fastq all_promethION_reads.fastq --N50 --loglength -o NanoPlot_QC_dev3 -t 12 Can someone help me please ? thanks Saila

wdecoster commented 6 years ago

Please open a separate thread for this. Is your fastq file gzipped?

biowackysci commented 6 years ago

Yes it was gzipped and i have gunzipped it and now when I give the Nanoplot command it sits for hours together and does not do anything

wdecoster commented 6 years ago

Please open a separate thread for this.

rmsaha commented 6 years ago

NanPlot version 1.18.2 fixed the issue. Thanks for your help! Ruta

wdecoster commented 6 years ago

Thanks for confirming!