Unknown error

[azzaea]

Hello,

Thank you for developing this tool. I used v0.6.6 to download some SRA data before and it was fine. Now, I'm trying to download a different dataset, and parallel-fastq-dump is strangely failing, producing logs like below. I'm not sure if it is a a parallel-fastq-dump problem, fastq-dump or prefetch problem per say, but it is repeatable on my system for all of the 84 samples in the dataset. I'm attaching the report mentioned in the log as well.

Would you have any suggestions as to what could be causing this error and how to work around it? Any pointers you may be able to provide are highly appreciated.

Best, Azza

$ parallel-fastq-dump -V
parallel-fastq-dump : 0.6.6

"fastq-dump" version 2.10.8

$ parallel-fastq-dump --sra-id SRR9070188 --threads 4 --split-files --gzip
SRR ids: ['SRR9070188']
extra args: ['--split-files', '--gzip']
tempdir: /tmp/pfd_5k4ylnch
SRR9070188 spots: 323486139
blocks: [[1, 80871534], [80871535, 161743068], [161743069, 242614602], [242614603, 323486139]]
2020-12-07T13:22:07 fastq-dump.2.10.8 err: timeout exhausted while waiting condition within process system module - failed SRR9070188

=============================================================
An error occurred during processing.
A report was generated into the file '/home/azzaea/ncbi_error_report.txt'.
If the problem persists, you may consider sending the file
to 'sra-tools@ncbi.nlm.nih.gov' for assistance.
=============================================================

2020-12-07T13:39:06 fastq-dump.2.10.8 err: transfer canceled while allocating buffer within file system module - Cannot KHttpFileTimedReadChunked: to=480
fastq-dump quit with error code 3
2020-12-07T14:08:23 fastq-dump.2.10.8 err: timeout exhausted while waiting condition within process system module - failed SRR9070188

=============================================================
An error occurred during processing.
A report was generated into the file '/home/azzaea/ncbi_error_report.txt'.
If the problem persists, you may consider sending the file
to 'sra-tools@ncbi.nlm.nih.gov' for assistance.
=============================================================

2020-12-07T14:12:23 fastq-dump.2.10.8 err: storage exhausted while writing file within file system module - system bad file descriptor error fd='6'
2020-12-07T14:12:23 fastq-dump.2.10.8 err: storage exhausted while writing file within file system module - failed SRR9070188
2020-12-07T14:12:23 fastq-dump.2.10.8 err: storage exhausted while writing file within file system module - system bad file descriptor error fd='5'
2020-12-07T14:12:23 fastq-dump.2.10.8 err: storage exhausted while writing file within file system module - failed SRR9070188
2020-12-07T14:12:23 fastq-dump.2.10.8 err: storage exhausted while writing file within file system module - system bad file descriptor error fd='5'
2020-12-07T14:12:23 fastq-dump.2.10.8 err: storage exhausted while writing file within file system module - system bad file descriptor error fd='5'
2020-12-07T14:12:23 fastq-dump.2.10.8 err: storage exhausted while writing file within file system module - system bad file descriptor error fd='6'
2020-12-07T14:12:23 fastq-dump.2.10.8 err: storage exhausted while writing file within file system module - system bad file descriptor error fd='6'

=============================================================
An error occurred during processing.
A report was generated into the file '/home/azzaea/ncbi_error_report.txt'.
If the problem persists, you may consider sending the file
to 'sra-tools@ncbi.nlm.nih.gov' for assistance.
=============================================================

=============================================================
An error occurred during processing.
A report was generated into the file '/home/azzaea/ncbi_error_report.txt'.
If the problem persists, you may consider sending the file
to 'sra-tools@ncbi.nlm.nih.gov' for assistance.
=============================================================

fastq-dump quit with error code 3
fastq-dump quit with error code 3
fastq-dump error! exit code: 3

ncbi_error_report.txt

[azzaea]

For completion also, I have configured the sra toolkit to store data in /scratch/sra- which is on external hard disk with large storage space:

$ vdb-config -i

From another attempt downloading the data more manually, prefetch is working fine and the error seems to have resulted from fastq-dump:

$ prefetch --max-size 100GB SRR9070264

2020-12-07T10:22:02 prefetch.2.10.8: 1) Downloading 'SRR9070264'...
2020-12-07T10:22:02 prefetch.2.10.8:  Downloading via HTTPS...
2020-12-07T10:22:02 prefetch.2.10.8:    Continue download of 'SRR9070264' from 37942800152
2020-12-07T13:48:11 prefetch.2.10.8:  HTTPS download succeed
2020-12-07T13:48:11 prefetch.2.10.8:   verifying 'SRR9070264'...
2020-12-07T13:56:26 prefetch.2.10.8:  'SRR9070264' is valid
2020-12-07T13:56:27 prefetch.2.10.8: 1) 'SRR9070264' was downloaded successfully
2020-12-07T13:56:27 prefetch.2.10.8: 'SRR9070264' has 0 unresolved dependencies

$ fastq-dump SRR9070264
2020-12-07T16:25:42 fastq-dump.2.10.8 err: storage exhausted while writing file within file system module - system bad file descriptor error fd='4'
2020-12-07T16:25:42 fastq-dump.2.10.8 err: storage exhausted while writing file within file system module - failed SRR9070264
2020-12-07T16:25:42 fastq-dump.2.10.8 err: storage exhausted while writing file within file system module - system bad file descriptor error fd='4'

=============================================================
An error occurred during processing.
A report was generated into the file '/home/azzaea/ncbi_error_report.txt'.
If the problem persists, you may consider sending the file
to 'sra-tools@ncbi.nlm.nih.gov' for assistance.
=============================================================

fastq-dump quit with error code 3

[rvalieris]

hello, this message says you are running out of space: fastq-dump.2.10.8 err: storage exhausted while writing file within file system module

are you executing inside /scratch/sra/ ? also set --tmpdir to use the scratch, otherwise it will use /tmp