JohnHadish
commented
3 years ago Tested on Kamiak using the Slurm Scheduler. Found the following Issues:
1) Unexpected behavior in failed_runs.metadata.txt
There is unexpected behavior in the failed_runs.metadata.txt file, it appears to be reporting that other files with the same SRX number were not imported, but does not report on the actual "Corrupted" run (SRR3461660).
The failed_runs.metadata.txt file:
{
"SRR3461661": "Notice: the run, SRR3461661, is part of experiment SRX1734704 but was not included in the input SRA_IDS.txt file. Do you want to include this run?",
"SRR3461659": "Notice: the run, SRR3461659, is part of experiment SRX1734704 but was not included in the input SRA_IDS.txt file. Do you want to include this run?",
"SRR3461658": "Notice: the run, SRR3461658, is part of experiment SRX1734704 but was not included in the input SRA_IDS.txt file. Do you want to include this run?",
"SRR3461657": "Notice: the run, SRR3461657, is part of experiment SRX1734704 but was not included in the input SRA_IDS.txt file. Do you want to include this run?",
"SRR3461656": "Notice: the run, SRR3461656, is part of experiment SRX1734704 but was not included in the input SRA_IDS.txt file. Do you want to include this run?",
"SRR3461670": "Notice: the run, SRR3461670, is part of experiment SRX1734704 but was not included in the input SRA_IDS.txt file. Do you want to include this run?",
"SRR3461669": "Notice: the run, SRR3461669, is part of experiment SRX1734704 but was not included in the input SRA_IDS.txt file. Do you want to include this run?",
"SRR3461668": "Notice: the run, SRR3461668, is part of experiment SRX1734704 but was not included in the input SRA_IDS.txt file. Do you want to include this run?",
"SRR3461667": "Notice: the run, SRR3461667, is part of experiment SRX1734704 but was not included in the input SRA_IDS.txt file. Do you want to include this run?",
"SRR3461666": "Notice: the run, SRR3461666, is part of experiment SRX1734704 but was not included in the input SRA_IDS.txt file. Do you want to include this run?",
"SRR3461665": "Notice: the run, SRR3461665, is part of experiment SRX1734704 but was not included in the input SRA_IDS.txt file. Do you want to include this run?",
"SRR3461664": "Notice: the run, SRR3461664, is part of experiment SRX1734704 but was not included in the input SRA_IDS.txt file. Do you want to include this run?",
"SRR3461663": "Notice: the run, SRR3461663, is part of experiment SRX1734704 but was not included in the input SRA_IDS.txt file. Do you want to include this run?",
"SRR3461662": "Notice: the run, SRR3461662, is part of experiment SRX1734704 but was not included in the input SRA_IDS.txt file. Do you want to include this run?",
"SRR2927686": "Metadata was not returned by NCBI for this run."
}2) Directory with Actual Run from SRA (SRR649944 from SRX218012) has failed run files which are almost empty
Here is the directory:
total 2136
-rw-rw-r-- 2 john.hadish its_p_sys_ur_kam-ficklin 671859 Mar 25 12:10 SRX218012_1_fastqc.html
-rw-rw-r-- 2 john.hadish its_p_sys_ur_kam-ficklin 402113 Mar 25 12:10 SRX218012_1_fastqc.zip
-rw-rw-r-- 2 john.hadish its_p_sys_ur_kam-ficklin 679856 Mar 25 12:10 SRX218012_2_fastqc.html
-rw-rw-r-- 2 john.hadish its_p_sys_ur_kam-ficklin 403427 Mar 25 12:10 SRX218012_2_fastqc.zip
-rw-rw-r-- 2 john.hadish its_p_sys_ur_kam-ficklin 2 Mar 25 12:08 SRX218012.failed_runs.download.txt
-rw-rw-r-- 2 john.hadish its_p_sys_ur_kam-ficklin 2 Mar 25 12:08 SRX218012.failed_runs.fastq-dump.txt
-rw-rw-r-- 2 john.hadish its_p_sys_ur_kam-ficklin 592 Mar 25 12:10 SRX218012.kallisto.log
-rw-rw-r-- 2 john.hadish its_p_sys_ur_kam-ficklin 24 Mar 25 12:11 SRX218012_vs_CORG.Kallisto.raw
-rw-rw-r-- 2 john.hadish its_p_sys_ur_kam-ficklin 39 Mar 25 12:11 SRX218012_vs_CORG.Kallisto.tpmEverything appears to have ran fine, but there are two unexpected files (SRX218012.failed_runs.download.txt and SRX218012.failed_runs.fastq-dump.txt ) that only contain a {}
The SRX1734704/SRX1734704.failed_runs.fastq-dump.txt file:
{
"SRR3461660": "2021-03-25T19:10:37 fastq-dump.2.10.0 err: data corrupt while executing function within virtual database module - failed SRR3461660.sra\n\n=============================================================\nAn error occurred during processing.\nA report was generated into the file '/home/john.hadish/ncbi_error_report.txt'.\nIf the problem persists, you may consider sending the file\nto 'sra-tools@ncbi.nlm.nih.gov' for assistance.\n=============================================================\n\n"The ``
ll
total 4
-rw-rw-r-- 2 john.hadish its_p_sys_ur_kam-ficklin 2 Mar 25 12:09 SRX1734704.failed_runs.download.txt
-rw-rw-r-- 2 john.hadish its_p_sys_ur_kam-ficklin 519 Mar 25 12:10 SRX1734704.failed_runs.fastq-dump.txt
3 Output Directory Created for SRX1734704 (containing SRR3461660) even though it failed. Has 2 files:
The directory looks like this:
-rw-rw-r-- 2 john.hadish its_p_sys_ur_kam-ficklin 2 Mar 25 12:09 SRX1734704.failed_runs.download.txt
-rw-rw-r-- 2 john.hadish its_p_sys_ur_kam-ficklin 519 Mar 25 12:10 SRX1734704.failed_runs.fastq-dump.txtwith the first file only containing a {} and the second file containing:
{
"SRR3461660": "2021-03-25T19:10:37 fastq-dump.2.10.0 err: data corrupt while executing function within virtual database module - failed SRR3461660.sra\n\n=============================================================\nAn error occurred during processing.\nA report was generated into the file '/home/john.hadish/ncbi_error_report.txt'.\nIf the problem persists, you may consider sending the file\nto 'sra-tools@ncbi.nlm.nih.gov' for assistance.\n=============================================================\n\n"
}You will remember from issue 1 that this run (SRR3461660) was not reported in the failed_runs.metadata.txt file
4 There is no failed_runs_report.py
I think you forgot to push it probably. Process fails because there is no file for it.
spficklin
I apologize for the size of this PR. I didn't mean for it to get so big, but I just kept fixing issues trying to get the Arabidopsis 26k to run with GEMmaker.
Issue #193 (cleanup of files) Issue #173 (missing data on NCBI) And other issues I found
Description
I've been having trouble running with the
main.nf17K arabidopsis samples (26K runs) on Kamiak and though many days of restarting the workflow I've resolved several bugs and problems.Here are examples of problems
Problem 1: During the sra2fastq process.
The SRA is downloaded just fine from NCBI but is corrupted
Solution: see the fix described for problem 4 below.
Problem 2: Too many open files
This problem seemed to be caused by file sockets not being closed.
Solution: I think this was caused by not setting the lock file to NULL in the groovy code of the
next_sampleprocess. I'm not 100% sure, but so far after doing this I have not encountered the problem again.Problem 3: Files Not Being Cleaned up.
Both in the
sra2fastq.pyscript and theretreive_sra.pyscript there was a cleanup step if a job failed and the cleanup was not happening. So, files could build up if there was continuous failures.Solution: See the fix for problem 4 below.
Problem 4: Failed Runs Stop Everything
If the
sra2fastq.pyorretrieve_sra.pyscripts failed then the entire workflow stops. With 26K samples that becomes too time consuming to keep up with whenever a sample is missing from NCBI or is corrupted through no fault of GEMmaker.Solution: Neither the
sra2fastq.pyor theretrieve_sra.pyscripts fail if a sample fails. Instead both scripts now do the following.sample_failedindicating to GEMmaker that this sample should not continue.sample_failedfile. If present then it automatically triggers thenext_sampleprocess and does not proceed with the next steps.failed_run_report. It checks all of the files described in item 1 above and generate an HTML report that lists all of the failed runs and the reason for it. The following is a screenshot of the report.Problem 5: Bug in
retrieve_sra.pyThe
retrieve_sra.pyhad a bug in it that prevented the https:// protocol from being used as a second attempt if Aspera failed.Solution: It was a problem in checking the return code of Aspera and the code was fixed.
Notes:
Testing?
For Problems 1, 3 and 4:
You can test these fixes by adding the following to your SRA_IDs file and just running GEMmaker with the local defaults:
For problems 2:
I don't know how to offer a way to test this. We'll just have to wait and see if it happens again.
For problem 5:
I don't have a way to test this either. Just take a look at the code in the
retrieve_sra.pyfile where the code switches to the https protocol if Aspera failed and make sure it makes sense to you.