search

kfuku52 / amalgkit

RNA-seq data amalgamation for a large-scale evolutionary transcriptomics
BSD 3-Clause "New" or "Revised" License
7 stars 1 forks source link

NameError: name 'work_dir' is not defined #128

Closed kfuku52 closed 1 year ago

kfuku52 commented 1 year ago 
Traceback (most recent call last):
  File "/home/kfuku/miniconda3/bin/amalgkit", line 372, in <module>
    args.handler(args)
  File "/home/kfuku/miniconda3/bin/amalgkit", line 43, in command_quant
    quant_main(args)
  File "/home/kfuku/miniconda3/lib/python3.9/site-packages/amalgkit/quant.py", line 191, in quant_main
    run_quant(args, metadata, sra_id, index)
  File "/home/kfuku/miniconda3/lib/python3.9/site-packages/amalgkit/quant.py", line 111, in run_quant
    sys.stderr.write('getfastq output not found in: {}, layout = {}\n'.format(work_dir, sra_stat['layout']))
NameError: name 'work_dir' is not defined
kfuku52 commented 1 year ago

--batch 1718

kfuku52 commented 1 year ago

full stdout:

Started at Wed May 31 03:33:51 JST 2023
AMALGKIT version: 0.9.11
AMALGKIT command: /home/kfuku/miniconda3/bin/amalgkit getfastq --out_dir /lustre7/home/lustre4/kfuku/my_project/evolutionary_transcriptomics/20230527_gfe_pipeline/gfe_data --threads 4 --batch 1718 --redo no
AMALGKIT bug report: https://github.com/kfuku52/amalgkit/issues
amalgkit getfastq: start
pigz found. It will be used for compression/decompression in read name formatting.
2023-05-31 03:33:52.428040: Loading metadata from: /lustre7/home/lustre4/kfuku/my_project/evolutionary_transcriptomics/20230527_gfe_pipeline/gfe_data/metadata/metadata.tsv
--batch is specified. Processing one SRA per job.
This is 1,718th job. In total, 4,302 jobs will be necessary for this metadata table. 0 SRAs were excluded from the table (is_sampled==no).
Individual SRA size of SRR499409: 287,509,530.0 bp
Number of SRAs to be processed: 1
Total target size (--max_bp): 999,999,999,999,999 bp
The sum of SRA sizes: 287,509,530.0 bp
Target size per SRA: 999,999,999,999,999 bp

Processing SRA ID: SRR499409
Library layout: single
Number of reads: 8,712,410
Single/Paired read length: 33 bp
Total bases: 287,509,530 bp
Processing SRR499409 as publicly available data from SRA.
Previously-downloaded sra file was not detected. New sra file will be downloaded.
Trying to fetch SRR499409 from AWS: https://sra-pub-run-odp.s3.amazonaws.com/sra/SRR499409/SRR499409
SRA file was downloaded with urllib.request from AWS
Total sampled bases: 287,509,530 bp
Command: parallel-fastq-dump -t 4 --minReadLen 25 --qual-filter-1 --skip-technical --split-3 --clip --gzip --outdir /lustre7/home/lustre4/kfuku/my_project/evolutionary_transcriptomics/20230527_gfe_pipeline/gfe_data/getfastq/SRR499409 --tmpdir /lustre7/home/lustre4/kfuku/my_project/evolutionary_transcriptomics/20230527_gfe_pipeline/gfe_data/getfastq/SRR499409 --minSpotId 1 --maxSpotId 8712410 -s /lustre7/home/lustre4/kfuku/my_project/evolutionary_transcriptomics/20230527_gfe_pipeline/gfe_data/getfastq/SRR499409/SRR499409.sra
parallel-fastq-dump stdout:
Rejected 2178102 READS because of Quality-Filtering
Read 2178102 spots for /lustre7/home/lustre4/kfuku/my_project/evolutionary_transcriptomics/20230527_gfe_pipeline/gfe_data/getfastq/SRR499409/SRR499409.sra
Written 0 spots for /lustre7/home/lustre4/kfuku/my_project/evolutionary_transcriptomics/20230527_gfe_pipeline/gfe_data/getfastq/SRR499409/SRR499409.sra
Rejected 2178104 READS because of Quality-Filtering
Read 2178104 spots for /lustre7/home/lustre4/kfuku/my_project/evolutionary_transcriptomics/20230527_gfe_pipeline/gfe_data/getfastq/SRR499409/SRR499409.sra
Written 0 spots for /lustre7/home/lustre4/kfuku/my_project/evolutionary_transcriptomics/20230527_gfe_pipeline/gfe_data/getfastq/SRR499409/SRR499409.sra
Rejected 2178102 READS because of Quality-Filtering
Read 2178102 spots for /lustre7/home/lustre4/kfuku/my_project/evolutionary_transcriptomics/20230527_gfe_pipeline/gfe_data/getfastq/SRR499409/SRR499409.sra
Written 0 spots for /lustre7/home/lustre4/kfuku/my_project/evolutionary_transcriptomics/20230527_gfe_pipeline/gfe_data/getfastq/SRR499409/SRR499409.sra
Rejected 2178102 READS because of Quality-Filtering
Read 2178102 spots for /lustre7/home/lustre4/kfuku/my_project/evolutionary_transcriptomics/20230527_gfe_pipeline/gfe_data/getfastq/SRR499409/SRR499409.sra
Written 0 spots for /lustre7/home/lustre4/kfuku/my_project/evolutionary_transcriptomics/20230527_gfe_pipeline/gfe_data/getfastq/SRR499409/SRR499409.sra

parallel-fastq-dump stderr:
2023-05-31 03:34:08,223 - SRR ids: ['/lustre7/home/lustre4/kfuku/my_project/evolutionary_transcriptomics/20230527_gfe_pipeline/gfe_data/getfastq/SRR499409/SRR499409.sra']
2023-05-31 03:34:08,223 - extra args: ['--minReadLen', '25', '--qual-filter-1', '--skip-technical', '--split-3', '--clip', '--gzip']
2023-05-31 03:34:08,224 - tempdir: /lustre7/home/lustre4/kfuku/my_project/evolutionary_transcriptomics/20230527_gfe_pipeline/gfe_data/getfastq/SRR499409/pfd_qm427oan
2023-05-31 03:34:08,224 - CMD: sra-stat --meta --quick /lustre7/home/lustre4/kfuku/my_project/evolutionary_transcriptomics/20230527_gfe_pipeline/gfe_data/getfastq/SRR499409/SRR499409.sra
2023-05-31 03:34:08,257 - /lustre7/home/lustre4/kfuku/my_project/evolutionary_transcriptomics/20230527_gfe_pipeline/gfe_data/getfastq/SRR499409/SRR499409.sra spots: 8712410
2023-05-31 03:34:08,257 - blocks: [[1, 2178102], [2178103, 4356204], [4356205, 6534306], [6534307, 8712410]]
2023-05-31 03:34:08,257 - CMD: fastq-dump -N 1 -X 2178102 -O /lustre7/home/lustre4/kfuku/my_project/evolutionary_transcriptomics/20230527_gfe_pipeline/gfe_data/getfastq/SRR499409/pfd_qm427oan/0 --minReadLen 25 --qual-filter-1 --skip-technical --split-3 --clip --gzip /lustre7/home/lustre4/kfuku/my_project/evolutionary_transcriptomics/20230527_gfe_pipeline/gfe_data/getfastq/SRR499409/SRR499409.sra
2023-05-31 03:34:08,259 - CMD: fastq-dump -N 2178103 -X 4356204 -O /lustre7/home/lustre4/kfuku/my_project/evolutionary_transcriptomics/20230527_gfe_pipeline/gfe_data/getfastq/SRR499409/pfd_qm427oan/1 --minReadLen 25 --qual-filter-1 --skip-technical --split-3 --clip --gzip /lustre7/home/lustre4/kfuku/my_project/evolutionary_transcriptomics/20230527_gfe_pipeline/gfe_data/getfastq/SRR499409/SRR499409.sra
2023-05-31 03:34:08,261 - CMD: fastq-dump -N 4356205 -X 6534306 -O /lustre7/home/lustre4/kfuku/my_project/evolutionary_transcriptomics/20230527_gfe_pipeline/gfe_data/getfastq/SRR499409/pfd_qm427oan/2 --minReadLen 25 --qual-filter-1 --skip-technical --split-3 --clip --gzip /lustre7/home/lustre4/kfuku/my_project/evolutionary_transcriptomics/20230527_gfe_pipeline/gfe_data/getfastq/SRR499409/SRR499409.sra
2023-05-31 03:34:08,262 - CMD: fastq-dump -N 6534307 -X 8712410 -O /lustre7/home/lustre4/kfuku/my_project/evolutionary_transcriptomics/20230527_gfe_pipeline/gfe_data/getfastq/SRR499409/pfd_qm427oan/3 --minReadLen 25 --qual-filter-1 --skip-technical --split-3 --clip --gzip /lustre7/home/lustre4/kfuku/my_project/evolutionary_transcriptomics/20230527_gfe_pipeline/gfe_data/getfastq/SRR499409/SRR499409.sra

Time elapsed for 1st-round sequence extraction: SRR499409, 25.0 sec

--- getfastq 1st-round sequence generation report ---
Individual target size: 999,999,999,999,999 bp
Sum of fastq_dump dumped reads: 287,509,530 bp
Sum of fastq_dump rejected reads: 287,509,530 bp
Sum of fastq_dump written reads: 0 bp
Sum of fastp input reads: 0 bp
Sum of fastp output reads: 0 bp

nan% of reads were obtained in the 1st-round sequence generation: 0 bp out of the individual target amount of 999,999,999,999,999 bp
Time elapsed for 1st-round sequence extraction: SRR499409, 10.5 sec

Enough data were obtained in the 1st-round sequence extraction. Proceeding without the 2nd round.
2nd round read extraction improved % bp from 0.00% to 0.00%

Deleting: /lustre7/home/lustre4/kfuku/my_project/evolutionary_transcriptomics/20230527_gfe_pipeline/gfe_data/getfastq/SRR499409/SRR499409.sra

--- getfastq final report ---
Target size (--max_bp): 999,999,999,999,999 bp
Sum of fastq_dump dumped reads: 287,509,530 bp
Sum of fastq_dump rejected reads: 287,509,530 bp
Sum of fastq_dump written reads: 0 bp
Sum of fastp input reads: 0 bp
Sum of fastp output reads: 0 bp
Individual SRA IDs: SRR499409
Individual fastq_dump dumped reads (bp): 287,509,530
Individual fastq_dump rejected reads (bp): 287,509,530
Individual fastq_dump written reads (bp): 0
Individual fastp input reads (bp): 0
Individual fastp output reads (bp): 0

Time elapsed: 26 sec
amalgkit getfastq: end
AMALGKIT version: 0.9.11
AMALGKIT command: /home/kfuku/miniconda3/bin/amalgkit quant --out_dir /lustre7/home/lustre4/kfuku/my_project/evolutionary_transcriptomics/20230527_gfe_pipeline/gfe_data --threads 4 --clean_fastq yes --build_index no --batch 1718 --redo no
AMALGKIT bug report: https://github.com/kfuku52/amalgkit/issues
amalgkit quant: start
2023-05-31 03:34:19.592972: Loading metadata from: /lustre7/home/lustre4/kfuku/my_project/evolutionary_transcriptomics/20230527_gfe_pipeline/gfe_data/metadata/metadata.tsv
--batch is specified. Processing one SRA per job.
This is 1,718th job. In total, 4,302 jobs will be necessary for this metadata table. 0 SRAs were excluded from the table (is_sampled==no).

Species: Glycine soja
SRA Run ID: SRR499409
Looking for index folder in  /lustre7/home/lustre4/kfuku/my_project/evolutionary_transcriptomics/20230527_gfe_pipeline/gfe_data
Kallisto index file found: /lustre7/home/lustre4/kfuku/my_project/evolutionary_transcriptomics/20230527_gfe_pipeline/gfe_data/index/Glycine_soja.idx
Output file was not detected: /lustre7/home/lustre4/kfuku/my_project/evolutionary_transcriptomics/20230527_gfe_pipeline/gfe_data/quant/SRR499409/SRR499409_abundance.tsv
Ended at Wed May 31 03:34:19 JST 2023
kfuku52 commented 1 year ago

Sum of fastq_dump dumped reads: 287,509,530 bp Sum of fastq_dump rejected reads: 287,509,530 bp

All reads were rejected by fastq-dump. This is mRNA data, and the read length may be too short. https://www.ncbi.nlm.nih.gov/sra/SRR499409