rpetit3
commented
1 year ago So good news and bad news...
Good news first:
fastq-dl --accession SRR25316086 --verbose --outdir sra-normalized --provider sra --only-provider
...
2023-08-02 22:40:31 DEBUG 2023-08-02 22:40:31:executor.process:DEBUG - Got return code 0 from synchronous process (bash -c 'prefetch SRR25316086 --max-size 10T -o SRR25316086.sra'). __init__.py:1638
DEBUG 2023-08-02 22:40:31:root:DEBUG - fastq_dl.py:92
DEBUG 2023-08-02 22:40:31:root:DEBUG - 2023-08-02T22:40:25 prefetch.3.0.6: Current preference is set to retrieve SRA Normalized Format files with full base quality scores. fastq_dl.py:93
2023-08-02T22:40:25 prefetch.3.0.6: 1) Downloading 'SRR25316086'...
2023-08-02T22:40:25 prefetch.3.0.6: SRA Normalized Format file is being retrieved, if this is different from your preference, it may be due to current file availability.
2023-08-02T22:40:25 prefetch.3.0.6: Downloading via HTTPS...
2023-08-02T22:40:30 prefetch.3.0.6: HTTPS download succeed
2023-08-02T22:40:31 prefetch.3.0.6: 'SRR25316086' is valid
2023-08-02T22:40:31 prefetch.3.0.6: 1) 'SRR25316086' was downloaded successfully
2023-08-02T22:40:31 prefetch.3.0.6: 'SRR25316086' has 0 unresolved dependencies
...
zcat sra-normalized/SRR25316086_1.fastq.gz | fastq-scan -q
{
"qc_stats": {
"total_bp": 150934381,
"coverage": 0.00,
"read_total": 1010912,
"read_min": 35,
"read_mean": 149.305,
"read_std": 9.49214,
"read_median": 151,
"read_max": 151,
"read_25th": 150,
"read_75th": 151,
"qual_min": 2,
"qual_mean": 36.9532,
"qual_std": 2.18818,
"qual_max": 38,
"qual_median": 38,
"qual_25th": 37,
"qual_75th": 38
}
}Q -scores range from 2-38, so this should get you want you need. However I can add a way to allow the user to switch between SRA Normalized and SRA Lite, with Normalized being the default.
Now the bad news:
fastq-dl --accession SRR25316086 --verbose --outdir ena
...
zcat ena/SRR25316086_1.fastq.gz | fastq-scan -q
{
"qc_stats": {
"total_bp": 150934381,
"coverage": 0.00,
"read_total": 1010912,
"read_min": 35,
"read_mean": 149.305,
"read_std": 9.49214,
"read_median": 151,
"read_max": 151,
"read_25th": 150,
"read_75th": 151,
"qual_min": 30,
"qual_mean": 30,
"qual_std": 0,
"qual_max": 30,
"qual_median": 30,
"qual_25th": 30,
"qual_75th": 30
}
}
# Force SRA Lite
vdb-config --simplified-quality-scores yes
fastq-dl --accession SRR25316086 --verbose --outdir sra-lite --provider sra --only-provider
...
DEBUG 2023-08-02 22:44:13:root:DEBUG - 2023-08-02T22:44:11 prefetch.3.0.6: Current preference is set to retrieve SRA Lite files with simplified base quality scores. fastq_dl.py:93
2023-08-02T22:44:11 prefetch.3.0.6: 1) Downloading 'SRR25316086.lite'...
2023-08-02T22:44:11 prefetch.3.0.6: SRA Lite file is being retrieved, if this is different from your preference, it may be due to current file availability.
2023-08-02T22:44:11 prefetch.3.0.6: Downloading via HTTPS...
2023-08-02T22:44:12 prefetch.3.0.6: HTTPS download succeed
2023-08-02T22:44:13 prefetch.3.0.6: 'SRR25316086.lite' is valid
2023-08-02T22:44:13 prefetch.3.0.6: 1) 'SRR25316086.lite' was downloaded successfully
2023-08-02T22:44:13 prefetch.3.0.6: 'SRR25316086' has 0 unresolved dependencies
...
zcat sra-lite/SRR25316086_1.fastq.gz | fastq-scan -q
{
"qc_stats": {
"total_bp": 150934381,
"coverage": 0.00,
"read_total": 1010912,
"read_min": 35,
"read_mean": 149.305,
"read_std": 9.49214,
"read_median": 151,
"read_max": 151,
"read_25th": 150,
"read_75th": 151,
"qual_min": 30,
"qual_mean": 30,
"qual_std": 0,
"qual_max": 30,
"qual_median": 30,
"qual_25th": 30,
"qual_75th": 30
}
}It looks like ENA synced the SRA Lite version of the reads, and not the Normalized. This was also the case for SRR13086318.
Hmmm, this bugs me because I usually use ENA as the default provider because they provide FASTQs directly. But I also want the original quality scores which SRA sync'd reads may or may not provide. The blog post above has a October 2021 date, so I'm unsure if after this date the reads synced from SRA to ENA have the SRA Lite Q scores.
I'm wondering if a solution might be to add a third provider: source and based on the accession download from the original provider (e.g. SRR from SRA, ERR from ENA, DRR either SRA or ENA).
mbhall88
kapsakcj
I've hit an odd issue where
fastq-dlpulls FASTQs without issue, but they are in SRA Lite format instead of the typical SRA Normalized format.FASTQs in SRA Lite format have
?for all Qscores for all bases, which equates to Q30. This leads to issues wheretrimmomaticor other typical downstream softwares are unable to detect the Phred quality encoding and the Qscore are not useful during assembly (and probably other applications that utilize the Qscores)FASTQs in SRA Normalized are the original format that contains the full base quality scores
Some examples where I encountered this issue
I'm guessing it will be a big effort, but would it be possible for
fastq-dlto download the SRA-normalized format of FASTQs?Not sure how ENA deals with this issue, but sra-toolkit has an option for using this format
More info: