Open corneliusroemer opened 1 year ago
Getting an error (that incidentally should also be handled better)
$ augur curate passthru --fasta data/gisaid.fasta.zst --metadata data/gisaid_metadata.tsv --seq-field seqfield --seq-id-column strain | zstd -c > data/gisaid.ndjson.zst
Traceback (most recent call last):
File "/opt/homebrew/Caskroom/miniforge/base/envs/nextstrain/lib/python3.10/site-packages/augur/__init__.py", line 66, in run
return args.__command__.run(args)
File "/opt/homebrew/Caskroom/miniforge/base/envs/nextstrain/lib/python3.10/site-packages/augur/curate/__init__.py", line 188, in run
dump_ndjson(modified_records)
File "/opt/homebrew/Caskroom/miniforge/base/envs/nextstrain/lib/python3.10/site-packages/augur/io/json.py", line 63, in dump_ndjson
for item in iterable:
File "/opt/homebrew/Caskroom/miniforge/base/envs/nextstrain/lib/python3.10/site-packages/augur/curate/passthru.py", line 14, in run
yield from records
File "/opt/homebrew/Caskroom/miniforge/base/envs/nextstrain/lib/python3.10/site-packages/augur/io/metadata.py", line 278, in read_metadata_with_sequences
sequences = pyfastx.Fasta(fasta)
RuntimeError: data/gisaid.fasta.zst is not plain or gzip compressed fasta formatted file
An error occurred (see above) that has not been properly handled by Augur.
To report this, please open a new issue including the original command and the error above:
<https://github.com/nextstrain/augur/issues/new/choose>
The command should be able to accept zstd compressed metadata since we've add the extra zstd
dependency for xopen
.
The NDJSONs are expected to be streamed to the command, so this should work:
zstdcat file.ndjson.zst | augur curate ...
The FASTA file is currently limited by the pyfastx
library that we use for random access of sequences. I created an issue to ask for xz
support where you also asked for zst
support 😄
The author's response is the not the most enthusiastic, so we may have to extend the library ourselves. I have no experience with C so I will probably need extra time/help here. It is also not clear how much of a priority to make this as we are still in the process of building out the subcommands.
Context
Just trying out
augur curate
. First painpoint is that it doesn't seem to supportzst
compressed fasta (at least according to docs).Description
Support zst compressed input metadata and fasta, and maybe also ndjson to reduce storage requirements.
Any thoughts @joverlee521?