Closed jfy133 closed 6 months ago
There are two problems. First the file order in --taxonomy-files
should be: nodes, names [, merged] or one .tar.gz
file containing standard NCBI file names (nodes.dmp names.dmp merged.dmp
). In your example: --taxonomy-files prot_nodes.dmp prot_names.dmp
will work. The second issue is that the prot_names.dmp
has a not expected empty line causing the parsing error.
I will try to document better the taxonomy file ordering and skip empty lines when parsing the taxonomy in the next release.
I also notice that this small example will not build properly. There are no assembly accession information in the file names, which is expected by default. The easiest in this case is to generate a file linking each file to the taxonomic target and use it with --input-file
, example:
haemophilus_influenzae.fna.gz haemophilus_influenzae 727
sarscov2.fasta sarscov2 2697049
A longer explanation, since this already appeared before #277
Before version 2.0.0, you could use one or more files in any format and with --input-target sequence
ganon would extract the sequence accession from all files and automatically retrieve taxonomy and build the database. In the newer versions (>=2.0.0), --input-target sequence
is not possible due to way the new filter is built. However, parsing by sequence still works with the old filter --filter-type ibf
, but you don't get the benefits of the new filter.
That's why the creating the --input-file
is the best solution for now. You could also rename your inputs for NCBI assembly file formats, e.g. GCF_900478275.1.fna.gz
instead of haemophilus_influenzae.fna.gz
and GCA_011545545.1.fasta
instead of sarscov2.fasta
.
I'm working in a solution for this in ganon2 to bring back the old functionality.
Thanks very much @pirovc for the detailed explanations! I will attempt to update the nf-core module and dataset accordingly (where the original failures came from).
I'll let you close this issue when you're ready (e.g. if you want to keep it open until you've updated the documentation).
@pirovc I'm working on this, regarding the Ok ignore that, all the viewers I'm using are stripping it without telling me :sweat_smile: prot_names.dmp
, i don't see any empty line myself when I inspect the file, do you mean the empty column?
I also realise I'm not really following what the input-file
file columns is meant to contain...
From here: https://pirovc.github.io/ganon/custom_databases/#non-standardcustom-accessions
I understand I can have 3 or 5 columns:
I don't think I'm following the terminology of 'target' and 'specialization' based on the information on that page...I al
Following your example above of
I also notice that this small example will not build properly. There are no assembly accession information in the file names, which is expected by default. The easiest in this case is to generate a file linking each file to the taxonomic target and use it with
--input-file
, example:haemophilus_influenzae.fna.gz haemophilus_influenzae 727 sarscov2.fasta sarscov2 2697049
I get the following error:
Parsing --input-file input-file.txt
Traceback (most recent call last):
File "/home/james/bin/miniconda3/envs/ganon/bin/ganon", line 33, in <module>
sys.exit(load_entry_point('ganon==2.0.0', 'console_scripts', 'ganon')())
File "/home/james/bin/miniconda3/envs/ganon/lib/python3.10/site-packages/ganon/ganon.py", line 53, in main_cli
sys.exit(0 if main() else 1)
File "/home/james/bin/miniconda3/envs/ganon/lib/python3.10/site-packages/ganon/ganon.py", line 36, in main
ret = build_custom(cfg)
File "/home/james/bin/miniconda3/envs/ganon/lib/python3.10/site-packages/ganon/build_update.py", line 269, in build_custom
info = load_input(cfg, input_files)
File "/home/james/bin/miniconda3/envs/ganon/lib/python3.10/site-packages/ganon/build_update.py", line 485, in load_input
info = parse_input_file(cfg.input_file, info, cfg.input_target)
File "/home/james/bin/miniconda3/envs/ganon/lib/python3.10/site-packages/ganon/build_update.py", line 460, in parse_input_file
info = pd.read_csv(input_file,
File "/home/james/.local/lib/python3.10/site-packages/pandas/util/_decorators.py", line 211, in wrapper
return func(*args, **kwargs)
File "/home/james/.local/lib/python3.10/site-packages/pandas/util/_decorators.py", line 331, in wrapper
return func(*args, **kwargs)
File "/home/james/.local/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 950, in read_csv
return _read(filepath_or_buffer, kwds)
File "/home/james/.local/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 611, in _read
return parser.read(nrows)
File "/home/james/.local/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 1778, in read
) = self._engine.read( # type: ignore[attr-defined]
File "/home/james/.local/lib/python3.10/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 230, in read
chunks = self._reader.read_low_memory(nrows)
File "pandas/_libs/parsers.pyx", line 808, in pandas._libs.parsers.TextReader.read_low_memory
File "pandas/_libs/parsers.pyx", line 890, in pandas._libs.parsers.TextReader._read_rows
File "pandas/_libs/parsers.pyx", line 952, in pandas._libs.parsers.TextReader._convert_column_data
pandas.errors.ParserError: Too many columns specified: expected 5 and found 3
Which is confusing: it needs 5, but I've given 3, but there are two many columns?
I also get the same error when I specify a similar file, but with the sequence accession ID here:
genome.fasta MT192765.1 2697049
If I extend to what I presume is a 5 column file:
genome.fasta MT192765.1 2697049 Severe acute respiratory syndrome coronavirus 2 isolate SARS-CoV-2/human/USA/PC00101P/2020
Then I get
Building index (raptor)
raptor prepare
The following command failed to run:
/home/james/bin/miniconda3/envs/ganon/bin/raptor prepare --input 'test_files/build/hibf.txt' --output 'test_files/build/' --kmer 19 --window 31 --quiet --threads 2
Error code: 1
The test files from the issues above
I see that it's a bit confusing, the --input-file
is used to perform many different inputs for ganon build-custom
. Don't worry about the sequence headers (MT192765.1), they are not used in file mode. --input-target file
tells ganon that each input file is one unit (e.g. multi-fasta files are considered one input, sequence headers ignored).
target
here is just a name for your file. You could use the filename itself: genome.fasta <tab> genome.fasta <tab> 2697049
. The file (named with target) will be another unique level on the taxonomy below the given node, depending on the parameter --level
:
--level file
(default) -> use the file (named with target) as a tax. node with 2697049 as parent
--level leaves
-> files are grouped by leaf nodes (2697049)
--level species
-> files are grouped by species (694009)
--level genus
-> ...
Specialization (cols. 4 and 5) are only needed if you want to create a specialized taxonomic level with a custom name, with the option to group files under this node. Example:
genome.fasta <tab> genome.fasta <tab> 2697049 <tab> MyTaxID <tab> MyCustomNameForTheNode
genome2.fasta <tab> genome2.fasta <tab> 2697049 <tab> MyTaxID <tab> MyCustomNameForTheNode
--level custom
-> use the specialization (MyTaxID) as a tax. node with 2697049 as parent.
When --input-target sequence
the columns are used slightly differently. I will improve the docs on this part, it can get very confusing with so many option.
Answering the number of columns, they are all optional, with the following behavior:
file [<tab> target <tab> node <tab> specialization <tab> specialization_name]
I could not replicate the bug, the files you sent are incomplete with just the symbolic links.
I actually managed to replicate it and a fix is underway with better documentation and --input-target sequence
for any filter type.
Most issues fixed in v2.1.0. The empty line bug will be fixed later on multitax.
Let me know if something is still unclear about the --input-file
in the updated docs: https://pirovc.github.io/ganon/custom_databases/
However, if your files are NCBI header standard you can now use --input-target sequence
and skip the --input-file
creation.
Great thank you very much @pirovc ! I will be continuing this on Thursday, will review the docs etc.
I saw the bioconda recipe hasn't been updated yet so I didn't continue today. I'm travelling for a month from next week, so will have to provide feedback (if any needed!) then :)
From nf-core/createtaxdb
(ignore the 'prot' in the nodes/names.dmp files, it is standard taxonomy :) ).
The relevant files:
sarscov2 = https://raw.githubusercontent.com/nf-core/test-datasets/createtaxdb/data/fasta/sarscov2.fasta influenzae = https://raw.githubusercontent.com/nf-core/test-datasets/createtaxdb/data/fasta/haemophilus_influenzae.fna.gz nodesdmp = 'https://raw.githubusercontent.com/nf-core/test-datasets/createtaxdb/data/taxonomy/prot_nodes.dmp' namesdmp = 'https://raw.githubusercontent.com/nf-core/test-datasets/createtaxdb/data/taxonomy/prot_names.dmp'