Closed bioinfwithjudith closed 6 months ago
All modified and coverable lines are covered by tests :white_check_mark:
Comparison is base (
15e943a
) 80.30% compared to head (0d02661
) 75.75%.:exclamation: Current head 0d02661 differs from pull request most recent head 920e242. Consider uploading reports for the commit 920e242 to get more accurate results
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
Quick question @jsrdrgz , did you test this on .sig
, .lca
, .sbt
etc. to see if it behaves as intended? I recall @ShaopengLiu1 mentioning that LCA isn't a suitable database format for YACHT
Kudos, no new issues were introduced!
0 New issues
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code
@dkoslicki
Apologies, I should have tested each one. 😅
Testing the different file formats, I run into the following issues:
signatures
file does not produce when using a .sig
filesourmash multisearch
cannot run for a .sig.zip
filesketch
a fresh sig.zip
file but cannot eitherthe signatures
file does not produce when using a .sig
file
(yacht_env) hey there, jzr5814! YACHT:$ yacht train --ref_file /data/jzr5814/repositories/YACHT/tests/testdata/bug_YAC-98.sig --ksize 31 --num_threads 32 --ani_thresh 0.95 --prefix 'bug_YAC-98_sig_file' --outdir ./
2024-01-23 10:53:34 - INFO - Checking reference database file
.sig
2024-01-23 10:53:34 - INFO - Creating a temporary directory
2024-01-23 10:53:34 - INFO - Extracting signature information
Traceback (most recent call last):
File "/home/grads/jzr5814/miniconda3/envs/yacht_env/bin/yacht", line 8, in <module>
sys.exit(main())
^^^^^^
File "/data/jzr5814/repositories/YACHT/yacht/__init__.py", line 51, in main
args.func(args)
File "/data/jzr5814/repositories/YACHT/yacht/make_training_data_from_sketches.py", line 68, in main
sig_info_dict = utils.collect_signature_info(num_threads, ksize, path_to_temp_dir)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/jzr5814/repositories/YACHT/yacht/utils.py", line 92, in collect_signature_info
signatures = p.starmap(get_info_from_single_sig, [(os.path.join(path_to_temp_dir, 'signatures', file), ksize) for file in os.listdir(os.path.join(path_to_temp_dir, 'signatures'))])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/data/jzr5814/repositories/YACHT/bug_YAC-98_sig_file_intermediate_files/signatures'
the sourmash multisearch
cannot run for a .sig.zip
file
(yacht_env) hey there, jzr5814! YACHT:$ yacht train --ref_file /data/jzr5814/repositories/YACHT/tests/testdata/sample.sig.zip --ksize 31 --num_threads 32 --ani_thresh 0.95 --prefix 'bug_YAC-98_sample_sig_zip_file' --outdir /data/jzr5814/repositories/YACHT/tests/testdata/
2024-01-23 10:56:23 - INFO - Checking reference database file
.zip
2024-01-23 10:56:23 - INFO - Creating a temporary directory
2024-01-23 10:56:23 - INFO - Unzipping the sourmash signature file to the temporary directory
2024-01-23 10:56:23 - INFO - Extracting signature information
100%|████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 11554.56it/s]
2024-01-23 10:56:24 - INFO - Checking if all signatures have the same scaled
2024-01-23 10:56:24 - INFO - Finding the closely related genomes with ANI > ani_thresh from the reference database
2024-01-23 10:56:24 - INFO - Running sourmash multisearch with command: sourmash scripts multisearch /data/jzr5814/repositories/YACHT/tests/testdata/bug_YAC-98_sample_sig_zip_file_intermediate_files/training_sig_files.txt /data/jzr5814/repositories/YACHT/tests/testdata/bug_YAC-98_sample_sig_zip_file_intermediate_files/training_sig_files.txt -k 31 -s 1000 -c 32 -t 0.2039068257457904 -o /data/jzr5814/repositories/YACHT/tests/testdata/bug_YAC-98_sample_sig_zip_file_intermediate_files/training_multisearch_result.csv
Traceback (most recent call last):
File "/home/grads/jzr5814/miniconda3/envs/yacht_env/bin/sourmash", line 11, in <module>
sys.exit(main())
^^^^^^
File "/home/grads/jzr5814/miniconda3/envs/yacht_env/lib/python3.12/site-packages/sourmash/__main__.py", line 10, in main
args = sourmash.cli.parse_args(arglist)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/grads/jzr5814/miniconda3/envs/yacht_env/lib/python3.12/site-packages/sourmash/cli/__init__.py", line 160, in parse_args
return get_parser().parse_args(arglist)
^^^^^^^^^^^^
File "/home/grads/jzr5814/miniconda3/envs/yacht_env/lib/python3.12/site-packages/sourmash/cli/__init__.py", line 141, in get_parser
getattr(sys.modules[__name__], op).subparser(sub)
File "/home/grads/jzr5814/miniconda3/envs/yacht_env/lib/python3.12/site-packages/sourmash/cli/scripts/__init__.py", line 48, in subparser
_extension_dict.update(sourmash.plugins.add_cli_scripts(s))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/grads/jzr5814/miniconda3/envs/yacht_env/lib/python3.12/site-packages/sourmash/plugins.py", line 164, in add_cli_scripts
subparser = parser.add_parser(script_cls.command,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/grads/jzr5814/miniconda3/envs/yacht_env/lib/python3.12/argparse.py", line 1214, in add_parser
raise ArgumentError(self, _('conflicting subparser: %s') % name)
argparse.ArgumentError: argument subcmd: conflicting subparser: manysearch
Traceback (most recent call last):
File "/home/grads/jzr5814/miniconda3/envs/yacht_env/bin/yacht", line 8, in <module>
sys.exit(main())
^^^^^^
File "/data/jzr5814/repositories/YACHT/yacht/__init__.py", line 51, in main
args.func(args)
File "/data/jzr5814/repositories/YACHT/yacht/make_training_data_from_sketches.py", line 78, in main
multisearch_result = utils.run_multisearch(num_threads, ani_thresh, ksize, scale, path_to_temp_dir)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/jzr5814/repositories/YACHT/yacht/utils.py", line 119, in run_multisearch
raise ValueError(f"Error running sourmash multisearch with command: {cmd}")
ValueError: Error running sourmash multisearch with command: sourmash scripts multisearch /data/jzr5814/repositories/YACHT/tests/testdata/bug_YAC-98_sample_sig_zip_file_intermediate_files/training_sig_files.txt /data/jzr5814/repositories/YACHT/tests/testdata/bug_YAC-98_sample_sig_zip_file_intermediate_files/training_sig_files.txt -k 31 -s 1000 -c 32 -t 0.2039068257457904 -o /data/jzr5814/repositories/YACHT/tests/testdata/bug_YAC-98_sample_sig_zip_file_intermediate_files/training_multisearch_result.csv
I tried to sketch
a fresh sig.zip
file but cannot either
(yacht_env) hey there, jzr5814! testdata:$ sourmash sketch dna -f -p k=31,scaled=1000,abund -o bug_YAC-98.sig.zip bug_YAC-98.fastn
Traceback (most recent call last):
File "/home/grads/jzr5814/miniconda3/envs/yacht_env/bin/sourmash", line 11, in <module>
sys.exit(main())
^^^^^^
File "/home/grads/jzr5814/miniconda3/envs/yacht_env/lib/python3.12/site-packages/sourmash/__main__.py", line 10, in main
args = sourmash.cli.parse_args(arglist)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/grads/jzr5814/miniconda3/envs/yacht_env/lib/python3.12/site-packages/sourmash/cli/__init__.py", line 160, in parse_args
return get_parser().parse_args(arglist)
^^^^^^^^^^^^
File "/home/grads/jzr5814/miniconda3/envs/yacht_env/lib/python3.12/site-packages/sourmash/cli/__init__.py", line 141, in get_parser
getattr(sys.modules[__name__], op).subparser(sub)
File "/home/grads/jzr5814/miniconda3/envs/yacht_env/lib/python3.12/site-packages/sourmash/cli/scripts/__init__.py", line 48, in subparser
_extension_dict.update(sourmash.plugins.add_cli_scripts(s))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/grads/jzr5814/miniconda3/envs/yacht_env/lib/python3.12/site-packages/sourmash/plugins.py", line 164, in add_cli_scripts
subparser = parser.add_parser(script_cls.command,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/grads/jzr5814/miniconda3/envs/yacht_env/lib/python3.12/argparse.py", line 1214, in add_parser
raise ArgumentError(self, _('conflicting subparser: %s') % name)
argparse.ArgumentError: argument subcmd: conflicting subparser: manysearch
Oof, well something is definitely wrong then, since we know from tests that the .sig
and .sig.zip
files should work. First, be sure that using the code before your changes those files worked. No point in testing them if they aren't already confirmed to work on the stable, working branch
@jsrdrgz FYI, the sourmash python API fits all data format (including LCA). However, in yacht we specifically asked for single sketch input. Check this command (used in load sample sig): https://github.com/KoslickiLab/YACHT/blob/6b8b7c889acd4de91843e8906f684b181f0864c2/srcs/utils.py#L14
Therefore, David suggests only use sig or sig.zip for sample signatures.
hello @jsrdrgz , could you please tell what is the status of this task? Looks like pipeline is failing.
I was able to reproduce this error and fixed the condition on the type of file that is being accepted, so now a .sig.zip and .sig files can be accepted.
However, now I get the following error where the file will be extracted so a zipfile is expected.
Is this necessary in YACHT? Unsure how to move on from here.