Missing output during test run

[ganiatgithub]

Hi Vini,

Thanks for the nice program. I'm doing a test run with metaphor test -c 4 and encountered the following. I cannot find the --latency-wait flag in the metaphor call. Also, is the subsequent error in job 79 related to this? I cannot find .snakemake/log/2023-05-10T110426.717873.snakemake.log, perhaps it is deleted?

Kind regards, Gaofeng

Error message:

MissingOutputException in rule plot_cog_taxonomy in file /home/gnii0001/rp24/gaofeng/tools/Miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/rules/annotation.smk, line 393:
Job 80  completed successfully, but some output files are missing. Missing files after 5 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait:
output/annotation/cog/cobinning/plots/cobinning_phylum_relative.png
[Wed May 10 13:48:28 2023]
Finished job 89.
76 of 92 steps (83%) done
[Wed May 10 13:48:28 2023]
Finished job 81.
77 of 92 steps (84%) done
05/10/2023 13:48:28 Generated plot: 'output/annotation/cog/cobinning/plots/cobinning_species_relative.png'.
05/10/2023 13:48:28 Done.

[Wed May 10 13:48:28 2023]
Finished job 75.
78 of 92 steps (85%) done
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2023-05-10T110426.717873.snakemake.log
Metaphor did not finish with exit code 1. Please see the error message below.
Traceback (most recent call last):
  File "/home/gnii0001/rp24/gaofeng/tools/Miniconda3/envs/metaphor/bin/metaphor", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/gnii0001/rp24/gaofeng/tools/Miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/cli/cli.py", line 260, in main
    args.func(args)
  File "/home/gnii0001/rp24/gaofeng/tools/Miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/cli/test.py", line 182, in main
    retcode = run_cmd(cmd)
              ^^^^^^^^^^^^
  File "/home/gnii0001/rp24/gaofeng/tools/Miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/utils.py", line 92, in run_cmd
    retcode = check_call(cmd.split())
              ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gnii0001/rp24/gaofeng/tools/Miniconda3/envs/metaphor/lib/python3.11/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['snakemake', '--snakefile', '/home/gnii0001/rp24/gaofeng/tools/Miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/Snakefile', '--configfile', '/home/gnii0001/rp24/gaofeng/tools/Miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/test-config.yaml', '--cores', '4', '-p', '-r', '--use-conda', '--wrapper-prefix', 'https://github.com/snakemake/snakemake-wrappers/raw/', '--conda-prefix', '/home/gnii0001/rp24/gaofeng/tools/Miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda', '--config', 'max_mb=8192']' returned non-zero exit status 1.

[vinisalazar]

Dear @ganiatgithub,

Thank you for reporting this issue.

Could you please advise what version of Metaphor are you using? You can see that with the metaphor --help command. If you are using an older version, please try updating it and re-running the tests.

I cannot find the --latency-wait flag in the metaphor call.

--latency-wait is a Snakemake flag. I don't that specific flag will help in this case, but you can add any extra Snakemake flags to Metaphor by passing the --extras flag, e.g.:

metaphor --extras " --latency-wait"

Please note that values passed to the --extras flag must start with a space (this is an old Python unexpected behaviour).

I cannot find .snakemake/log/2023-05-10T110426.717873.snakemake.log, perhaps it is deleted?

This should be in the same directory where you ran the Metaphor command. If you run ls -a in that directory, the .snakemake should show (it needs the -a flag as it is a hidden directory).

Those are the Snakemake logs, but you can also check the log in output/logs/annotation directory for additional logs which can help you troubleshoot your problem. You can delete the output and .snakemake directories to run the tests from scratch.

Please let me know how it goes.

Thank you, Vini

[ganiatgithub]

Thanks for the quick reply. The version I'm using v1.7.5 CLI I'll try again.

[ganiatgithub]

Hi,

When I submit the following as a slurm job, it returns the following error, which seems related to python. Is the test run designed for directly running in terminal? (Didn't want to run test on the login node)

#!/bin/bash

#SBATCH --job-name=metaphor_test
#SBATCH --partition=m3g,comp,m3m,m3j,m3h
#SBATCH --time=3-00:00:00
#SBATCH --ntasks=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=12

mamba activate metaphor
cd /home/gnii0001/rk81_scratch/temporary_storage/test_data_metaphor
metaphor test -c 12

100%|██████████| 8/8 [00:21<00:00,  2.63s/it]
Traceback (most recent call last):
  File "/home/gnii0001/rp24/gaofeng/tools/Miniconda3/envs/metaphor/bin/metaphor", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/gnii0001/rp24/gaofeng/tools/Miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/cli/cli.py", line 260, in main
    args.func(args)
  File "/home/gnii0001/rp24/gaofeng/tools/Miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/cli/test.py", line 139, in main
    confirm_message(cores, max_mb)
  File "/home/gnii0001/rp24/gaofeng/tools/Miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/utils.py", line 82, in confirm_message
    yn = input(
         ^^^^^^
EOFError: EOF when reading a line

[vinisalazar]

Dear @ganiatgithub,

From your error message, it seems that your code got stuck waiting for the confirmation to start Metaphor. You can bypass that by adding the -y flag to your command.

Thanks, V

[ganiatgithub]

Many thanks Vini.

By running in an interactive mode, I'm able to get the testing done. So in case of a real analysis that I submit to the Slurm scheduler, I would always need to use the -v flag?

Also, by the end of the test run, it says that future analysis can be done in the test folder. In reality I have structure my analysis elsewhere, and I will delete the fest folder. Does it mean the dependencies I have installed during test run will be downloaded again?

This is fine, just want to clarify.

[vinisalazar]

Dear @ganiatgithub,

If you submit the Metaphor command as a job to the scheduler, yes you will need that flag. When I was running the pipeline on HPC I would run the Metaphor command on the login node and provided it with a cluster execution profile (see this page) and it would submit each task to the scheduler as a separate job. Do note you have to set the scheduler setting to True in the config for that to work.

Regarding dependencies, they should be in the Metaphor installation folder, so they will persist even if you delete the test directory. You can see the exact path with metaphor config show --conda-prefix.

Please let me know if you have any further questions.

Best, V