defense-finder fails to run in parallel

[ezherman]

Hi,

I have been able to run defense-finder on a single core without any issues. However, I receive the error below when running defense-finder in parallel. My guess is that the issue comes from the program always writing to /tmp/defense-finder. Would it be possible to have defense-finder write to sub-directories within /tmp/defense-finder, so that the program can be run in parallel?

Traceback (most recent call last):
  File "/mnt/lustre/users/elh605/pseudomonas-defence-systems/.snakemake/conda/f3d97927ad0dc604bee4b9e739f0fac1/bin/defense-finder", line 8, in <module>
    sys.exit(cli())
  File "/mnt/lustre/users/elh605/pseudomonas-defence-systems/.snakemake/conda/f3d97927ad0dc604bee4b9e739f0fac1/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/mnt/lustre/users/elh605/pseudomonas-defence-systems/.snakemake/conda/f3d97927ad0dc604bee4b9e739f0fac1/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/mnt/lustre/users/elh605/pseudomonas-defence-systems/.snakemake/conda/f3d97927ad0dc604bee4b9e739f0fac1/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/mnt/lustre/users/elh605/pseudomonas-defence-systems/.snakemake/conda/f3d97927ad0dc604bee4b9e739f0fac1/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/mnt/lustre/users/elh605/pseudomonas-defence-systems/.snakemake/conda/f3d97927ad0dc604bee4b9e739f0fac1/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/mnt/lustre/users/elh605/pseudomonas-defence-systems/.snakemake/conda/f3d97927ad0dc604bee4b9e739f0fac1/lib/python3.9/site-packages/defense_finder_cli/main.py", line 46, in run
    defense_finder_posttreat.run(outdir)
  File "/mnt/lustre/users/elh605/pseudomonas-defence-systems/.snakemake/conda/f3d97927ad0dc604bee4b9e739f0fac1/lib/python3.9/site-packages/defense_finder_posttreat/__init__.py", line 7, in run
    bs = best_solutions.get()
  File "/mnt/lustre/users/elh605/pseudomonas-defence-systems/.snakemake/conda/f3d97927ad0dc604bee4b9e739f0fac1/lib/python3.9/site-packages/defense_finder_posttreat/best_solutions.py", line 4, in get
    parsed = parse_all()
  File "/mnt/lustre/users/elh605/pseudomonas-defence-systems/.snakemake/conda/f3d97927ad0dc604bee4b9e739f0fac1/lib/python3.9/site-packages/defense_finder_posttreat/best_solutions.py", line 8, in parse_all
    tsv_file = open('/tmp/defense-finder/all_best_solutions.tsv')
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/defense-finder/all_best_solutions.tsv'

[Alexandre-Herve]

Hi, sorry my notifications were incorrectly setup and I was unaware of your issue.

There is now a -o option that let you chose the output path you want for each run, and temporary results will be stored there as well which means you shouldn't have any issue with parallel runs anymore.

Good luck and sorry again for answering this late.

[ezherman]

Thank you @Alexandre-Herve, I really appreciate it!