Problem with running

[wertick01]

When I'm running the hecatomb, program stops at: Downloading and installing remote packages...

No processes are running, the program is not executed.

[beardymcjohnface]

Hi, That would be the conda/mamba stage; there should be a mamba process running. Does it terminate or does it just hang at that point? How are you running the pipeline and what sort of system are you running it on?

[wertick01]

I'm running commands in ubuntu 20.04 lts server. The process just hang at this point.

hecatomb run --test --fast Running Hecatomb Running snakemake command: snakemake -j 32 --use-conda --conda-frontend mamba --rerun-incomplete --printshellcmds --nolock --show-failed-logs --conda-prefix /home/user/anaconda3/envs/hecatomb/snakemake/workflow/conda -s /home/user/anaconda3/envs/hecatomb/snakemake/workflow/Hecatomb.smk -C Reads=/home/user/anaconda3/envs/hecatomb/test_data Host=human Output=hecatomb_out SkipAssembly=False Fast=True Config file /home/user/anaconda3/envs/hecatomb/snakemake/workflow/../config/config.yaml is extended by additional config specified via the command line. Building DAG of jobs... Creating conda environment anaconda3/envs/hecatomb/snakemake/workflow/envs/plotly.yaml... Downloading and installing remote packages.

And nothing else is happening.

[beardymcjohnface]

The conda env creation steps can take a little while. It should only be a matter of minutes each and it will only need to do it once. If it's taking longer than say 30 minutes and you can't see any conda/mamba jobs with top/htop then something is definitely broken.

You could also check your mamba install to see if it's working correctly for the above plotly env. To do this your could run:

mamba create -n testEnv -c defaults -c bioconda -c conda-forge plotly=5.3.1 python-kaleido=0.2.1

[wertick01]

Everything is fine with mamba. Env is working properly. But problem is the same.

[wertick01]

And every time it hangs on downloading packages for different envs: krona.yaml, mmseqs2.yaml and etc. I tried updating them manually, but it didn't help. Could it be a conflict between the versions of the layouts in each envs?

[beardymcjohnface]

Hi, I'm able to recreate the problem with new builds of hecatomb, both for the conda build and my older builds on my anaconda channel. The problem is mamba is dying for some reason and Snakemake is not picking it up for some other reason. I'll see what I can to do fix the issue but for now you can copy and paste the snakemake command for running the test dataset and manually substitute out --conda-frontend mamba with --conda-frontend conda. Once the environments are created you should be find to go back to using the launcher.

[wertick01]

Thanks very much, thats helped.

[beardymcjohnface]

So this is a new issue with snakemake + mamba; specifically mamba 18.* seems to be having issues https://github.com/snakemake/snakemake/issues/1287

There's a new release of mamba--19.0.0--which seems to be working with snakemake again. You should be able to activate your Hecatomb environment and update mamba:

mamba update mamba

Assuming everything is working again, when I push the next Hecatomb update I'll also update the build with the current mamba and snakemake versions and close this issue then.

[wertick01]

Thanks, updating mamba helped. And hecatomb has one more problem with importing pysam. Your algorithm uses pysam 0.10.0 wich works with python 3.4.

I'm running hecatomb with assembly works out at 92% and giving an error /home/user/anaconda3/envs/hecatomb/bin/python3.10 /home/user/.snakemake/scripts/tmpeofqx9q6.contigReadTaxon.py Traceback (most recent call last): File "/home/user/.snakemake/scripts/tmpeofqx9q6.contigReadTaxon.py", line 8, in import pysam ModuleNotFoundError: No module named 'pysam' [Tue Nov 30 17:58:23 2021] Error in rule contig_read_taxonomy: jobid: 51 output: hecatomb_out/RESULTS/contigSeqTable.tsv log: hecatomb_out/STDERR/contig_read_taxonomy.log (check log file(s) for error message) Logfile hecatomb_out/STDERR/contig_read_taxonomy.log not found.

Traceback (most recent call last): File "/home/user/anaconda3/envs/hecatomb/lib/python3.10/site-packages/snakemake/executors/init.py", line 593, in _callback raise ex File "/home/user/anaconda3/envs/hecatomb/lib/python3.10/concurrent/futures/thread.py", line 52, in run result = self.fn(*self.args, *self.kwargs) File "/home/user/anaconda3/envs/hecatomb/lib/python3.10/site-packages/snakemake/executors/init.py", line 579, in cached_or_run run_func(args) File "/home/user/anaconda3/envs/hecatomb/lib/python3.10/site-packages/snakemake/executors/init.py", line 2461, in run_wrapper raise ex File "/home/user/anaconda3/envs/hecatomb/lib/python3.10/site-packages/snakemake/executors/init.py", line 2358, in run_wrapper run( File "/home/user/anaconda3/envs/hecatomb/snakemake/workflow/rules/03_mapping.smk", line 97, in __rule_contig_read_taxonomy File "/home/user/anaconda3/envs/hecatomb/lib/python3.10/site-packages/snakemake/script.py", line 1369, in script executor.evaluate() File "/home/user/anaconda3/envs/hecatomb/lib/python3.10/site-packages/snakemake/script.py", line 381, in evaluate self.execute_script(fd.name, edit=edit) File "/home/user/anaconda3/envs/hecatomb/lib/python3.10/site-packages/snakemake/script.py", line 580, in execute_script self._execute_cmd( File "/home/user/anaconda3/envs/hecatomb/lib/python3.10/site-packages/snakemake/script.py", line 412, in _execute_cmd return shell( File "/home/user/anaconda3/envs/hecatomb/lib/python3.10/site-packages/snakemake/shell.py", line 287, in new raise sp.CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'set -euo pipefail; /home/user/anaconda3/envs/hecatomb/bin/python3.10 /home/user/.snakemake/scripts/tmpeofqx9q6.contigReadTaxon.py' returned non-zero exit status 1.

I tried to install pysam with mamba, but...

Encountered problems while solving:

• package pysam-0.10.0-py34_1 requires python 3.4*, but none of the providers can be installed

[beardymcjohnface]

ah, there is a conda env for that rule that calls for pysam>=0.16 but I forgot to add that to the rule when I moved it from a run: directive to a script: directive. I'll fix that in the next release but for now you should be able to add pysam to your hecatomb env after changing to python 3.9

mamba install python=3.9
mamba install -c bioconda pysam

[beardymcjohnface]

I've created a new release for hecatomb that fixes this issue, and it should be available soon from bioconda (it will be version 1.0.0.beta.3). You should be able to run mamba update -c bioconda hecatomb once it's available.