Snakemake stuck creating environments/installing packages checkm2 and DAS_Tool

[Wattsupdocatron]

Hi.

Using the HiFi_MAG_Pipeline.

Snakemake/conda successfully created four environments (metabat, samtools, python, semibin), but appears to get 'stuck' creating environments/installing checkm2 and DAS_Tool packages. I initially waited six days after initiating the job through an sbatch script on our HPC system (100 gb RAM and 36 CPUs). The job didn't fail, but eventually timed out before the environments were created. There didn't seem to be any activity during this time. Not sure if I've done something wrong or missed something? Or maybe I'm being impatient, but the other environments were created quite quickly.

Code I tried initially wrapped in an sbatch script:

snakemake --snakefile Snakefile-hifimags.smk --configfile configs/Sample-Config.yaml -j 36 --use-conda

Code I tried after environments weren't installing: snakemake --snakefile Snakefile-hifimags.smk --configfile configs/Sample-Config.yaml --use-conda --conda-create-envs-only --cluster "sbatch --account=pa12 --partition=m3g --cpus-per-task={threads}" -j 5 --jobname "{rule}.{wildcards}.{jobid}" --latency-wait 60

Currently in my snakemake environment: snakemake version=5.19.3 conda version=4.8.3 python version=3.10.15

Log example - stuck on DAS_Tool: Building DAG of jobs... Creating conda environment envs/metabat.yml... Downloading and installing remote packages. Environment for envs/metabat.yml created (location: .snakemake/conda/80eeee22) Creating conda environment envs/samtools.yml... Downloading and installing remote packages. Environment for envs/samtools.yml created (location: .snakemake/conda/a756a7bf) Creating conda environment envs/semibin.yml... Downloading and installing remote packages. Environment for envs/semibin.yml created (location: .snakemake/conda/a4802c6e) Creating conda environment envs/python.yml... Downloading and installing remote packages. Environment for envs/python.yml created (location: .snakemake/conda/f70b9962) Creating conda environment envs/dastool.yml... Downloading and installing remote packages.

SLURM output: Building DAG of jobs... Creating conda environment envs/metabat.yml... Downloading and installing remote packages. Environment for envs/metabat.yml created (location: .snakemake/conda/80eeee22) Creating conda environment envs/samtools.yml... Downloading and installing remote packages. Environment for envs/samtools.yml created (location: .snakemake/conda/a756a7bf) Creating conda environment envs/semibin.yml... Downloading and installing remote packages. Environment for envs/semibin.yml created (location: .snakemake/conda/a4802c6e) Creating conda environment envs/python.yml... Downloading and installing remote packages. Environment for envs/python.yml created (location: .snakemake/conda/f70b9962) Creating conda environment envs/dastool.yml... Downloading and installing remote packages. slurmstepd: error: acct_gather_profile/influxdb _send_data: curl_easy_perform failed to send data (discarded). Reason: Couldn't connect to server slurmstepd: error: acct_gather_profile/influxdb _send_data: curl_easy_perform failed to send data (discarded). Reason: SSL connect error slurmstepd: error: acct_gather_profile/influxdb _send_data: curl_easy_perform failed to send data (discarded). Reason: Couldn't connect to server slurmstepd: error: acct_gather_profile/influxdb _send_data: curl_easy_perform failed to send data (discarded). Reason: Couldn't connect to server slurmstepd: error: acct_gather_profile/influxdb _send_data: curl_easy_perform failed to send data (discarded). Reason: Couldn't connect to server slurmstepd: error: acct_gather_profile/influxdb _send_data: curl_easy_perform failed to send data (discarded). Reason: Couldn't connect to server slurmstepd: error: acct_gather_profile/influxdb _send_data: curl_easy_perform failed to send data (discarded). Reason: Couldn't connect to server

Any help you could provide would be excellent. Thanks!

[dportik]

Hi @Wattsupdocatron , Not sure what might be the specific issue here, but have you tried restarting the analysis again?

You are correct, normally the environments build fairly quickly. Anything more than a couple hours would indicate an issue. This could be on the conda side of things, the package side of things, or the recipe. Occasionally I have also experienced an environment getting 'stuck', and I killed the job and restarted and it installed correctly.

[Wattsupdocatron]

Hi @dportik,

Thanks for your reply.

I've definitely tried restarting the analysis multiple times and it always seems to get stuck on DAS_Tool or Checkm2 installation. I've tried installing these packages independently of snakemake in their own environments, and they install correctly. I also tried to be sneaky and manually installed the packages into the .snakemake/conda/'relevant_env' directory using a -p flag with conda install, but snakemake removes these packages and begins to install its own (but never gets there). Is there some way I can direct snakemake to use manually install environments rather than the environments it tries to create?

[dportik]

Hi @Wattsupdocatron , It looks like you can specify the existing environment in the snakemake file:

https://stackoverflow.com/questions/59107413/activating-existing-conda-enviornments-in-snakemake

Maybe try this and see if you can get things to run correctly? As this appears to be a conda issue, there isn't much else I can do on my end to help out here. Let me know if this works for you.