@percyfal there is a snakemake-specific issue for you. When creating job-specific environments, there is an non-trivial error, which seems to be related to https://github.com/snakemake/snakemake/issues/2728. Apparently there is some race condition in snakemake. The error message is below
(aMeta) r336 /proj/snic2022-6-144/nobackup/NIKOLAY/aMeta $ snakemake --snakefile workflow/Snakefile --use-conda --conda-create-envs-only -j 20
No validator found for JSON Schema version identifier 'http://json-schema.org/draft/2020-12/schema#'
Defaulting to validator for JSON Schema version 'https://json-schema.org/draft/2020-12/schema'
Note that schema file may not be validated correctly.
No validator found for JSON Schema version identifier 'http://json-schema.org/draft/2020-12/schema#'
Defaulting to validator for JSON Schema version 'https://json-schema.org/draft/2020-12/schema'
Note that schema file may not be validated correctly.
Changing directory from /crex/proj/snic2022-6-144/nobackup/NIKOLAY/aMeta to /crex/proj/snic2022-6-144/nobackup/NIKOLAY/aMeta/workflow
Changing directory back to /crex/proj/snic2022-6-144/nobackup/NIKOLAY/aMeta
Building DAG of jobs...
Your conda installation is not configured to use strict channel priorities. This is however crucial for having robust and correct environments (for details, see https://conda-forge.org/docs/user/tipsandtricks.html). Please consider to configure strict priorities by executing 'conda config --set channel_priority strict'.
Creating conda environment workflow/envs/multiqc.yaml...
Downloading and installing remote packages.
Environment for /crex/proj/snic2022-6-144/nobackup/NIKOLAY/aMeta/workflow/rules/../envs/multiqc.yaml created (location: .snakemake/conda/42dc6e495c06c03869cdd284dac6ee42_)
Creating conda environment workflow/envs/malt.yaml...
Downloading and installing remote packages.
Environment for /crex/proj/snic2022-6-144/nobackup/NIKOLAY/aMeta/workflow/rules/../envs/malt.yaml created (location: .snakemake/conda/7071188cca1f51e1b53e06cb471bdf0e_)
Creating conda environment workflow/envs/mapdamage.yaml...
Downloading and installing remote packages.
Environment for /crex/proj/snic2022-6-144/nobackup/NIKOLAY/aMeta/workflow/rules/../envs/mapdamage.yaml created (location: .snakemake/conda/25ba1bdc0e74b9e6d5ac6dd026518b93_)
Creating conda environment workflow/envs/fastqc.yaml...
Downloading and installing remote packages.
Environment for /crex/proj/snic2022-6-144/nobackup/NIKOLAY/aMeta/workflow/rules/../envs/fastqc.yaml created (location: .snakemake/conda/e5535616ee601b4d6bec6e3db5765d74_)
Creating conda environment workflow/envs/r.yaml...
Downloading and installing remote packages.
Environment for /crex/proj/snic2022-6-144/nobackup/NIKOLAY/aMeta/workflow/rules/../envs/r.yaml created (location: .snakemake/conda/ab6bd172336d305f440f7d9a1a607632_)
Creating conda environment workflow/envs/bowtie2.yaml...
Downloading and installing remote packages.
Environment for /crex/proj/snic2022-6-144/nobackup/NIKOLAY/aMeta/workflow/rules/../envs/bowtie2.yaml created (location: .snakemake/conda/79920d5d4780e08afb57b10ceba5145e_)
Creating conda environment workflow/envs/krakenuniq.yaml...
Downloading and installing remote packages.
Environment for /crex/proj/snic2022-6-144/nobackup/NIKOLAY/aMeta/workflow/rules/../envs/krakenuniq.yaml created (location: .snakemake/conda/2a206bcec4651eead419898abdc56c99_)
Creating conda environment workflow/envs/krona.yaml...
Downloading and installing remote packages.
Environment for /crex/proj/snic2022-6-144/nobackup/NIKOLAY/aMeta/workflow/rules/../envs/krona.yaml created (location: .snakemake/conda/089384cd5c77a448cc59b4650ce57211_)
Creating conda environment workflow/envs/cutadapt.yaml...
Downloading and installing remote packages.
Environment for /crex/proj/snic2022-6-144/nobackup/NIKOLAY/aMeta/workflow/rules/../envs/cutadapt.yaml created (location: .snakemake/conda/0661a075ccb959f1d794aba5fd1708f5_)
Traceback (most recent call last):
File "/proj/snic2022-6-144/nobackup/NIKOLAY/miniconda3/envs/aMeta/lib/python3.10/weakref.py", line 667, in _exitfunc
f()
File "/proj/snic2022-6-144/nobackup/NIKOLAY/miniconda3/envs/aMeta/lib/python3.10/weakref.py", line 591, in __call__
return info.func(*info.args, **(info.kwargs or {}))
File "/proj/snic2022-6-144/nobackup/NIKOLAY/miniconda3/envs/aMeta/lib/python3.10/tempfile.py", line 868, in _cleanup
cls._rmtree(name, ignore_errors=ignore_errors)
File "/proj/snic2022-6-144/nobackup/NIKOLAY/miniconda3/envs/aMeta/lib/python3.10/tempfile.py", line 864, in _rmtree
_shutil.rmtree(name, onerror=onerror)
File "/proj/snic2022-6-144/nobackup/NIKOLAY/miniconda3/envs/aMeta/lib/python3.10/shutil.py", line 725, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/proj/snic2022-6-144/nobackup/NIKOLAY/miniconda3/envs/aMeta/lib/python3.10/shutil.py", line 658, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File "/proj/snic2022-6-144/nobackup/NIKOLAY/miniconda3/envs/aMeta/lib/python3.10/shutil.py", line 658, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File "/proj/snic2022-6-144/nobackup/NIKOLAY/miniconda3/envs/aMeta/lib/python3.10/shutil.py", line 658, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
[Previous line repeated 5 more times]
File "/proj/snic2022-6-144/nobackup/NIKOLAY/miniconda3/envs/aMeta/lib/python3.10/shutil.py", line 664, in _rmtree_safe_fd
onerror(os.rmdir, fullname, sys.exc_info())
File "/proj/snic2022-6-144/nobackup/NIKOLAY/miniconda3/envs/aMeta/lib/python3.10/shutil.py", line 662, in _rmtree_safe_fd
os.rmdir(entry.name, dir_fd=topfd)
OSError: [Errno 39] Directory not empty: 'envs'
I was trying with different versions of snakemake / python but no susccess. This seems to be a very fresh error. It was not there two weeks ago when I installed aMeta from scratch on my laptop and Uppmax. I only found it it today. The issue seems to be due to cleaning up runtime cache https://github.com/snakemake/snakemake/pull/2859 and looks like it was fixed in the latest snakemake version. Adding --conda-cleanup-pkgs cache to the snakemake command line did not help.
HOWEVER: I realized that despite the error, everything worked correctly in the aMeta run. SO I BELIEVE: this is rather a warning and probably can be ignored because it does not seem to affect the results of aMeta.
@percyfal there is a snakemake-specific issue for you. When creating job-specific environments, there is an non-trivial error, which seems to be related to https://github.com/snakemake/snakemake/issues/2728. Apparently there is some race condition in snakemake. The error message is below
I was trying with different versions of snakemake / python but no susccess. This seems to be a very fresh error. It was not there two weeks ago when I installed aMeta from scratch on my laptop and Uppmax. I only found it it today. The issue seems to be due to cleaning up runtime cache https://github.com/snakemake/snakemake/pull/2859 and looks like it was fixed in the latest snakemake version. Adding
--conda-cleanup-pkgs cacheto the snakemake command line did not help.HOWEVER: I realized that despite the error, everything worked correctly in the aMeta run. SO I BELIEVE: this is rather a warning and probably can be ignored because it does not seem to affect the results of aMeta.