Test not working

[mudymudy]

Dear developers,

I used aMeta January-February 2023 and it was working perfectly. Now I need to run it again but when I run the test is not working.

./runtest.sh -j 30 This looks like the first test run... Installing bioconda packages... No validator found for JSON Schema version identifier 'http://json-schema.org/draft/2020-12/schema#' Defaulting to validator for JSON Schema version 'https://json-schema.org/draft/2020-12/schema' Note that schema file may not be validated correctly. Excluding samples 'foobar' from analysis Restricting analysis to samples 'foo','bar' No validator found for JSON Schema version identifier 'http://json-schema.org/draft/2020-12/schema#' Defaulting to validator for JSON Schema version 'https://json-schema.org/draft/2020-12/schema' Note that schema file may not be validated correctly. Changing directory from /home/user/aMeta/.test to /home/user/aMeta/workflow Changing directory back to /home/user/aMeta/.test Building DAG of jobs... Creating conda environment ../workflow/envs/krona.yaml... Downloading and installing remote packages. Cleaning up conda package tarballs and package cache. Environment for /home/user/aMeta/workflow/rules/../envs/krona.yaml created (location: .snakemake/conda/01d22bc3188d41766b4b53f4d700f1c9) Creating conda environment ../workflow/envs/multiqc.yaml... Downloading and installing remote packages. Cleaning up conda package tarballs and package cache. Environment for /home/user/aMeta/workflow/rules/../envs/multiqc.yaml created (location: .snakemake/conda/73ec0bf71681323174e5c299cdb30232) Creating conda environment ../workflow/envs/r.yaml... Downloading and installing remote packages. Cleaning up conda package tarballs and package cache. Environment for /home/user/aMeta/workflow/rules/../envs/r.yaml created (location: .snakemake/conda/af3138f89c22b5caa67dfec4b96a8933) Creating conda environment ../workflow/envs/bowtie2.yaml... Downloading and installing remote packages. Cleaning up conda package tarballs and package cache. Environment for /home/user/aMeta/workflow/rules/../envs/bowtie2.yaml created (location: .snakemake/conda/c42776cb10655143deab9a0ad9c60817) Creating conda environment ../workflow/envs/malt.yaml... Downloading and installing remote packages. Cleaning up conda package tarballs and package cache. Environment for /home/user/aMeta/workflow/rules/../envs/malt.yaml created (location: .snakemake/conda/2b924bbb596d214e9d92d63009fdc853) Creating conda environment ../workflow/envs/cutadapt.yaml... Downloading and installing remote packages. Cleaning up conda package tarballs and package cache. Environment for /home/user/aMeta/workflow/rules/../envs/cutadapt.yaml created (location: .snakemake/conda/066d8bc8ace1569fd902f9fc765557d8) Creating conda environment ../workflow/envs/mapdamage.yaml... Downloading and installing remote packages.

The last bit (mapdamage) is taking hours. I don't remember the test run to take this long when I ran it last year. Could this be related to this problem? : https://github.com/NBISweden/aMeta/issues/164

[LeandroRitter]

Thanks for reporting @mudymudy! The test does not seem to be crushing from the output you posted above. Are you running the test on a node / laptop with at least 10-16 GB of RAM? Due to some latest modifications, mapdamage might take a bit more RAM than previously. Also, if the installation of the mapdamage environment takes hours I would suspect problems with internet connection or conda / mamba because I regularly run the test in different clusters and so far has not discovered that it hangs on the installation step. However, I suspect that if you are running the test on an login node that has less than 10-16 GB or unstable internet connection or your conda / mamba are somehow broken / not updated, this might be an explanation. Please try to run the test either on your laptop or a compute node on an HPC with more RAM and let us know if that helped, and please also let us know your conda / mamba versions. Otherwise, we will need to take a closer look and inspect the issue in details. Thanks for your valuable feedback!

[mudymudy]

Thank you for your reply.

I killed the test run because it was frozen in that step, and decided to just run aMeta with just 1 sample. Before running it I checked the RAM:

$ free -hg total used free shared buff/cache available Mem: 1.0Ti 7.4Gi 26Gi 11Mi 972Gi 992Gi Swap: 1.0Ti 3.4Gi 1.0Ti

So I don't think RAM is a problem. After running for a bit it failed here:

[bam_index_build2] fail to open the BAM file. Waiting at most 5 seconds for missing files. MissingOutputException in rule Bowtie2_Alignment in file /user/home/aMeta/workflow/rules/align.smk, line 27: Job 5 completed successfully, but some output files are missing. Missing files after 5 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait: results/BOWTIE2/file1.concat/AlignedToBowtie2DB.bam results/BOWTIE2/file1.concat/AlignedToBowtie2DB.bam.bai

I tried with incrementing the latency but the same happened. After a minute every file in the BOWTIE2 results folder was deleted. Not sure why. My conda versions: $ mamba -V mamba 1.5.7 conda 24.5.0

Any help would be appreciated!! Thank you again.

[LeandroRitter]

Dear @mudymudy, thanks! I advise you not to proceed with the real sample run until the test run completed successfully. If you have not set aMeta properly up there can be a number of issues with very misleading error messages that do not point at the right cause of the problem. Are you a member of the SPAAM community and using their slack channel? If not, please drop me a line here nikolay.oskolkov@scilifelab.se and we could schedule a zoom where I can assist you with installation. I would first make sure that the testrun works.

[mudymudy]

Hi Nikolay,

Thank you for all your help, it is very much appreciated! After doing what you suggested:

rm -rf aMeta conda remove -n aMeta --all

Then install it again but this time adding this line:

• snakemake=6.3.0

in: aMeta/workflow/envs/environment.yaml

and finally replacing conda with mamba frontend in:

/aMeta/.test/runtesh.sh

Made aMeta test run work!

[LeandroRitter]

Excellent @mudymudy, this is good to know. We should probably make mamba default frontend in the tesrun script @percyfal. Strangely enough it is still conda by default while I thought we switched to mamba long time ago.

[percyfal]

@LeandroRitter as of conda version >= 23.10.0 the solver is libmamba by default, therefore there should be no need to change to mamba. As the documentation of the runtest script states, if you have a conda version that is lower than 23.10.0 you can run the test with the command CONDA_FRONTEND=mamba ./runtest.sh -j 1

[LeandroRitter]

@percyfal true, it is written in the runtest.sh but in the case of @mudymudy there was likely an old conda used, therefore environment installation was taking too long, and the info about conda vs. mamba was somehow missed. So I wonder whether we should replace this line in testrun.sh:

CONDA_FRONTEND="${CONDA_FRONTEND:=conda}"

with this line

CONDA_FRONTEND=mamba

to avoid this issues in the future? Unless there is a particular gain in keeping it as is CONDA_FRONTEND="${CONDA_FRONTEND:=conda}"

In the case of @mudymudy, specifying mamba explicitly was the solution, while it took us quite some time to figure this out :))

[percyfal]

Addressed in #172. Closing for now as this seems related to the conda frontend.