Error executing process > 'concatenateFastq'

[ianm0001]

Hi! Thank you for providing a very promising script for Nanopore amplicon data. However, I am running into issues when trying to run it. There seem to be some issue with the concatenateFastq process, and the error message goes (I have removed the real path in all the files/text):

Feb-05 09:44:01.829 [Task submitter] DEBUG nextflow.processor.TaskRun - Unable to dump error of process 'null' -- Cause: java.nio.file.NoSuchFileException: /my/directory/work/18/36b7a3b82c4dc6c5029ff3147a278a/.command.log Feb-05 09:44:01.831 [Task submitter] ERROR nextflow.processor.TaskProcessor - Error executing process > 'concatenateFastq'

Caused by: java.io.IOException: Cannot run program "qsub" (in directory "/my/directory/work/18/36b7a3b82c4dc6c5029ff3147a278a"): error=2, No such file or directory

Command executed:

qsub -N nf-concatenateF .command.run

Command exit status:

Command output: (empty)

Work dir: /my/directory/work/18/36b7a3b82c4dc6c5029ff3147a278a

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named .command.sh Feb-05 09:44:01.833 [Task submitter] DEBUG nextflow.Session - Session aborted -- Cause: Error submitting process 'concatenateFastq' for execution

Prior to submitting the script to SLURM (sbatch system on UPPMAX/Rackham, a HPC) I load the Nextflow module. I have tried both running the command (nextflow -c metontiime.conf run metontiime2.nf -profile singularity) with the singularity and docker profile, but I think singularity is the right way to go as I am running on a HPC. I have also tried changing the singularity.cacheDir and containerOptions after suggestions about that in previous issue threads, but it didn't help me. What is confusing is that I have turned off the concatenateFastq function in the configuration file, as my data has already been concatenated, trimmed and filtered (see two of the configuration files I have tried to run, enclosed here), but still it is this function that seem to fail the script. I have gzipped the data as it seemed that the pipeline requires that. metontiime_singularity_issue.txt metontiime2_docker_issue.txt

Please tell me if I need to provide any other information, I am new with this and may have done some stupid mistake. I have already consulted staff at the HPC and tried running the script with additional configuration files for this HPC specifically, but it did no good.

Thanks for your input Best /Ida

[MaestSi]

Hi, it seems you are using pbspro as the executor, instead of slurm. Was that a typo in your message? Moreover, in case your data are not in the /home directory, did you mount the relevant directories? I see at line 54 of the Singularity script: containerOptions = '--bind /home/:/home' Best, SM

[ianm0001]

Hi,

Thanks a lot for your input, and the configuration file has been amended according to your suggestions, along with additional amendments proposed by a bioinformatician that I consulted. However, the pipeline still will not run. The new error we are dealing with, and are unable to solve, is: Feb-05 14:16:48.892 [main] ERROR nextflow.cli.Launcher - @unknown java.lang.StackOverflowError: null I enclose the most recent configuration file so you could potentially help me get along, and the commands I am running are: module load uppmax bioinfo-tools Nextflow nextflow -c metontiime_5feb_uppmax.conf run metontiime2.nf -profile uppmax,singularity

Thank you for your input! Best/Ida metontiime_5feb_issue.txt

[MaestSi]

Hi, I never had to add any lines to the config file before, so I can't tell if those may be of help in your specific case. Just remember that Nextflow needs java executable to be in PATH to run, you may need to load another module, if that is not available yet. Personally, I usually create a conda environment where I have nextflow (and java) available, and I check singularity executable is in PATH too. That's it. I do not create other profiles. Please also check that you do not need to specify any queues (now queue = 'workq' is commented out). Then, I write the command in a text file and submit it. For example, the content of the file (say RunMetONTIIME.sh) may be like this one:

#!/bin/bash

#SBATCH --mem=2G
#SBATCH --job-name=MetONTIIME
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1

cd /path/to/MetONTIIME/directory
source /path/to/miniconda3/bin/activate /path/to/miniconda3/envs/nextflow_env

nextflow -c metontiime.conf run metontiime2.nf -profile singularity

And, as a last step I would submit it with: sbatch RunMetONTIIME.sh Best, SM

[ianm0001]

Hi again,

Thank you for your input. The previous error has been resolved, along with a number of other errors, but despite the help of several bioinformatics experts, I have not been able to come past this following error: _Feb-09 09:39:35.203 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'importDb (1)'

Caused by: Process importDb (1) terminated with an error exit status (1)

Command executed:

mkdir -p /crex/proj/[my_proj]/MetONTIIME/Output/importDb

qiime tools import --type 'FeatureData[Sequence]' --input-path sh_refs_qiime_ver9_dynamic_all_25.07.2023.fasta --output -path /crex/proj/[my_proj]/MetONTIIME/Output/importDb/db_sequences.qza

qiime tools import --type 'FeatureData[Taxonomy]' --input-path noheader_taxonomy_qiime_ver9_dynamic_alleukaryotes_25.07.2023.tsv
--input-format HeaderlessTSVTaxonomyFormat --output-path /crex/proj/[my_proj]/MetONTIIME/Output/importDb/db_taxonomy.qza

Command exit status: 1 [my_proj]/MetONTIIME/Output/importDb/db_taxonomy.qza

Command exit status: 1

Command output: (empty)

Command error: INFO: Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred INFO: Environment variable SINGULARITYENV_NXF_TASK_WORKDIR is set, but APPTAINERENV_NXF_TASK_WORKDIR is preferred INFO: Environment variable SINGULARITYENV_SNIC_TMP is set, but APPTAINERENV_SNICTMP is preferred Matplotlib created a temporary config/cache directory at /scratch/44979750/matplotlib-v93uog6f because the default path (/home/ianm0001/.config/matplo tlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed

I enclose the files I am trying to use in my run. My data has already been concatenated, filtered and trimmed so I have modified the pipeline(s) to my needs.

Thankful for any input! Ida

MetONTIIME_9feb_issue.txt metontiime2_9feb_issue.txt metontiime_9feb_issue.conf.txt

[MaestSi]

Hi, I don't see any evident errors in the files (except the workflow chunk missing in the metontiime2_9feb_issue.txt file, but that is probably a typo. The error: Matplotlib created a temporary config/cache directory at /scratch/44979750/matplotlib-v93uog6f because the default path (/home/ianm0001/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed_ [...] makes me think the Singularity container is also trying to access /scratch and /home directories. I would try and edit line 57 of conf file with: containerOptions = '--bind /crex/proj:/crex/proj --bind /scratch:/scratch --bind /home/ianm0001:/home/ianm0001' Let me know if it works. SM

[ianm0001]

Thank you for your input. The workflow chunk is still there, just moved up to almost the top of the script.

I edited line 57 according to your suggestion and now I have another error code: Feb-09 11:35:11.811 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for task: name=importDb (1); work-dir=/crex/[My_proj]/MetONTIIME/work/be/0c9f039abfc3fa861b88d1239ff199 error [nextflow.exception.ProcessFailedException]: Process importDb (1) terminated with an error exit status (140) Feb-09 11:35:11.910 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'importDb (1)'

Caused by: Process importDb (1) terminated with an error exit status (140)

Command executed:

mkdir -p /crex/proj/[My_proj]/MetONTIIME/Output/importDb

qiime tools import --type 'FeatureData[Sequence]' --input-path sh_refs_qiime_ver9_dynamic_all_25.07.2023.fasta --output -path /crex/proj/[My_proj]/MetONTIIME/Output/importDb/db_sequences.qza

qiime tools import --type 'FeatureData[Taxonomy]' --input-path noheader_taxonomy_qiime_ver9_dynamic_alleukaryotes_25.07.2023.tsv
--input-format HeaderlessTSVTaxonomyFormat --output-path /crex/proj/[My_proj]/MetONTIIME/Output/importDb/db_taxonomy.qza

Command exit status: 140

Command output: (empty)

Command error: INFO: Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred INFO: Environment variable SINGULARITYENV_NXF_TASK_WORKDIR is set, but APPTAINERENV_NXF_TASK_WORKDIR is preferred INFO: Environment variable SINGULARITYENV_SNIC_TMP is set, but APPTAINERENV_SNIC_TMP is preferred

So I don't know exactly what to make of that.

Best/Ida

[MaestSi]

Based on this post it seems error code 140 is due to time limit or insufficient memory. Isn't #SBATCH -t 00:15:00, which you have in your file, setting the walltime at 15 seconds? SM

[ianm0001]

That’s right, but now I’m just trying to make the script work(short test runs are given higher queue priority), and I’ve tried running it with a 10 day time limit too, which didn’t help. I might try increasing both time and the RAM called though.

Mvh, Ida

9 feb. 2024 kl. 15:16 skrev Simone Maestri @.***>:



CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Based on this posthttps://superuser.com/questions/1811930/error-code-140-in-command-running-through-nextflow-on-slurm it seems error code 140 is due to time limit or insufficient memory. Isn't #SBATCH -t 00:15:00, which you have in your file, setting the walltime at 15 seconds? SM

— Reply to this email directly, view it on GitHubhttps://github.com/MaestSi/MetONTIIME/issues/87#issuecomment-1936008202, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BCQ4PEVOHMBSKY5XV6ZZBJ3YSYVR3AVCNFSM6AAAAABCZYF4IOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZWGAYDQMRQGI. You are receiving this because you authored the thread.Message ID: @.***>

---

När du skickar e-post till SLU så innebär detta att SLU behandlar dina personuppgifter. För att läsa mer om hur detta går till, klicka här https://www.slu.se/om-slu/kontakta-slu/personuppgifter/ E-mailing SLU will result in SLU processing your personal data. For more information on how this is done, click here https://www.slu.se/en/about-slu/contact-slu/personal-data/

[MaestSi]

I know, but 15s might be too little to realise whether it works or not. SM

[ianm0001]

It’s 15 minutes, and typically the script fails around 5-6 minutes.

Mvh, Ida

9 feb. 2024 kl. 16:57 skrev Simone Maestri @.***>:



CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

I know, but 15s might be too little to realise whether it works or not. SM

— Reply to this email directly, view it on GitHubhttps://github.com/MaestSi/MetONTIIME/issues/87#issuecomment-1936177528, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BCQ4PETURPIWGRTAXUYWHP3YSZBNDAVCNFSM6AAAAABCZYF4IOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZWGE3TONJSHA. You are receiving this because you authored the thread.Message ID: @.***>

---

När du skickar e-post till SLU så innebär detta att SLU behandlar dina personuppgifter. För att läsa mer om hur detta går till, klicka här https://www.slu.se/om-slu/kontakta-slu/personuppgifter/ E-mailing SLU will result in SLU processing your personal data. For more information on how this is done, click here https://www.slu.se/en/about-slu/contact-slu/personal-data/

[MaestSi]

Ah, sorry. I don’t have any further suggestions, I would just try running a test Nextflow pipeline exploiting Singularity, as I’m not sure the error is MetONTIIME-specific. SM

[MaestSi]

Just an additional question. Is the file /crex/proj/[my_proj]/MetONTIIME/Output/importDb/db_sequences.qza created? Or, at least, is the /crex/proj/[my_proj]/MetONTIIME/Output/importDb/ folder created? If the answer to at least one question is yes, you may try importing another database, just for finding out if the error is due to the files you are trying to import. SM

[MaestSi]

Hi, I am going to close the issue due to inactivity. Feel free to reopen it in case you have any updates or tried to run the pipeline with the test dataset. Best, SM