cannot remove 'nospace.*': No such file or directory

[Tclack88]

Hello, I ran through the demo data given and there wasn't a problem. I have a few fastq files that I converted to fastq.gz using the provided convertIllumunaRawDataToEnaFormat.sh. This created a directory and file in the mydata dir (namely mapping_files_for_PEMA.txt rawDataInEnaFormat/) I ran with those files in the directory and got an error (as others have gotten by keeping the README.md as I thought I addressed initially here with Akhilbiju01's question) I promptly moved them out so that I only have the fastq.gz files and am getting this error involving no "no space*". I looked through the source code in PEMA_v1.2.bds line 507 to 515 the file is created (a temp file I imagine) then deleted. It appears this non-existent folder is supposed to # merge all lines of a fastq entry into one and only one line given by this line: sys awk 'NR==1 {print ; next} {printf /^>/ ? "\n"$0"\n" : $1} END {printf "\n"}' se.$derepl > nospace.$derepl

So at this point I'm uncertain if the data is bad or if there's bug in the code. Here's my full output from the run:

$ singularity run  -B /p/home/tclack/bio/pema-1.2/test/analysis_folder/:/mnt/analysis ./pema_v.1.1.sif
Picked up JAVA_TOOL_OPTIONS: -XX:+UseContainerSupport
this ouput file already exists
1E_S29_L001_R1_001.fastq.gz1E_S29_L001_R2_001.fastq.gz2G_S42_L001_R2_001.fastq.gz2H_S48_L001_R1_001.fastq.gz3C_S17_L001_R1_001.fastq.gz3C_S17_L001_R2_001.fastq.gzperl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = "en_US.UTF-8",
    LC_ALL = (unset),
    LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = "en_US.UTF-8",
    LC_ALL = (unset),
    LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = "en_US.UTF-8",
    LC_ALL = (unset),
    LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = "en_US.UTF-8",
    LC_ALL = (unset),
    LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = "en_US.UTF-8",
    LC_ALL = (unset),
    LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
Picked up JAVA_TOOL_OPTIONS: -XX:+UseContainerSupport
Picked up JAVA_TOOL_OPTIONS: -XX:+UseContainerSupport
Picked up JAVA_TOOL_OPTIONS: -XX:+UseContainerSupport
Picked up JAVA_TOOL_OPTIONS: -XX:+UseContainerSupport
Picked up JAVA_TOOL_OPTIONS: -XX:+UseContainerSupport
Started analysis of 3C_S17_L001_R1_001.fastq.gz
Started analysis of 2H_S48_L001_R1_001.fastq.gz
Started analysis of 1E_S29_L001_R1_001.fastq.gz
Started analysis of 1E_S29_L001_R2_001.fastq.gz
Started analysis of 2G_S42_L001_R2_001.fastq.gz
Approx 5% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 5% complete for 2H_S48_L001_R1_001.fastq.gz
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = "en_US.UTF-8",
    LC_ALL = (unset),
    LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
Picked up JAVA_TOOL_OPTIONS: -XX:+UseContainerSupport
Started analysis of 3C_S17_L001_R2_001.fastq.gz
Approx 5% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 10% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 10% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 15% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 5% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 10% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 15% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 15% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 20% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 10% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 20% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 25% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 15% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 5% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 5% complete for 1E_S29_L001_R2_001.fastq.gz
Approx 20% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 25% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 25% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 30% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 20% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 30% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 30% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 35% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 40% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 25% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 35% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 35% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 45% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 50% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 30% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 10% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 40% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 45% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 40% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 55% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 35% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 10% complete for 1E_S29_L001_R2_001.fastq.gz
Approx 50% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 45% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 60% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 55% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 60% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 50% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 65% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 70% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 40% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 15% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 65% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 55% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 75% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 45% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 70% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 60% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 80% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 50% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 15% complete for 1E_S29_L001_R2_001.fastq.gz
Approx 75% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 65% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 85% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 55% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 20% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 80% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 70% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 90% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 95% complete for 2H_S48_L001_R1_001.fastq.gz
Approx 60% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 85% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 75% complete for 3C_S17_L001_R1_001.fastq.gz
Analysis complete for 2H_S48_L001_R1_001.fastq.gz
Approx 65% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 20% complete for 1E_S29_L001_R2_001.fastq.gz
Approx 90% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 95% complete for 2G_S42_L001_R2_001.fastq.gz
Approx 80% complete for 3C_S17_L001_R1_001.fastq.gz
Analysis complete for 2G_S42_L001_R2_001.fastq.gz
Approx 70% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 25% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 85% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 75% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 90% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 80% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 95% complete for 3C_S17_L001_R1_001.fastq.gz
Approx 30% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 25% complete for 1E_S29_L001_R2_001.fastq.gz
Analysis complete for 3C_S17_L001_R1_001.fastq.gz
Approx 85% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 90% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 95% complete for 3C_S17_L001_R2_001.fastq.gz
Approx 35% complete for 1E_S29_L001_R1_001.fastq.gz
Analysis complete for 3C_S17_L001_R2_001.fastq.gz
Approx 30% complete for 1E_S29_L001_R2_001.fastq.gz
Approx 40% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 35% complete for 1E_S29_L001_R2_001.fastq.gz
Approx 45% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 50% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 40% complete for 1E_S29_L001_R2_001.fastq.gz
Approx 55% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 45% complete for 1E_S29_L001_R2_001.fastq.gz
Approx 60% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 50% complete for 1E_S29_L001_R2_001.fastq.gz
Approx 55% complete for 1E_S29_L001_R2_001.fastq.gz
Approx 65% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 60% complete for 1E_S29_L001_R2_001.fastq.gz
Approx 70% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 65% complete for 1E_S29_L001_R2_001.fastq.gz
Approx 75% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 70% complete for 1E_S29_L001_R2_001.fastq.gz
Approx 80% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 85% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 75% complete for 1E_S29_L001_R2_001.fastq.gz
Approx 80% complete for 1E_S29_L001_R2_001.fastq.gz
Approx 90% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 95% complete for 1E_S29_L001_R1_001.fastq.gz
Approx 85% complete for 1E_S29_L001_R2_001.fastq.gz
Analysis complete for 1E_S29_L001_R1_001.fastq.gz
Approx 90% complete for 1E_S29_L001_R2_001.fastq.gz
Approx 95% complete for 1E_S29_L001_R2_001.fastq.gz
Analysis complete for 1E_S29_L001_R2_001.fastq.gz
FastQC is completed!
readF is: 1E_S29_L001_R1_001.fastq.gz
readF is: 1E_S29_L001_R2_001.fastq.gz
readF is: 2G_S42_L001_R2_001.fastq.gz
readF is: 2H_S48_L001_R1_001.fastq.gz
readF is: 3C_S17_L001_R1_001.fastq.gz
readF is: 3C_S17_L001_R2_001.fastq.gz
Trimmomatic  is done

Error correction using BayesHammer is completed!
Merging step by SPAdes is completed
all the first steps are done! clustering is about to start!
rm: cannot remove 'se.*': No such file or directory
rm: cannot remove 'nospace.*': No such file or directory
Fatal error: /home/PEMA_v1.bds, line 517, pos 1. Exec failed.
    Exit value : 1
    Command    :  rm se.* nospace.*
PEMA_v1.bds, line 517 : sys rm se.* nospace.*

[Tclack88]

@hariszaf

If there's anything I can provide to further debug or reproduce, I would be more than happy to

[hariszaf]

Hi there! Sorry for the late reply!

Could you tell me which pema container and version are you using and on what OS ?

[Tclack88]

All good, it's the holiday season afterall and I'm sure you're probably quit busy

I've tried on SUSE and Fedora

SLES (SUSE Linux Enterprise Server) version 12 SP3 -- x86_64 architecture if it maters and Red Hat Enterpise Linux Server 7.7 (Maipo)

And PEMA 1.2 using Singularity (V 3.5.3)

I can also send some sample files over if you need to recreate the error with that

[hariszaf]

Many wishes for a happy new year! :)

Could you try first to run the PEMA v.1.3.2 container? I think this should work fine as after v1.3.1 the converting part is included in pema.

You should check on having the latest parameters.tsv file as well.

I am actually working on pema these days trying to fix some things, make the repo a bit better and also add a new feature enabling custom ref dbs, so my apologies for any inconvenience! I hope I will have everything ready quite soon! :slightly_smiling_face:

[Tclack88]

I pulled the latest singularity container. 1.3.2. I've tried te parameters.tsv as you currently have on here in /pema/analysis_directory/parameters.tsv

That was last updated 4 months ago. I'm uncertain if you have a newer version

Same error. As I said, I can send you some sample files fastqz if it may be helpful

[hariszaf]

Yes we should go for that.

You may send me a sample at pema@hcmr.gr or at haris-zaf@hcmr.gr.

and I ll check this as soon as possible.

[Tclack88]

Great! Sent to your pema email.

Thanks again for your support

[Tclack88]

For future readers, this issue was likely caused from not compressing my .fastq files. I misread a response on gitter and thought I needed to use the convertIllumunaRawDataToEnaFormat.sh on my .fastq files. But nope, I just needed to gzip them. Strangely this "concert to ena" shell script gave me gzipped files, but resulted in the "nospace" error