search

mazzalab / fastqwiper

An ensemble method to recover corrupted FASTQ files, drop or fix pesky lines, remove unpaired reads, and settle reads interleaving.
GNU General Public License v3.0
24 stars 3 forks source link

unable to install mamba using conda as you suggested #5

Closed shanmugavadivelps closed 1 year ago

shanmugavadivelps commented 1 year ago

I am using ubuntu 22.04 LTS and using conda, I am trying to install MAMBA and snakemake.

But I am unable to install MAMBA.

is it work with python 2.7 environment?.

pls help me to solve the issue.

mazzalab commented 1 year ago

Hi, can you post here the commands and error message?

Anyway, if you use Docker, you can use the Docker image available from dockerhub:

docker pull mazzalab/fastqwiper

It will include both fastqwiper and the Snakemake workflows for complete wiping. Once downloaded the image, you can type:

docker run --rm -ti --name fastqwiper -v "YOUR_STATIC_PATH_TO_DATA_FOLDER:/fastqwiper/data" mazzalab/fastqwiper paired 8 sample

where:

  1. YOUR_STATIC_PATH_TO_DATA_FOLDER should be the path to the folder where you have the fastq.gz files to be wiped
  2. "paired" trigger the cleaning of R1 and R2 while "single" that of individual files
  3. "8" is the number of cores to be spawned;
  4. "sample" is part of the names of the files to be wiped. In this regard, remember that: 4.1 - for paired-end files (e.g., "sample_R1.fastq.gz" and "sample_R2.fastq.gz"), your files must finish with "_R1.fastq.gz" and "_R2.fastq.gz". The text to pass is everything before this text, "sample" in this case. 4.2 - for single end files (e.g., "excerpt_R1_001.fastq.gz"), your file must ends with the string ".fastq.gz"; all the preceding text, i.e., "excerpt_R1_001" will be the text to be passed to the command above:

docker run --rm -ti --name fastqwiper -v "YOUR_STATIC_PATH_TO_DATA_FOLDER:/fastqwiper/data" mazzalab/fastqwiper single 8 excerpt_R1_001

shanmugavadivelps commented 1 year ago

Thanks for your reply.

Sir/Madam,

I am using as below

fastqwiper --fastq_in SRR13556219_2.fastq.gz --fastq_out 22.fastq.gz Start wiping SRR13556219_2.fastq.gz 2023-08-24 09:53:53,391 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 500000 reads 2023-08-24 09:53:53,392 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 500000 reads 2023-08-24 09:53:53,392 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 500000 reads 2023-08-24 09:53:53,392 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 500000 reads 2023-08-24 09:55:04,876 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 1000000 reads 2023-08-24 09:55:04,876 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 1000000 reads 2023-08-24 09:55:04,876 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 1000000 reads 2023-08-24 09:55:04,876 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 1000000 reads 2023-08-24 09:55:04,876 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 1000000 reads 2023-08-24 09:55:04,876 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 1000000 reads 2023-08-24 09:55:04,876 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 1000000 reads 2023-08-24 09:55:04,876 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 1000000 reads Traceback (most recent call last): File "/home/microbiology/anaconda3/envs/FastqWiper/bin/fastqwiper", line 11, in sys.exit(wipe_fastq()) File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/site-packages/click/core.py", line 829, in call return self.main(args, kwargs) File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, ctx.params) File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(args, **kwargs) File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/site-packages/fastq_wiper/wiper.py", line 88, in wipe_fastq for line in fin: File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/gzip.py", line 305, in read1 return self._buffer.read1(size) File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/_compression.py", line 68, in readinto data = self.read(len(byte_view)) File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/gzip.py", line 470, in read self._read_eof() File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/gzip.py", line 516, in _read_eof raise BadGzipFile("CRC check failed %s != %s" % (hex(crc32), gzip.BadGzipFile: CRC check failed 0x7f073a84 != 0x6287761b

I am not using docker and I am not a pure bioinformatician. Here I have used a single file to solve the read issue. Pls help me to solve the issue

On Thu, Aug 24, 2023 at 12:14 AM MazzaLab @.***> wrote:

Hi, can you post here the commands and error message?

Anyway, if you use Docker, you can use the Docker image available from dockerhub:

docker pull mazzalab/fastqwiper

It will include both fastqwiper and the Snakemake workflows for complete wiping. Once downloaded the image, you can type:

docker run --rm -ti --name fastqwiper -v "YOUR_STATIC_PATH_TO_DATA_FOLDER:/fastqwiper/data" mazzalab/fastqwiper paired 8 sample

where:

  1. YOUR_STATIC_PATH_TO_DATA_FOLDER should be the path to the folder where you have the fastq.gz files to be wiped
  2. "paired" trigger the cleaning of R1 and R2 while "single" that of individual files
  3. "8" is the number of cores to be spawned;
  4. "sample" is part of the names of the files to be wiped. In this regard, remember that: 4.1 - for paired-end files (e.g., "sample_R1.fastq.gz" and "sample_R2.fastq.gz"), your files must finish with "_R1.fastq.gz" and "_R2.fastq.gz". The text to pass is everything before this text, "sample" in this case. 4.2 - for single end files (e.g., "excerpt_R1_001.fastq.gz"), your file must ends with the string ".fastq.gz"; all the preceding text, i.e., "excerpt_R1_001" will be the text to be passed to the command above:

docker run --rm -ti --name fastqwiper -v "YOUR_STATIC_PATH_TO_DATA_FOLDER:/fastqwiper/data" mazzalab/fastqwiper single 8 excerpt_R1_001

— Reply to this email directly, view it on GitHub https://github.com/mazzalab/fastqwiper/issues/5#issuecomment-1690459603, or unsubscribe https://github.com/notifications/unsubscribe-auth/AN2V7HUUU2V5RPLBXX4ZXATXWZFQJANCNFSM6AAAAAA33VR53M . You are receiving this because you authored the thread.Message ID: @.***>

-- Shanmugavadivel, P. S. Scientist (Agricultural Biotechnology),

#216, Block A, ICAR-Indian Institute of Pulses Research,

Min. of Agriculture & Farmers Welfare,

Govt. of India,Kanpur - 208 024. *email: @. @.> https://iipr.icar.gov.in/ https://iipr.icar.gov.in/*

mazzalab commented 1 year ago

OK, FastWiper alone (i.e., without Snakemake pipelines) is not able to read corrupted files. Yours seems to be corrupted.

If you cannot install Snakemake, please send me your file, I'll clean it for you.

best wishes,

tom

-

Tommaso Mazza, Eng. Ph.D.

Laboratory of Bioinformatics, PI

Fondazione IRCCS Casa Sollievo della Sofferenza

Viale Regina Margherita 261 - 00198 Roma IT

Tel: +39 06 44160526 - Fax: +39 06 44160548

E-mail: @.***

Web page: http://www.css-mendel.it/

Web page: http://bioinformatics.css-mendel.it

From: shanmugavadivelps @.***> Sent: Thursday, August 24, 2023 7:26 AM To: mazzalab/fastqwiper Cc: Tommaso Mazza; Comment Subject: Re: [mazzalab/fastqwiper] unable to install mamba using conda as you suggested (Issue #5)

Thanks for your reply.

Sir/Madam,

I am using as below

fastqwiper --fastq_in SRR13556219_2.fastq.gz --fastq_out 22.fastq.gz Start wiping SRR13556219_2.fastq.gz 2023-08-24 09:53:53,391 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 500000 reads 2023-08-24 09:53:53,392 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 500000 reads 2023-08-24 09:53:53,392 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 500000 reads 2023-08-24 09:53:53,392 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 500000 reads 2023-08-24 09:55:04,876 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 1000000 reads 2023-08-24 09:55:04,876 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 1000000 reads 2023-08-24 09:55:04,876 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 1000000 reads 2023-08-24 09:55:04,876 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 1000000 reads 2023-08-24 09:55:04,876 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 1000000 reads 2023-08-24 09:55:04,876 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 1000000 reads 2023-08-24 09:55:04,876 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 1000000 reads 2023-08-24 09:55:04,876 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 1000000 reads Traceback (most recent call last): File "/home/microbiology/anaconda3/envs/FastqWiper/bin/fastqwiper", line 11, in sys.exit(wipe_fastq()) File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/site-packages/click/core.py", line 829, in call return self.main(args, kwargs) File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, ctx.params) File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(args, **kwargs) File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/site-packages/fastq_wiper/wiper.py", line 88, in wipe_fastq for line in fin: File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/gzip.py", line 305, in read1 return self._buffer.read1(size) File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/_compression.py", line 68, in readinto data = self.read(len(byte_view)) File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/gzip.py", line 470, in read self._read_eof() File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/gzip.py", line 516, in _read_eof raise BadGzipFile("CRC check failed %s != %s" % (hex(crc32), gzip.BadGzipFile: CRC check failed 0x7f073a84 != 0x6287761b

I am not using docker and I am not a pure bioinformatician. Here I have used a single file to solve the read issue. Pls help me to solve the issue

On Thu, Aug 24, 2023 at 12:14 AM MazzaLab @.***> wrote:

Hi, can you post here the commands and error message?

Anyway, if you use Docker, you can use the Docker image available from dockerhub:

docker pull mazzalab/fastqwiper

It will include both fastqwiper and the Snakemake workflows for complete wiping. Once downloaded the image, you can type:

docker run --rm -ti --name fastqwiper -v "YOUR_STATIC_PATH_TO_DATA_FOLDER:/fastqwiper/data" mazzalab/fastqwiper paired 8 sample

where:

  1. YOUR_STATIC_PATH_TO_DATA_FOLDER should be the path to the folder where you have the fastq.gz files to be wiped
  2. "paired" trigger the cleaning of R1 and R2 while "single" that of individual files
  3. "8" is the number of cores to be spawned;
  4. "sample" is part of the names of the files to be wiped. In this regard, remember that: 4.1 - for paired-end files (e.g., "sample_R1.fastq.gz" and "sample_R2.fastq.gz"), your files must finish with "_R1.fastq.gz" and "_R2.fastq.gz". The text to pass is everything before this text, "sample" in this case. 4.2 - for single end files (e.g., "excerpt_R1_001.fastq.gz"), your file must ends with the string ".fastq.gz"; all the preceding text, i.e., "excerpt_R1_001" will be the text to be passed to the command above:

docker run --rm -ti --name fastqwiper -v "YOUR_STATIC_PATH_TO_DATA_FOLDER:/fastqwiper/data" mazzalab/fastqwiper single 8 excerpt_R1_001

— Reply to this email directly, view it on GitHub https://github.com/mazzalab/fastqwiper/issues/5#issuecomment-1690459603, or unsubscribe https://github.com/notifications/unsubscribe-auth/AN2V7HUUU2V5RPLBXX4ZXATXWZFQJANCNFSM6AAAAAA33VR53M . You are receiving this because you authored the thread.Message ID: @.***>

-- Shanmugavadivel, P. S. Scientist (Agricultural Biotechnology),

#216, Block A, ICAR-Indian Institute of Pulses Research,

Min. of Agriculture & Farmers Welfare,

Govt. of India,Kanpur - 208 024. *email: @. @.> https://iipr.icar.gov.in/ https://iipr.icar.gov.in/*

— Reply to this email directly, view it on GitHubhttps://github.com/mazzalab/fastqwiper/issues/5#issuecomment-1691025013, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACMMOGOTQJUE6FQNLFFKAFDXW3QZZANCNFSM6AAAAAA33VR53M. You are receiving this because you commented.Message ID: @.***>

shanmugavadivelps commented 1 year ago

i have installed now and executed as below py38) @.***:~ $ snakemake -s /home/microbiology/fastqwiper/pipeline/fix_wipe_pairs_reads.smk --use-conda --cores 2 Building DAG of jobs... Your conda installation is not configured to use strict channel priorities. This is however crucial for having robust and correct environments (for details, see https://conda-forge.org/docs/user/tipsandtricks.html). Please consider to configure strict priorities by executing 'conda config --set channel_priority strict'. Using shell: /usr/bin/bash Provided cores: 2 Rules claiming more threads will be scaled down. Job stats: job count min threads max threads

all 1 1 1 drop_unpaired 1 2 2 fix_interleaving 1 1 1 wipe_fastq 2 1 1 total 5 1 2

Select jobs to execute...

[Thu Aug 24 13:36:26 2023] rule wipe_fastq: input: data/sample_S1_R1_001_fixed.fastq output: data/sample_S1_R1_001_fixed_wiped.fastq.gz log: logs/wipe_fastq/wipe_fastq.sample_S1_R1_001.log jobid: 3 reason: Missing output files: data/sample_S1_R1_001_fixed_wiped.fastq.gz wildcards: sample=sample_S1_R1_001 resources: tmpdir=/tmp

[Thu Aug 24 13:36:26 2023] rule wipe_fastq: input: data/sample_S1_R2_001_fixed.fastq output: data/sample_S1_R2_001_fixed_wiped.fastq.gz log: logs/wipe_fastq/wipe_fastq.sample_S1_R2_001.log jobid: 5 reason: Missing output files: data/sample_S1_R2_001_fixed_wiped.fastq.gz wildcards: sample=sample_S1_R2_001 resources: tmpdir=/tmp

[Thu Aug 24 13:36:27 2023] Error in rule wipe_fastq: jobid: 5 input: data/sample_S1_R2_001_fixed.fastq output: data/sample_S1_R2_001_fixed_wiped.fastq.gz log: logs/wipe_fastq/wipe_fastq.sample_S1_R2_001.log (check log file(s) for error details) shell: python fastq_wiper/wiper.py --fastq_in data/sample_S1_R2_001_fixed.fastq --fastq_out data/sample_S1_R2_001_fixed_wiped.fastq.gz 2> logs/wipe_fastq/wipe_fastq.sample_S1_R2_001.log (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

[Thu Aug 24 13:36:27 2023] Error in rule wipe_fastq: jobid: 3 input: data/sample_S1_R1_001_fixed.fastq output: data/sample_S1_R1_001_fixed_wiped.fastq.gz log: logs/wipe_fastq/wipe_fastq.sample_S1_R1_001.log (check log file(s) for error details) shell: python fastq_wiper/wiper.py --fastq_in data/sample_S1_R1_001_fixed.fastq --fastq_out data/sample_S1_R1_001_fixed_wiped.fastq.gz 2> logs/wipe_fastq/wipe_fastq.sample_S1_R1_001.log (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: .snakemake/log/2023-08-24T133623.085748.snakemake.log

On Thu, Aug 24, 2023 at 1:03 PM MazzaLab @.***> wrote:

OK, FastWiper alone (i.e., without Snakemake pipelines) is not able to read corrupted files. Yours seems to be corrupted.

If you cannot install Snakemake, please send me your file, I'll clean it for you.

best wishes,

tom

-

Tommaso Mazza, Eng. Ph.D.

Laboratory of Bioinformatics, PI

Fondazione IRCCS Casa Sollievo della Sofferenza

Viale Regina Margherita 261 - 00198 Roma IT

Tel: +39 06 44160526 - Fax: +39 06 44160548

E-mail: @.***

Web page: http://www.css-mendel.it/

Web page: http://bioinformatics.css-mendel.it

From: shanmugavadivelps @.***> Sent: Thursday, August 24, 2023 7:26 AM To: mazzalab/fastqwiper Cc: Tommaso Mazza; Comment Subject: Re: [mazzalab/fastqwiper] unable to install mamba using conda as you suggested (Issue #5)

Thanks for your reply.

Sir/Madam,

I am using as below

fastqwiper --fastq_in SRR13556219_2.fastq.gz --fastq_out 22.fastq.gz Start wiping SRR13556219_2.fastq.gz 2023-08-24 09:53:53,391 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 500000 reads 2023-08-24 09:53:53,392 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 500000 reads 2023-08-24 09:53:53,392 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 500000 reads 2023-08-24 09:53:53,392 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 500000 reads 2023-08-24 09:55:04,876 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 1000000 reads 2023-08-24 09:55:04,876 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 1000000 reads 2023-08-24 09:55:04,876 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 1000000 reads 2023-08-24 09:55:04,876 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 1000000 reads 2023-08-24 09:55:04,876 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 1000000 reads 2023-08-24 09:55:04,876 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 1000000 reads 2023-08-24 09:55:04,876 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 1000000 reads 2023-08-24 09:55:04,876 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 1000000 reads Traceback (most recent call last): File "/home/microbiology/anaconda3/envs/FastqWiper/bin/fastqwiper", line 11, in sys.exit(wipe_fastq()) File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/site-packages/click/core.py",

line 829, in call return self.main(*args, **kwargs) File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/site-packages/click/core.py",

line 782, in main rv = self.invoke(ctx) File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/site-packages/click/core.py",

line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/site-packages/click/core.py",

line 610, in invoke return callback(*args, **kwargs) File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/site-packages/fastq_wiper/wiper.py",

line 88, in wipe_fastq for line in fin: File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/gzip.py", line 305, in read1 return self._buffer.read1(size) File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/_compression.py",

line 68, in readinto data = self.read(len(byte_view)) File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/gzip.py", line 470, in read self._read_eof() File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/gzip.py", line 516, in _read_eof raise BadGzipFile("CRC check failed %s != %s" % (hex(crc32), gzip.BadGzipFile: CRC check failed 0x7f073a84 != 0x6287761b

I am not using docker and I am not a pure bioinformatician. Here I have used a single file to solve the read issue. Pls help me to solve the issue

On Thu, Aug 24, 2023 at 12:14 AM MazzaLab @.***> wrote:

Hi, can you post here the commands and error message?

Anyway, if you use Docker, you can use the Docker image available from dockerhub:

docker pull mazzalab/fastqwiper

It will include both fastqwiper and the Snakemake workflows for complete wiping. Once downloaded the image, you can type:

docker run --rm -ti --name fastqwiper -v "YOUR_STATIC_PATH_TO_DATA_FOLDER:/fastqwiper/data" mazzalab/fastqwiper paired 8 sample

where:

  1. YOUR_STATIC_PATH_TO_DATA_FOLDER should be the path to the folder where you have the fastq.gz files to be wiped
  2. "paired" trigger the cleaning of R1 and R2 while "single" that of individual files
  3. "8" is the number of cores to be spawned;
  4. "sample" is part of the names of the files to be wiped. In this regard, remember that: 4.1 - for paired-end files (e.g., "sample_R1.fastq.gz" and "sample_R2.fastq.gz"), your files must finish with "_R1.fastq.gz" and "_R2.fastq.gz". The text to pass is everything before this text, "sample" in this case. 4.2 - for single end files (e.g., "excerpt_R1_001.fastq.gz"), your file must ends with the string ".fastq.gz"; all the preceding text, i.e., "excerpt_R1_001" will be the text to be passed to the command above:

docker run --rm -ti --name fastqwiper -v "YOUR_STATIC_PATH_TO_DATA_FOLDER:/fastqwiper/data" mazzalab/fastqwiper single 8 excerpt_R1_001

— Reply to this email directly, view it on GitHub https://github.com/mazzalab/fastqwiper/issues/5#issuecomment-1690459603,

or unsubscribe < https://github.com/notifications/unsubscribe-auth/AN2V7HUUU2V5RPLBXX4ZXATXWZFQJANCNFSM6AAAAAA33VR53M>

. You are receiving this because you authored the thread.Message ID: @.***>

-- Shanmugavadivel, P. S. Scientist (Agricultural Biotechnology),

#216, Block A, ICAR-Indian Institute of Pulses Research,

Min. of Agriculture & Farmers Welfare,

Govt. of India,Kanpur - 208 024. *email: @. @.> https://iipr.icar.gov.in/ https://iipr.icar.gov.in/*

— Reply to this email directly, view it on GitHub< https://github.com/mazzalab/fastqwiper/issues/5#issuecomment-1691025013>, or unsubscribe< https://github.com/notifications/unsubscribe-auth/ACMMOGOTQJUE6FQNLFFKAFDXW3QZZANCNFSM6AAAAAA33VR53M>.

You are receiving this because you commented.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/mazzalab/fastqwiper/issues/5#issuecomment-1691160150, or unsubscribe https://github.com/notifications/unsubscribe-auth/AN2V7HV7LWYYGFCL4GWSH23XW37UPANCNFSM6AAAAAA33VR53M . You are receiving this because you authored the thread.Message ID: @.***>

-- Shanmugavadivel, P. S. Scientist (Agricultural Biotechnology),

#216, Block A, ICAR-Indian Institute of Pulses Research,

Min. of Agriculture & Farmers Welfare,

Govt. of India,Kanpur - 208 024. *email: @. @.> https://iipr.icar.gov.in/ https://iipr.icar.gov.in/*

shanmugavadivelps commented 1 year ago

i have modified .smk file and attached for your reference

On Thu, Aug 24, 2023 at 1:38 PM @. < @.> wrote:

i have installed now and executed as below py38) @.***:~ $ snakemake -s /home/microbiology/fastqwiper/pipeline/fix_wipe_pairs_reads.smk --use-conda --cores 2 Building DAG of jobs... Your conda installation is not configured to use strict channel priorities. This is however crucial for having robust and correct environments (for details, see https://conda-forge.org/docs/user/tipsandtricks.html). Please consider to configure strict priorities by executing 'conda config --set channel_priority strict'. Using shell: /usr/bin/bash Provided cores: 2 Rules claiming more threads will be scaled down. Job stats: job count min threads max threads

all 1 1 1 drop_unpaired 1 2 2 fix_interleaving 1 1 1 wipe_fastq 2 1 1 total 5 1 2

Select jobs to execute...

[Thu Aug 24 13:36:26 2023] rule wipe_fastq: input: data/sample_S1_R1_001_fixed.fastq output: data/sample_S1_R1_001_fixed_wiped.fastq.gz log: logs/wipe_fastq/wipe_fastq.sample_S1_R1_001.log jobid: 3 reason: Missing output files: data/sample_S1_R1_001_fixed_wiped.fastq.gz wildcards: sample=sample_S1_R1_001 resources: tmpdir=/tmp

[Thu Aug 24 13:36:26 2023] rule wipe_fastq: input: data/sample_S1_R2_001_fixed.fastq output: data/sample_S1_R2_001_fixed_wiped.fastq.gz log: logs/wipe_fastq/wipe_fastq.sample_S1_R2_001.log jobid: 5 reason: Missing output files: data/sample_S1_R2_001_fixed_wiped.fastq.gz wildcards: sample=sample_S1_R2_001 resources: tmpdir=/tmp

[Thu Aug 24 13:36:27 2023] Error in rule wipe_fastq: jobid: 5 input: data/sample_S1_R2_001_fixed.fastq output: data/sample_S1_R2_001_fixed_wiped.fastq.gz log: logs/wipe_fastq/wipe_fastq.sample_S1_R2_001.log (check log file(s) for error details) shell: python fastq_wiper/wiper.py --fastq_in data/sample_S1_R2_001_fixed.fastq --fastq_out data/sample_S1_R2_001_fixed_wiped.fastq.gz 2> logs/wipe_fastq/wipe_fastq.sample_S1_R2_001.log (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

[Thu Aug 24 13:36:27 2023] Error in rule wipe_fastq: jobid: 3 input: data/sample_S1_R1_001_fixed.fastq output: data/sample_S1_R1_001_fixed_wiped.fastq.gz log: logs/wipe_fastq/wipe_fastq.sample_S1_R1_001.log (check log file(s) for error details) shell: python fastq_wiper/wiper.py --fastq_in data/sample_S1_R1_001_fixed.fastq --fastq_out data/sample_S1_R1_001_fixed_wiped.fastq.gz 2> logs/wipe_fastq/wipe_fastq.sample_S1_R1_001.log (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: .snakemake/log/2023-08-24T133623.085748.snakemake.log

On Thu, Aug 24, 2023 at 1:03 PM MazzaLab @.***> wrote:

OK, FastWiper alone (i.e., without Snakemake pipelines) is not able to read corrupted files. Yours seems to be corrupted.

If you cannot install Snakemake, please send me your file, I'll clean it for you.

best wishes,

tom

-

Tommaso Mazza, Eng. Ph.D.

Laboratory of Bioinformatics, PI

Fondazione IRCCS Casa Sollievo della Sofferenza

Viale Regina Margherita 261 - 00198 Roma IT

Tel: +39 06 44160526 - Fax: +39 06 44160548

E-mail: @.***

Web page: http://www.css-mendel.it/

Web page: http://bioinformatics.css-mendel.it

From: shanmugavadivelps @.***> Sent: Thursday, August 24, 2023 7:26 AM To: mazzalab/fastqwiper Cc: Tommaso Mazza; Comment Subject: Re: [mazzalab/fastqwiper] unable to install mamba using conda as you suggested (Issue #5)

Thanks for your reply.

Sir/Madam,

I am using as below

fastqwiper --fastq_in SRR13556219_2.fastq.gz --fastq_out 22.fastq.gz Start wiping SRR13556219_2.fastq.gz 2023-08-24 09:53:53,391 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 500000 reads 2023-08-24 09:53:53,392 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 500000 reads 2023-08-24 09:53:53,392 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 500000 reads 2023-08-24 09:53:53,392 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 500000 reads 2023-08-24 09:55:04,876 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 1000000 reads 2023-08-24 09:55:04,876 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 1000000 reads 2023-08-24 09:55:04,876 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 1000000 reads 2023-08-24 09:55:04,876 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 1000000 reads 2023-08-24 09:55:04,876 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 1000000 reads 2023-08-24 09:55:04,876 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 1000000 reads 2023-08-24 09:55:04,876 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 1000000 reads 2023-08-24 09:55:04,876 - fastq_wiper/wiper.py(wipe_fastq) - INFO - Cleaned 1000000 reads Traceback (most recent call last): File "/home/microbiology/anaconda3/envs/FastqWiper/bin/fastqwiper", line 11, in sys.exit(wipe_fastq()) File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/site-packages/click/core.py",

line 829, in call return self.main(*args, **kwargs) File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/site-packages/click/core.py",

line 782, in main rv = self.invoke(ctx) File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/site-packages/click/core.py",

line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/site-packages/click/core.py",

line 610, in invoke return callback(*args, **kwargs) File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/site-packages/fastq_wiper/wiper.py",

line 88, in wipe_fastq for line in fin: File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/gzip.py", line 305, in read1 return self._buffer.read1(size) File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/_compression.py",

line 68, in readinto data = self.read(len(byte_view)) File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/gzip.py", line 470, in read self._read_eof() File "/home/microbiology/anaconda3/envs/FastqWiper/lib/python3.8/gzip.py", line 516, in _read_eof raise BadGzipFile("CRC check failed %s != %s" % (hex(crc32), gzip.BadGzipFile: CRC check failed 0x7f073a84 != 0x6287761b

I am not using docker and I am not a pure bioinformatician. Here I have used a single file to solve the read issue. Pls help me to solve the issue

On Thu, Aug 24, 2023 at 12:14 AM MazzaLab @.***> wrote:

Hi, can you post here the commands and error message?

Anyway, if you use Docker, you can use the Docker image available from dockerhub:

docker pull mazzalab/fastqwiper

It will include both fastqwiper and the Snakemake workflows for complete wiping. Once downloaded the image, you can type:

docker run --rm -ti --name fastqwiper -v "YOUR_STATIC_PATH_TO_DATA_FOLDER:/fastqwiper/data" mazzalab/fastqwiper paired 8 sample

where:

  1. YOUR_STATIC_PATH_TO_DATA_FOLDER should be the path to the folder where you have the fastq.gz files to be wiped
  2. "paired" trigger the cleaning of R1 and R2 while "single" that of individual files
  3. "8" is the number of cores to be spawned;
  4. "sample" is part of the names of the files to be wiped. In this regard, remember that: 4.1 - for paired-end files (e.g., "sample_R1.fastq.gz" and "sample_R2.fastq.gz"), your files must finish with "_R1.fastq.gz" and "_R2.fastq.gz". The text to pass is everything before this text, "sample" in this case. 4.2 - for single end files (e.g., "excerpt_R1_001.fastq.gz"), your file must ends with the string ".fastq.gz"; all the preceding text, i.e., "excerpt_R1_001" will be the text to be passed to the command above:

docker run --rm -ti --name fastqwiper -v "YOUR_STATIC_PATH_TO_DATA_FOLDER:/fastqwiper/data" mazzalab/fastqwiper single 8 excerpt_R1_001

— Reply to this email directly, view it on GitHub < https://github.com/mazzalab/fastqwiper/issues/5#issuecomment-1690459603>,

or unsubscribe < https://github.com/notifications/unsubscribe-auth/AN2V7HUUU2V5RPLBXX4ZXATXWZFQJANCNFSM6AAAAAA33VR53M>

. You are receiving this because you authored the thread.Message ID: @.***>

-- Shanmugavadivel, P. S. Scientist (Agricultural Biotechnology),

#216, Block A, ICAR-Indian Institute of Pulses Research,

Min. of Agriculture & Farmers Welfare,

Govt. of India,Kanpur - 208 024. *email: @. @.> https://iipr.icar.gov.in/ https://iipr.icar.gov.in/*

— Reply to this email directly, view it on GitHub< https://github.com/mazzalab/fastqwiper/issues/5#issuecomment-1691025013>, or unsubscribe< https://github.com/notifications/unsubscribe-auth/ACMMOGOTQJUE6FQNLFFKAFDXW3QZZANCNFSM6AAAAAA33VR53M>.

You are receiving this because you commented.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/mazzalab/fastqwiper/issues/5#issuecomment-1691160150, or unsubscribe https://github.com/notifications/unsubscribe-auth/AN2V7HV7LWYYGFCL4GWSH23XW37UPANCNFSM6AAAAAA33VR53M . You are receiving this because you authored the thread.Message ID: @.***>

-- Shanmugavadivel, P. S. Scientist (Agricultural Biotechnology),

#216, Block A, ICAR-Indian Institute of Pulses Research,

Min. of Agriculture & Farmers Welfare,

Govt. of India,Kanpur - 208 024. *email: @. @.> https://iipr.icar.gov.in/ https://iipr.icar.gov.in/*

-- Shanmugavadivel, P. S. Scientist (Agricultural Biotechnology),

#216, Block A, ICAR-Indian Institute of Pulses Research,

Min. of Agriculture & Farmers Welfare,

Govt. of India,Kanpur - 208 024. *email: @. @.> https://iipr.icar.gov.in/ https://iipr.icar.gov.in/*

mazzalab commented 1 year ago

I cannot see the attached file Snakemake file. Please attach it through the GitHub web interface.

Can you also send me these files? logs/wipe_fastq/wipe_fastq.sample_S1_R2_001.log logs/wipe_fastq/wipe_fastq.sample_S1_R1_001.log

shanmugavadivelps commented 1 year ago

wipe_fastq.sample_S1_R1_001.log wipe_fastq.sample_S1_R2_001.log

shanmugavadivelps commented 1 year ago

fix_gzrt.sample_S1_R1_001.log fix_gzrt.sample_S1_R2_001.log

shanmugavadivelps commented 1 year ago

[Uploading fix_wipe_pairs_reads.smk…]()

mazzalab commented 1 year ago

wipe_fastq.sample_S1_R1_001.log wipe_fastq.sample_S1_R2_001.log

OK it seems it does not find the fastqwiper executable. Can you edit the "shell command " of tje Snakemake workflow here: image

into:

shell:
        "fastqwiper --fastq_in {input} --fastq_out {output} 2> {log}"
mazzalab commented 1 year ago

you need to execute the workflow within the cloned repository; where you have executed:

git clone https://github.com/mazzalab/fastqwiper.git

shanmugavadivelps commented 1 year ago

i have done as you said.

still i am encountering a failure

Error in rule fix_interleaving: jobid: 1 input: data/sample_S1_R1_001_fixed_wiped_paired.fastq.gz, data/sample_S1_R2_001_fixed_wiped_paired.fastq.gz output: data/sample_S1_R1_001_fixed_wiped_paired_interleaving.fastq.gz, data/sample_S1_R2_001_fixed_wiped_paired_interleaving.fastq.gz log: logs/pairing/pairing.sample.log (check log file(s) for error details) shell: bbmap/repair.sh in=data/sample_S1_R1_001_fixed_wiped_paired.fastq.gz in2=data/sample_S1_R2_001_fixed_wiped_paired.fastq.gz out=data/sample_S1_R1_001_fixed_wiped_paired_interleaving.fastq.gz out2=data/sample_S1_R2_001_fixed_wiped_paired_interleaving.fastq.gz outsingle=singletons.fastq.gz 2> logs/pairing/pairing.sample.log (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

[Thu Aug 24 15:03:33 2023] Error in rule fix_interleaving: jobid: 7 input: data/excerpt_S1_R1_001_fixed_wiped_paired.fastq.gz, data/excerpt_S1_R2_001_fixed_wiped_paired.fastq.gz output: data/excerpt_S1_R1_001_fixed_wiped_paired_interleaving.fastq.gz, data/excerpt_S1_R2_001_fixed_wiped_paired_interleaving.fastq.gz log: logs/pairing/pairing.excerpt.log (check log file(s) for error details) shell: bbmap/repair.sh in=data/excerpt_S1_R1_001_fixed_wiped_paired.fastq.gz in2=data/excerpt_S1_R2_001_fixed_wiped_paired.fastq.gz out=data/excerpt_S1_R1_001_fixed_wiped_paired_interleaving.fastq.gz out2=data/excerpt_S1_R2_001_fixed_wiped_paired_interleaving.fastq.gz outsingle=singletons.fastq.gz 2> logs/pairing/pairing.excerpt.log (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: .snakemake/log/2023-08-24T150326.687136.snakemake.log

also attaching log files wipe_fastq.excerpt_S1_R1_001.log wipe_fastq.excerpt_S1_R2_001.log wipe_fastq.sample_S1_R1_001.log wipe_fastq.sample_S1_R2_001.log pairing.excerpt.log pairing.sample.log fix_gzrt.excerpt_S1_R1_001.log fix_gzrt.excerpt_S1_R2_001.log fix_gzrt.sample_S1_R1_001.log fix_gzrt.sample_S1_R2_001.log

shanmugavadivelps commented 1 year ago

Now its working fine. I have downloaded BBmap again and its working fine.

mazzalab commented 1 year ago

great!

Anyway, I'll make and share a video-tutorial to explain installation and usage soon.

shanmugavadivelps commented 1 year ago

Thanks for your support sir. Really happy to useSent from my iPhoneOn 24-Aug-2023, at 4:39 PM, MazzaLab @.***> wrote: great! Anyway, I'll make and share a video-tutorial to explain installation and usage soon.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you modified the open/close state.Message ID: @.***>