search

hsgweon / pipits

Automated pipeline for analyses of fungal ITS from the Illumina
GNU General Public License v3.0
30 stars 16 forks source link

Stucked in the stage 'PIPITS_PROCESS' #18

Closed DH1983 closed 5 years ago

DH1983 commented 6 years ago

Hi, Hyun,

I followed your instructions, but I can not move forward from the stage 'PIPITS_PROCESS'.

I am running a test with your test file, and my problem is indicated below.

pipits_process 2.2, the PIPITS Project https://github.com/hsgweon/pipits

2018-09-10 17:27:16 pipits_process started 2018-09-10 17:27:16 Generating a sample list from the input sequences 2018-09-10 17:27:16 Dereplicating and removing unique sequences prior to picking OTUs vsearch v2.8.0_linux_x86_64, 7.6GB RAM, 4 cores https://github.com/torognes/vsearch

Reading file out_funits/ITS.fasta 100% 8584 nt in 52 seqs, min 144, max 275, avg 165 Dereplicating 100% Sorting 100% 15 unique sequences, avg cluster 3.5, median 2, max 12 Writing output file 100% 9 uniques written, 6 clusters discarded (40.0%) 2018-09-10 17:27:16 Picking OTUs [VSEARCH] vsearch v2.8.0_linux_x86_64, 7.6GB RAM, 4 cores https://github.com/torognes/vsearch

Reading file out_process/intermediate/input_nr.fasta 100% 1447 nt in 9 seqs, min 147, max 187, avg 161 Masking 100% Sorting by length 100% Counting k-mers 100% Clustering 100% Sorting clusters 100% Writing clusters 100% Clusters: 6 Size min 1, max 3, avg 1.5 Singletons: 4, 44.4% of seqs, 66.7% of clusters 2018-09-10 17:27:16 Removing chimeras [VSEARCH] vsearch v2.8.0_linux_x86_64, 7.6GB RAM, 4 cores https://github.com/torognes/vsearch

Unable to open file for reading ($HOME/pipits/refdb/uchime_reference_dataset_28.06.2017/uchime_reference_dataset_28.06.2017.fasta) 2018-09-10 17:27:16 Error: None zero returncode: vsearch --uchime_ref out_process/intermediate/input_nr_otus.fasta --db $PIPITS_UNITE_REFERENCE_DATA_CHIMERA --nonchimeras out_process/intermediate/input_nr_otus_nonchimeras.fasta --threads 1

I am a beginner, so would you please let me know in details, so that I can follow it.

Many thanks in advance.

DongHyeon

hsgweon commented 6 years ago

@DH1983 It appears that PIPITS can't find your the uchime reference dataset.

Can you make sure you have uchime_reference_dataset_28.06.2017.fasta in $HOME/pipits/refdb/uchime_reference_dataset_28.06.2017?

You can do so by: 

ls $HOME/pipits/refdb/uchime_reference_dataset_28.06.2017/uchime_reference_dataset_28.06.2017.fasta

You probably missed a step or two, but please do have a look at "A3. Reference datasets" in the manual. https://github.com/hsgweon/pipits#a3-reference-datasets

DH1983 commented 6 years ago

Thank you very much for your quick answer.

I have checked that the reference dataset is placed in the directory that PIPITS are trying to find by doing what you have instructed me.

This is what i have got from the terminal in my computer (pipits_env) lh001@lh001-300E4C-300E5C-300E7C:~/Desktop/PIPITS_Trial$ ls $HOME/pipits/refdb/uchime_reference_dataset_28.06.2017/uchime_reference_dataset_28.06.2017.fasta /home/lh001/pipits/refdb/uchime_reference_dataset_28.06.2017/uchime_reference_dataset_28.06.2017.fasta

However, I still got the same error message...

I supposed the problem arises from the A4 stage.

Would please checked if I do correctly?

This is how I did. I edited the file that is located in /home/environment by adding the following commands export PIPITS_UNITE_RETRAINED_DIR=$HOME/pipits/refdb/UNITE_retrained export PIPITS_UNITE_REFERENCE_DATA_CHIMERA=$HOME/pipits/refdb/uchime_reference_dataset_28.06.2017/uchime_reference_dataset_28.06.2017.fasta export PIPITS_WARCUP_RETRAINED_DIR=$HOME/pipits/refdb/warcup_retrained_V2 and saved, and subsequently restarted the terminal.

Could please tell me where the problem is arising from?

Thank you again for your help.

DongHyeon

hsgweon commented 6 years ago

Can you try:

echo $PIPITS_UNITE_REFERENCE_DATA_CHIMERA

and see if you get:

/home/lh001/pipits/refdb/uchime_reference_dataset_28.06.2017/uchime_reference_dataset_28.06.2017.fasta
DH1983 commented 6 years ago

Hi, Hyun,

This is what I got from the terminal.

lh001@lh001-300E4C-300E5C-300E7C:~$ echo $PIPITS_UNITE_REFERENCE_DATA_CHIMERA $HOME/pipits/refdb/uchime_reference_dataset_28.06.2017/uchime_reference_dataset_28.06.2017.fasta

hsgweon commented 6 years ago

Can you try and let me know the result?

vsearch --uchime_ref out_process/intermediate/input_nr_otus.fasta --db $PIPITS_UNITE_REFERENCE_DATA_CHIMERA --nonchimeras out_process/intermediate/input_nr_otus_nonchimeras.fasta --threads 1
DH1983 commented 6 years ago

Here, What I got..

lh001@lh001-300E4C-300E5C-300E7C:~/pipits$ vsearch --uchime_ref out_process/intermediate/input_nr_otus.fasta --db $PIPITS_UNITE_REFERENCE_DATA_CHIMERA --nonchimeras out_process/intermediate/input_nr_otus_nonchimeras.fasta --threads 1 vsearch v2.7.1_linux_x86_64, 7.6GB RAM, 4 cores https://github.com/torognes/vsearch Unable to open file out_process/intermediate/input_nr_otus_nonchimeras.fasta for writing

hsgweon commented 6 years ago

Let's start this from the beginning:

Can you try this again after running pipits_funits?

pipits_process -i out_funits/ITS.fasta -o out_process -v -r
hsgweon commented 6 years ago

We will get this sorted!!

DH1983 commented 6 years ago

Thank you for your time, I will try it out!

DH1983 commented 6 years ago

Hi Hyun, Below is what I got from the log file after I ran it

pipits_process 2.2, the PIPITS Project https://github.com/hsgweon/pipits

2018-09-10 19:27:14 pipits_process started 2018-09-10 19:27:14 Generating a sample list from the input sequences 2018-09-10 19:27:14 Dereplicating and removing unique sequences prior to picking OTUs vsearch v2.8.0_linux_x86_64, 7.6GB RAM, 4 cores https://github.com/torognes/vsearch

Reading file out_funits/ITS.fasta 100% 8584 nt in 52 seqs, min 144, max 275, avg 165 Dereplicating 100% Sorting 100% 15 unique sequences, avg cluster 3.5, median 2, max 12 Writing output file 100% 9 uniques written, 6 clusters discarded (40.0%) 2018-09-10 19:27:14 Picking OTUs [VSEARCH] vsearch v2.8.0_linux_x86_64, 7.6GB RAM, 4 cores https://github.com/torognes/vsearch

Reading file out_process/intermediate/input_nr.fasta 100% 1447 nt in 9 seqs, min 147, max 187, avg 161 Masking 100% Sorting by length 100% Counting k-mers 100% Clustering 100% Sorting clusters 100% Writing clusters 100% Clusters: 6 Size min 1, max 3, avg 1.5 Singletons: 4, 44.4% of seqs, 66.7% of clusters 2018-09-10 19:27:14 Removing chimeras [VSEARCH] vsearch v2.8.0_linux_x86_64, 7.6GB RAM, 4 cores https://github.com/torognes/vsearch

Unable to open file for reading ($HOME/pipits/refdb/uchime_reference_dataset_28.06.2017/uchime_reference_dataset_28.06.2017.fasta) 2018-09-10 19:27:14 Error: None zero returncode: vsearch --uchime_ref out_process/intermediate/input_nr_otus.fasta --db $PIPITS_UNITE_REFERENCE_DATA_CHIMERA --nonchimeras out_process/intermediate/input_nr_otus_nonchimeras.fasta --threads 1

hsgweon commented 6 years ago

Ok, now can you try:

vsearch --uchime_ref out_process/intermediate/input_nr_otus.fasta --db $PIPITS_UNITE_REFERENCE_DATA_CHIMERA --nonchimeras out_process/intermediate/input_nr_otus_nonchimeras.fasta --threads 1
DH1983 commented 6 years ago

Here what I got in this time.

(pipits_env) lh001@lh001-300E4C-300E5C-300E7C:~/Desktop/PIPITS_Trial$ vsearch --uchime_ref out_process/intermediate/input_nr_otus.fasta --db $PIPITS_UNITE_REFERENCE_DATA_CHIMERA --nonchimeras out_process/intermediate/input_nr_otus_nonchimeras.fasta --threads 1 vsearch v2.8.0_linux_x86_64, 7.6GB RAM, 4 cores https://github.com/torognes/vsearch

Unable to open file for reading ($HOME/pipits/refdb/uchime_reference_dataset_28.06.2017/uchime_reference_dataset_28.06.2017.fasta)

hsgweon commented 6 years ago

What about:

vsearch --uchime_ref out_process/intermediate/input_nr_otus.fasta --db ~/pipits/refdb/uchime_reference_dataset_28.06.2017/uchime_reference_dataset_28.06.2017.fasta --nonchimeras out_process/intermediate/input_nr_otus_nonchimeras.fasta --threads 1
DH1983 commented 6 years ago

It has run it through, and the last block that I have from the terminal is below.

16786547 nt in 30555 seqs, min 146, max 2570, avg 549 Masking 100%
Counting k-mers 100%
Creating k-mer index 100%
Detecting chimeras 100%
Found 0 (0.0%) chimeras, 6 (100.0%) non-chimeras, and 0 (0.0%) borderline sequences in 6 unique sequences. Taking abundance information into account, this corresponds to 0 (0.0%) chimeras, 37 (100.0%) non-chimeras, and 0 (0.0%) borderline sequences in 37 total sequences.

hsgweon commented 6 years ago

Hmm... are you on Ubuntu or macOS?

DH1983 commented 6 years ago

I am running in Ubuntu.

hsgweon commented 6 years ago

Can you try this, and see if this works?

vsearch --uchime_ref out_process/intermediate/input_nr_otus.fasta --db $HOME/pipits/refdb/uchime_reference_dataset_28.06.2017/uchime_reference_dataset_28.06.2017.fasta --nonchimeras out_process/intermediate/input_nr_otus_nonchimeras.fasta --threads 1
DH1983 commented 6 years ago

This seems working too.

Here is what I got from the terminal

_reference_dataset_28.06.2017.fasta 100% 16786547 nt in 30555 seqs, min 146, max 2570, avg 549 Masking 100%
Counting k-mers 100%
Creating k-mer index 100%
Detecting chimeras 100%
Found 0 (0.0%) chimeras, 6 (100.0%) non-chimeras, and 0 (0.0%) borderline sequences in 6 unique sequences. Taking abundance information into account, this corresponds to 0 (0.0%) chimeras, 37 (100.0%) non-chimeras, and 0 (0.0%) borderline sequences in 37 total sequences.

hsgweon commented 6 years ago

This is weird... can you confirm that the output from:

echo $PIPITS_UNITE_REFERENCE_DATA_CHIMERA

is the same as

ls $HOME/pipits/refdb/uchime_reference_dataset_28.06.2017/uchime_reference_dataset_28.06.2017.fasta
DH1983 commented 6 years ago

For the first command

and for the second one (pipits_env) lh001@lh001-300E4C-300E5C-300E7C:~/Desktop/PIPITS_Trial$ ls $HOME/pipits/refdb/uchime_reference_dataset_28.06.2017/uchime_reference_dataset_28.06.2017.fasta /home/lh001/pipits/refdb/uchime_reference_dataset_28.06.2017/uchime_reference_dataset_28.06.2017.fasta

So, it is different each other..

DH1983 commented 6 years ago

For the second one, The directory is located in /home/lh001/... but the first one, /HOME/pipits/...

hsgweon commented 6 years ago

How crazy... Try this? (going around in circles now...)

vsearch --uchime_ref out_process/intermediate/input_nr_otus.fasta --db $PIPITS_UNITE_REFERENCE_DATA_CHIMERA --nonchimeras out_process/intermediate/input_nr_otus_nonchimeras.fasta --threads 1
DH1983 commented 6 years ago

In this time,

Unable to open file for reading ($HOME/pipits/refdb/uchime_reference_dataset_28.06.2017/uchime_reference_dataset_28.06.2017.fasta)

hsgweon commented 6 years ago

Can you try:

conda install vsearch=2.8.0 -y

and then try:

vsearch --uchime_ref out_process/intermediate/input_nr_otus.fasta --db $PIPITS_UNITE_REFERENCE_DATA_CHIMERA --nonchimeras out_process/intermediate/input_nr_otus_nonchimeras.fasta --threads 1
DH1983 commented 6 years ago

Hi, Hyun,

I try it with 'conda install vsearch=2.8.0 -y', and the message is # All requested packages already installed.

I have read one of the posts, and for the bug from vsearch, if I remember correctly, I downgraded as followed by your instructions.

And I still got the error message after I ran it...

Unable to open file for reading ($HOME/pipits/refdb/uchime_reference_dataset_28.06.2017/uchime_reference_dataset_28.06.2017.fasta)

hsgweon commented 6 years ago

Probably some really trivial thing that we are missing here...

Can you (in the same terminal):

export $PIPITS_UNITE_REFERENCE_DATA_CHIMERA=/home/lh001/pipits/refdb/uchime_reference_dataset_28.06.2017/uchime_reference_dataset_28.06.2017.fasta

And then try:

vsearch --uchime_ref out_process/intermediate/input_nr_otus.fasta --db $PIPITS_UNITE_REFERENCE_DATA_CHIMERA --nonchimeras out_process/intermediate/input_nr_otus_nonchimeras.fasta --threads 1

And, please re-check that /home/lh001/pipits/refdb/uchime_reference_dataset_28.06.2017/uchime_reference_dataset_28.06.2017.fasta really exists, by seeing its content:

less  /home/lh001/pipits/refdb/uchime_reference_dataset_28.06.2017/uchime_reference_dataset_28.06.2017.fasta
DH1983 commented 6 years ago

If I put 'export ...' I got this error message.. (pipits_env) lh001@lh001-300E4C-300E5C-300E7C:~/Desktop/PIPITS_Trial$ export $PIPITS_UNITE_REFERENCE_DATA_CHIMERA=/home/lh001/pipits/refdb/uchime_reference_dataset_28.06.2017/uchime_reference_dataset_28.06.2017.fasta bash: export: `$HOME/pipits/refdb/uchime_reference_dataset_28.06.2017/uchime_reference_dataset_28.06.2017.fasta=/home/lh001/pipits/refdb/uchime_reference_dataset_28.06.2017/uchime_reference_dataset_28.06.2017.fasta': not a valid identifier

but If I try to find out the data really exists by putting less /home/lh001/pipits/refdb/uchime_reference_dataset_28.06.2017/uchime_reference_dataset_28.06.2017.fasta

it DOES...

hsgweon commented 6 years ago

Ooops my bad. It should be:

export PIPITS_UNITE_REFERENCE_DATA_CHIMERA=/home/lh001/pipits/refdb/uchime_reference_dataset_28.06.2017/uchime_reference_dataset_28.06.2017.fasta
DH1983 commented 6 years ago

In this time, it has all run it through.

And the last block that I see is below. _dataset_28.06.2017.fasta 100% 16786547 nt in 30555 seqs, min 146, max 2570, avg 549 Masking 100%
Counting k-mers 100%
Creating k-mer index 100%
Detecting chimeras 100%
Found 0 (0.0%) chimeras, 6 (100.0%) non-chimeras, and 0 (0.0%) borderline sequences in 6 unique sequences. Taking abundance information into account, this corresponds to 0 (0.0%) chimeras, 37 (100.0%) non-chimeras, and 0 (0.0%) borderline sequences in 37 total sequences.

hsgweon commented 6 years ago

Great! how about the whole process? pipits_process? if it works, then might be useful to put corrected PATH into .bashrc as per instruction.

export PIPITS_UNITE_RETRAINED_DIR=/home/lh001/pipits/refdb/UNITE_retrained
export PIPITS_UNITE_REFERENCE_DATA_CHIMERA=/home/lh001/pipits/refdb/uchime_reference_dataset_28.06.2017/uchime_reference_dataset_28.06.2017.fasta
export PIPITS_WARCUP_RETRAINED_DIR=/home/lh001/pipits/refdb/warcup_retrained_V2
DH1983 commented 6 years ago

Thank you very much, Hyun. It is now running completely. I am really appreciated for your time. Best regards, DongHyeon.

hsgweon commented 6 years ago

I don’t exactly know why you are seeing a such behaviour, but hope this solution will do for your data (well for now anyway). Good luck with the rest of your analysis and do let me know if you have any other issues.

From: DH1983 notifications@github.com Sent: Monday, September 10, 2018 12:35:00 PM To: hsgweon/pipits Cc: H. Soon Gweon; Comment Subject: Re: [hsgweon/pipits] Stucked in the stage 'PIPITS_PROCESS' (#18)

Thank you very much, Hyun. It is now running completely. I am really appreciated for your time. Best regards, DongHyeon.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/hsgweon/pipits/issues/18#issuecomment-419882467, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJtTUh58ceZn1FB4zt3_2ZKThTYXCMN1ks5uZk5kgaJpZM4Wg9Uz.

DH1983 commented 6 years ago

Thank you for your note again, I will now try with my data. Hope you have a great rest of the day. Best, DongHyeon