Closed sekhwal closed 4 years ago
Hi, after following the installation steps, you should set PIPELINE_DIR and MINICONDA_DIR to the paths corresponding to MetONTIIME dir and miniconda3 dir. For example, in my installation I set the following values:
#PIPELINE DIR
PIPELINE_DIR <- "/home/simone/MinION/MinION_scripts/MetONTIIME"
#MINICONDA DIR
MINICONDA_DIR <- "/home/simone/miniconda3"
In the config file, these are not the only 2 values you have to set. However, based on the error you reported in the QIIME2 forum, it looks like you have not run the installation step yet. Can you please confirm this? Let me know if you have any issues running the installation steps. Simone
Hi,
I have done installation steps. Please see my configuration in the file "config_MinION_mobile_lab". However, still it is showing same error, I am suspect if it for guppy?
######################################################################################################## PIPELINE_DIR <- "/home/kumarm/MinION-data/MetONTIIME"
MINICONDA_DIR <- "/home/kumarm/miniconda3"
BASECALLER_DIR <- "/home/kumarm/MinION-data/ont-guppy-cpu/guppy-0.1.11/"
DB <- "/path/to/PRJNA33175_Bacterial_sequences.qza"
TAXONOMY <- "/path/to/PRJNA33175_taxonomy.qza"
SAMPLE_METADATA <- "/path/to/sample-metadata.tsv" ########################################################################################################
suppressMessages(library(Biostrings))
MetONTIIME <- paste0(PIPELINE_DIR, "/home/kumarm/MinION-data/MetONTIIME/MetONTIIME.sh")
subsample_fast5 <- paste0(PIPELINE_DIR, "/home/kumarm/MinION-data/MetONTIIME/subsample_fast5.sh") #########################################################################################################
SEQTK <- paste0(MINICONDA_DIR, "/home/kumarm/miniconda3/envs/MetONTIIME_env/bin/seqtk")
PORECHOP <- paste0(MINICONDA_DIR, "/home/kumarm/miniconda3/envs/MetONTIIME_env/bin/porechop")
PYCOQC <- paste0(MINICONDA_DIR, "/home/kumarm/miniconda3/envs/MetONTIIME_env/bin/pycoQC")
NANOFILT <- paste0(MINICONDA_DIR, "/home/kumarm/miniconda3/envs/MetONTIIME_env/bin/NanoFilt")
Hi, the only variables you have to modify are in the
PIPELINE_DIR <- "/home/kumarm/MinION-data/MetONTIIME"
#MINICONDA DIR
MINICONDA_DIR <- "/home/kumarm/miniconda3"
#basecaller_dir
BASECALLER_DIR <- "/home/kumarm/MinION-data/ont-guppy-cpu/guppy-0.1.11/"
#NCBI-downloaded sequences (QIIME2 artifact)
DB <- "/path/to/PRJNA33175_Bacterial_sequences.qza"
#Taxonomy of NCBI-downloaded sequences (QIIME2 artifact)
TAXONOMY <- "/path/to/PRJNA33175_taxonomy.qza"
#sample-metadata file describing samples metadata; it is created automatically if it doesn't exist
SAMPLE_METADATA <- "/path/to/sample-metadata.tsv"
box (and in the above part of the script, for setting the desired options). I see you have edited also SEQTK, PORECHOP, PYCOQC and NANOFILT values, but those values should have been left unchanged. So, my advice is to copy the original value for all variables that are not in that box. Moreover, I notice that guppy v0.1.11 is a very old version, I think you need to download a newer one.
After you have done this, can you please tell me if the ./install.sh
instruction gave any errors?
I have improved my configuration in the file.."config_MinION_mobile_lab". However it is showing same error.....
######################################################################################################## PIPELINE_DIR <- "/home/kumarm/MinION-data/MetONTIIME"
MINICONDA_DIR <- "/home/kumarm/miniconda3/miniconda3"
BASECALLER_DIR <- "/home/kumarm/MinION-data/ont-guppy-cpu/guppy-0.1.11/"
DB <- "/path/to/PRJNA33175_Bacterial_sequences.qza"
TAXONOMY <- "/path/to/PRJNA33175_taxonomy.qza"
SAMPLE_METADATA <- "/path/to/sample-metadata.tsv" ########################################################################################################
suppressMessages(library(Biostrings))
MetONTIIME <- paste0(PIPELINE_DIR, "/MetONTIIME.sh")
subsample_fast5 <- paste0(PIPELINE_DIR, "/subsample_fast5.sh") #########################################################################################################
SEQTK <- paste0(MINICONDA_DIR, "/envs/MetONTIIME_env/bin/seqtk")
PORECHOP <- paste0(MINICONDA_DIR, "/envs/MetONTIIME_env/bin/porechop")
PYCOQC <- paste0(MINICONDA_DIR, "/envs/MetONTIIME_env/bin/pycoQC")
NANOFILT <- paste0(MINICONDA_DIR, "/envs/MetONTIIME_env/bin/NanoFilt")
Hi,
MINICONDA_DIR <- "/home/kumarm/miniconda3/miniconda3"
should probably be changed to
MINICONDA_DIR <- "/home/kumarm/miniconda3"
.
Then, as I told you, you should update Guppy to v3.4. Could you please report the error you had when running Import_database.sh?
Based on the error you posted on QIIME2 forum, it looks like the install.sh script was not run successfully yet, as there was error Could not find conda environment: entrez_qiime_env
.
Yes! This is the error one. Is it installation error? It occur while running Import_database.sh. Please see below error message..
(base) kumarm@microbiome:~/MinION-data/MetONTIIME$ ./Import_database.sh sequence.fasta
NotWritableError: The current user does not have write permissions to a required path. path: /anaconda3/envs/.conda_envs_dir_test uid: 1024 gid: 1023
If you feel that permissions on this path are set incorrectly, you can manually change them by executing
$ sudo chown 1024:1023 /anaconda3/envs/.conda_envs_dir_test
In general, it's not advisable to use 'sudo conda'.
Could not find conda environment: entrez_qiime_env
You can list all discoverable environments with conda info --envs
.
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output. File "./entrez_qiime/entrez_qiime.py", line 192 print "Need to supply either FASTA file or accession number list as input." # should never get here because either -i or -L is a required argument. ^ SyntaxError: Missing parentheses in call to 'print'. Did you mean print("Need to supply either FASTA file or accession number list as input." # should never g et here because either -i or -L is a required argument.)? Imported sequence.fasta as DNAFASTAFormat to /home/kumarm/MinION-data/MetONTIIME/sequence_sequence.qza Usage: qiime tools import [OPTIONS]
Import data to create a new QIIME 2 Artifact. See https://docs.qiime2.org/ for usage examples and details on the file types and associated semantic types that can be imported.
Options: --type TEXT The semantic type of the artifact that will be created upon importing. Use --show-importable-types to see what importable semantic types are available in the current deployment. [required] --input-path PATH Path to file or directory that should be imported. [required] --output-path ARTIFACT Path where output artifact should be written. [required] --input-format TEXT The format of the data to be imported. If not provided, data must be in the format expected by the semantic type provided via --type. --show-importable-types Show the semantic types that can be supplied to --type to import data into an artifact. --show-importable-formats Show formats that can be supplied to --input-format to import data into an artifact. --help Show this message and exit.
There was a problem with the command:
(1/1) Invalid value for "--input-path": Path "/home/kumarm/MinION- data/MetONTIIME/sequence_accession_taxonomy.txt" does not exist.
When I run following commands there are no any error generates..and entrez_qiime directory generate after running Import_database.sh sequence.fasta command, then it shows above that error..
git clone https://github.com/MaestSi/MetONTIIME.git cd MetONTIIME chmod 755 * ./install.sh
I have updated guppy3-3.0.9 but not sure where I should find v3.4. I think the error is not by guppy version..
Hi @sekhwal , this error is not generated by Guppy version for sure, I was just thinking about future configuration steps. It looks like the current error is caused by the impossibility to create a virtual environment named entrez_qiime_env
, which is based on python 2. The Syntax error you get is because the workflow is trying to run the code in another environment based on python 3. According to the errors, it looks like you also have anaconda installed, in addition to miniconda3. Is it the case? My guess is that anaconda/bin dir comes in the PATH environment variable before miniconda3/bin dir and they may be causing conflicts. To test this, could you please report the output of which conda
and of which activate
? Thanks
Hi, Here is the information... $ which conda /home/kumarm/miniconda3/bin/conda $ which activate /home/kumarm/miniconda3/bin/activate (base) kumarm@microbiome:~$
Ok, so I guess that if you typels -l /home/kumarm/miniconda3/envs
you get MetONTIIME_env but you don't get entrez_qiime_env, right?
Please, could you tell me what happens if you type conda create -n entrez_qiime_env python=2.7 gcc
? That should be the instruction in the Import_database.sh script that is causing issues.
Searching on the net, I found out that yours is a frequent conda error. As described here, the issue might be due to the fact that you installed miniconda3 using sudo. What owner do you get as a result of the ls command, your username or root? If the lattest is the case, a proposed solution would be to write:
sudo chown -R username /path/to/miniconda3
, of course replacing "username" with your username and the /path/to/miniconda3 with the corresponding path.
What still I don't understand is why you got the
NotWritableError: The current user does not have write permissions to a required path. path: /anaconda3/envs/.conda_envs_dir_test
error, that references anaconda3. Simone
Hi, I have tried following commands....please take a look..
~$ ls -l /home/kumarm/miniconda3/envs total 4 drwxrwxr-x 26 kumarm kumarm 4096 Jan 3 14:44 MetONTIIME_env ~$ conda create -n entrez_qiime_env python=2.7 gcc
NotWritableError: The current user does not have write permissions to a required path. path: /anaconda3/envs/.conda_envs_dir_test uid: 1024 gid: 1023
If you feel that permissions on this path are set incorrectly, you can manually change them by executing
$ sudo chown 1024:1023 /anaconda3/envs/.conda_envs_dir_test
In general, it's not advisable to use 'sudo conda'.
~$ sudo chown -R username /path/to/miniconda3 [sudo] password for kumarm: kumarm is not in the sudoers file. This incident will be reported.
Would you suggest uninstall miniconda3 and reinstall..?
Thank you for the information. What is not clear to me is how it has been possible to create one environment (MetONTIIME_env), but it looks like the location where new environments (e.g. entrez_qiime_env) are stored is somehow changed. What happens if you tell conda where to create the new environment?
Please try with:
conda create -p /home/kumarm/miniconda3/envs/entrez_qiime_env python=2.7 gcc
~$ conda create -p /home/kumarm/miniconda3/envs/entrez_qiime_env python=2.7 gcc Collecting package metadata (current_repodata.json): done Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source. Collecting package metadata (repodata.json): done Solving environment: failed
PackagesNotFoundError: The following packages are not available from current channels:
Current channels:
To search for alternate channels that may provide the conda package you're looking for, navigate to
https://anaconda.org
and use the search bar at the top of the page.
Apparently, gcc has been removed from conda-forge channel.
This should be solved adding anaconda channel with:
conda config --add channels anaconda
and then running:
conda create -p /home/kumarm/miniconda3/envs/entrez_qiime_env python=2.7 gcc
Hi, These commands work without any errors.. Now I am trying ./Import_database.sh sequence.fasta However it is showing following message.. ./Import_database.sh sequence.fasta WARNING: A conda environment already exists at '/home/kumarm/miniconda3/envs/entrez_qiime_env' Remove existing environment (y/[n])?
Please answer n
and tell me if it ends without any errors.
Hi, it works.......Thank you very much for your kind support...
Here is the process complete successfully without any error.......
$ ./Import_database.sh sequence.fasta WARNING: A conda environment already exists at '/home/kumarm/miniconda3/envs/entrez_qiime_env' Remove existing environment (y/[n])? n
CondaSystemExit: Exiting.
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support Collecting numpy Downloading https://files.pythonhosted.org/packages/3a/5f/47e578b3ae79e2624e205445ab77a1848acdaa2929a00eeef6b16eaaeb20/numpy-1.16.6-cp27-cp27mu-manylinux1_x86_64.whl (17.0MB) |████████████████████████████████| 17.0MB 1.2MB/s Installing collected packages: numpy Successfully installed numpy-1.16.6 DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support Processing /home/kumarm/.cache/pip/wheels/00/d6/78/4b98bf36fba9a1766ab94b453dedaef570bf01dbfcb02f585a/cogent-1.9-cp27-cp27mu-linux_x86_64.whl Installing collected packages: cogent Successfully installed cogent-1.9
Imported sequence.fasta as DNAFASTAFormat to /home/kumarm/MinION-data/MetONTIIME/sequence_sequence.qza Imported /home/kumarm/MinION-data/MetONTIIME/sequence_accession_taxonomy.txt as HeaderlessTSVTaxonomyFormat to /home/kumarm/MinION-data/MetONTIIME/sequence_taxonomy.qza $
Would you suggest to update python as it is showing in above message...?
Good! So, now you have those 2 QIIME2 artifacts, whose full path you have to set in the config file. And then you should be ready to go. No, I wouldn't suggest to update python as, python2 and python3 follow slightly different syntax rules, and it wouldn't be possible to run python2 code with python3 interpreter. The python2 version we are using, 2.7, is the latest available. The problem is that the entrez_qiime utility which we are using for importing the database, was written a few years ago using python2, and so is now a bit out of date, but I wouldn't be able to update it to python3, as I don't know that language. Python2 is the main reason why I had to create a separate virtual environment entrez_qiime_env, however, as long as it works, it should be ok. Let me know if you are able to start the full pipeline. Maybe you could try first with demo data I have uploaded, or subsampling a few fast5 files with appropriate option in configuration file. Simone
Sure! I appreciate...
Hi, I have RUN pipeline with fastq sequence files. Therefore, pipeline RUNNING with following message for long time. Please let me know how long usually it take to generate results?
$ ./Launch_MinION_mobile_lab.sh fastq_pass/ $ nohup: appending output to 'nohup.out'
Hi, you can check the step completed by the pipeline printing nohup.out file. The most time consuming step is the Blast taxonomy assignment, as it is single threaded. The time it takes to accomplish the job depends mainly on the size of the database you are using. With a quite big db, as the 16S bacterial database, classification should take about 5k reads per hour. Smaller dbs should speed up the analysis consistently. Simone
Mind the MetONTIIME pipeline is meant to run starting from fast5 files. If you want to start from already demultiplexed fastq.gz sequences, please have a look at the other open issue for proper command line instruction. I will shortly include that information in the README.
Found 0 fastq files.
0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----| Done in 0 ms. Demultiplexing finished at Thu Jan 9 10:13:50 2020
Error in file(con, "r") : cannot open the connection Calls: grep -> readLines -> file In addition: Warning message: In file(con, "r") : cannot open file 'NA': No such file or directory Execution halted
Do I need to provide fast5 files to RUN pipeline. I have these files as well.
I think I need to RUN pipeline with fast5 files.
Please, see my previous message.
Hi,
Here total number of reads in my files, please let me know which number should I take for -d
Number of reads assigned to BC01: 17 Number of reads assigned to BC02: 10 Number of reads assigned to BC03: 37 Number of reads assigned to BC04: 25 Number of reads assigned to BC05: 13 Number of reads assigned to BC06: 32 Number of reads assigned to BC07: 4 Number of reads assigned to BC08: 25999 Number of reads assigned to BC09: 23324 Number of reads assigned to BC10: 26465 Number of reads assigned to BC11: 27 Number of reads assigned to BC12: 61
Hi,
it looks like you only included samples in the pool with barcodes BC08, BC09 and BC10. Or, at least, these are the only samples for whom you got a good number of reads.
So, if you wanted to start the analysis from fast5 files you could have set:
BC_int <- c("BC08", "BC09", "BC10")
in the config file. However, you can now simply delete BC\<numbers>.fastq.gz files for the remaining 9 samples.
Do you confirm you got all the expected results in the analysis folder?
Your question about sampling depth seems to be related to the usage of Evaluate_diversity.sh script, which can only be run after previous analyses have completed successfully.
The Evaluate_diversity.sh script is kind of an experiment, as with Illumina reads you would use representative sequences, namely sequences obtained after performing clustering or error correction, for building a taxonomy tree and for evaluating diversity metrics. Since with Nanopore reads we can't perform any meaningful clustering due to high error rate, the script subsamples the same number of reads (-d
sampling depth) for each sample, considers these to be the representative sequences, and computes all the diversity metrics. I am not sure if this approach is good enough for evaluating diversity, I am open to suggestions. Maybe diversity metrics that also incorporate phylogenetic distance between the features, as UniFrac, could provide a meaningful representation. In my experience, this is a quite time consuming step, so I would recommend starting with few reads (e.g. 1,000) and then maybe increasing it, using as much threads as you have available, which you can specify with -t
parameter.
Simone
Yes, it seems pipeline generating all files in analysis folder...I RUN pipeline with fast5 files ./Launch_MinION_mobile_lab.sh fast5_pass/ command.
please let me know, is my analysis going correct. I need to add all samples in analysis. How I can improve the reads in other samples?
Hi,
it looks like your analysis completed successfully. You can monitor the stage of the analysis looking at nohup.out or logfile.txt files. Have a look at barplots following Results visualization section in the README. Unfortunately, if you already set
disable_porechop_demu_flag <- 1
which is the default value, there is no way to increase the number of reads assigned to other samples. For diagnosis, have a look at pycoQC report and histograms in qc folder. This way, you can see if reads were not assigned to those samples at all (pycoQC pie chart), or if they have been filtered out due to out of range length (read length histograms).
Simone
Finally I run the command "Evaluate_diversity.sh", but I did not perform delete BC
./Evaluate_diversity.sh -w /home/kumarm/MinION-data/MetONTIIME/ -m /home/kumarm/MinION-data/MetONTIIME/sequence-metadata.tsv -d 1000 -t 1
Also there is another step .."Starting analysis from fastq.gz files". Is the next step or another way to RUN pipeline with fastq.gz files?
Not deleting samples with less than "sampling depth" reads is in my opinion not a good idea, as you would be performing comparisons using different numbers of representative sequences, not due to different samples complexity but due to insufficient sequencing depth. taxa-bar-plots.qzv
and taxa-bar-plots-no-Unassigned.qzv
are the files you can visualize, and you already had them in /home/kumarm/MinION-data/MetONTIIME/fast5_pass_analysis
folder. Starting analysis from fastq.gz files is just another way to run the pipeline if you are halfway through.
Simone
Hi, I was trying to re-run the pipeline from starting. However, it is showing following error while running "./Import_database.sh sequence.fasta", please suggest..
./Import_database.sh: line 55: qiime: command not found ./Import_database.sh: line 61: qiime: command not found
Hi, as previously observed, it looks like your miniconda is having some issues in activating the proper environments, possibly due to conflicts with anaconda. Remember that there is no need in running again Import_database.sh, unless you changed the database. Before running the pipeline, you should also check that source activate MetONTIIME_env
is actually activating the environment; as a consequence of successfull activation, you should see the environment name in the prompt. If that is the case, run conda deactivate
to deactivate it before running the pipeline. Again, also have a look at ls -l /home/kumarm/miniconda3/envs
, to see if you have both environments available in the proper location.
Hi, Yes, entrez_qiime_env and MetONTIIME_env are properly installed..but still it showing error with ..../Import_database.sh sequence.fasta
drwxrwxr-x 13 kumarm kumarm 4096 Jan 11 21:50 entrez_qiime_env drwxrwxr-x 13 kumarm kumarm 4096 Jan 11 22:22 MetONTIIME_env
ERROR:
Requirement already satisfied: cogent in /home/kumarm/miniconda3/envs/entrez_qiime_env/lib/python2.7/site-packages (1.9) ./Import_database.sh: line 55: qiime: command not found ./Import_database.sh: line 61: qiime: command not found
At lines 55 and 61 the command qiime
is invoked, which should be available in the path if
source activate MetONTIIME_env
(line 47)
succeeded. Are you able to activate the environment, or is it failing?
It is showing..
$ source activate MetONTIIME_env
Could not find conda environment: MetONTIIME_env
You can list all discoverable environments with conda info --envs
.
$ which conda /usr/bin/anaconda/bin/conda kumarm@microbiome:~/MinION-data/MetONTIIME$ which activate /usr/bin/anaconda/bin/activate
That is the issue I was already thinking about a few days ago. Sometimes you have anaconda bin directory first in the PATH, while sometimes you have miniconda3 bin dir first instead.
To solve this, before running the pipeline, simply run in the shell:
export PATH=/home/kumarm/miniconda3/bin/:$PATH
If you prefer, you can write the instruction in the .bashrc file with the following command:
echo "export PATH=/home/kumarm/miniconda3/bin/:$PATH" >> /home/kumarm/.bashrc
In this way, every time you open a new shell, the .bashrc file is sourced and you should have miniconda3 bin directory before anaconda's.
This should solve all your conda related issues, let me know if this is the case!
Simone
There is no MetONTIIME_env directory so I am trying to run following command..but it is showing error..
conda create -p /home/kumarm/miniconda3/envs/MetONTIIME_env/ python=2.7 gcc
Collecting package metadata (repodata.json): failed
UnavailableInvalidChannel: The channel is not accessible or is invalid. channel name: MetONTIIME_env channel url: https://conda.anaconda.org/MetONTIIME_env error code: 404
You will need to adjust your conda configuration to proceed.
Use conda config --show channels
to view your configuration's current state,
and use conda config --show-sources
to view config file locations.
Why are you doing this? You told me you already had /home/kumarm/miniconda3/envs/MetONTIIME_env
and /home/kumarm/miniconda3/envs/entrez_qiime_env
directories.
If this is not the case anymore, then repeat installation after following my advice in the previous post.
Otherwise, simply run:
export PATH=/home/kumarm/miniconda3/bin/:$PATH
./Import_database.sh sequence.fasta
I am not sure, but I think these both env have been removed..I am trying to run following command but it is showing error. Is there any way to reinstall these both env or do I need to delete miniconda3 directory and re-install?
conda create -p /home/kumarm/miniconda3/envs/entrez_qiime_env python=2.7 gcc
conda create -p /home/kumarm/miniconda3/envs/MetONTIIME_env python=2.7 gcc
If those environments have been removed, simply repeat MetONTIIME installation following the README, namely git clone the repo and run the install script. Just remember to
echo "export PATH=/home/kumarm/miniconda3/bin/:$PATH" >> /home/kumarm/.bashrc
and then open a new shell where you can do all the installation.
You don't need to delete miniconda3.
Warning: 'bioconda' already in 'channels' list, moving to the top Warning: 'conda-forge' already in 'channels' list, moving to the top Warning: 'r' already in 'channels' list, moving to the top Warning: 'anaconda' already in 'channels' list, moving to the top
Hi, I am not sure which value I need to give here to configure pipeline..Please suggest...
A conda environment named MetONTIIME_env is created, where seqtk, porechop, pycoQC, NanoFilt and qiime2-2019.10 are installed. Then, you can open the config_MinION_mobile_lab.R file with a text editor and set the variables PIPELINE_DIR and MINICONDA_DIR to the value suggested by the installation step.