A flexible, reproducible and resource efficient metabarcoding amplicon pipeline for HPC
DeltaMP is a command line tool for high performance computer taking advantage of queueing systems to parallelize and standardize bioinformatic processing of metabarcoding raw read libraries.
DeltaMP is initially developed to process 16S, 18S, 28S, ITS, COI and rbcl raw read libraries with the most up-to-date bioinformatic workflows, but can also handle any other barcoding targets (e.g. 23/28S, rbcL).
DeltaMP intend to be accessible for non-bioinformatician users with its fully tunable workflows based on a TAB-separated configuration file.
DeltaMP integrate a checkpointing feature which enable easy and efficient comparisons between different workflows applied on the same set of read libraries.
Last but not least, DeltaMP produces version controlled, reproducible and fully documented OTU tables in TAB-separated and BIOM formats readily usable for downstream taxonomic and OTU diversity analyses.
Source code of DeltaMP version 0.6 is available at https://github.com/lentendu/DeltaMP/releases/tag/v0.6
The repository could also be cloned using for example:
git clone https://github.com/lentendu/DeltaMP.gitThen install following the installation instructions.
DeltaMP is intend to be used on a HPC with a job scheduler (i.e. batch-queuing system).
Grid Engine
All this dependencies need to be available through the $PATH environmental variable or need to be loaded by the DeltaMP module file.
Usages could be accessed via the command line:
deltamp -hNAME
DeltaMP version 0.6 - a flexible, reproducible and resource efficient metabarcoding amplicon pipeline for HPC
SYNOPSIS
Usage: deltamp [-a account_or_project_name] [-cdfhnqtx] [-m max_running_tasks] [-p reference_subproject] [-r step] configuration_file
DESCRIPTION
-h display this help and exit
-a ACCOUNT
account or project name for the job queuing system
-c check dry run (-d option) by preserving the SUBPROJECT and PROJECT directories in the output directory
-d dry run avoid any submission to the queuing system and only output submission informations
-f display default values for optional configuration parameters
-n avoid checkpointing (i.e. searching for previous SUBPROJECT) and run the pipeline from the beginning
-m max_running_tasks
fix a maximum number of concurrently running tasks per array jobs, default is 400
-p SUBRPOJECT
only execute additional jobs or with different input variables compared to the reference SUBPROJECT. This will only work if both configuration files have the same input libraries and
output directory.
-q proceed until quality step only
-r STEP
restart pipeline computation from STEP. Replace STEP by 'list' to list all available steps of the subproject associated with the provided configuration file.
-t proceed until demultiplexing step only
-x delete the subproject associated with the provided configuration file
AUTHOR
Guillaume Lentendu and Tesfaye Wubet
REPORTING BUGS
Submit suggestions and bug-reports at <https://github.com/lentendu/DeltaMP/issues>, send a pull request on <https://github.com/lentendu/DeltaMP>, or compose an e-mail to Guillaume Lentendu <guilaume.lentendu@unine.ch>.
COPYRIGHT
Copyright (C) 2018 Guillaume Lentendu and Tesfaye Wubet
This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of
the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.To run, DeltaMP always required a configuration file and, optionaly, local copies of raw sequence libraries.
The configuration is encoded in UTF8 and the fields are TAB-separated.
If the file is edited under Windows, pay attention to use Unix compliant end of line (\n).
The following sections describe the parameters to provide in the configuration file.
Only the project name parameter and a sample list (see SAMPLE section) are compulsory.
A parameter is recognize by its exact definition in the first column of the configuration file.
If an optional parameter is let empty or if its definition is not found in the first column, its default value will be used instead.
The default values for the optional parameters could be displayed by using the -f option (see the usage instructions).
Project name: name of your project. This can be shared among multiple datasets and targets.
User name: full name of the person in charge of the analysis. The default is $USER.
Path to output location: full path to the output location. All final outputs will be copied into sub-directories of that directory. The default is /home/$USER .
Path to execution location: full path of the working directory. All computations will take place in that directory. The default is /work/$USER .
Sequencing technology: “Illumina” or “454”. The default is Illumina.
Directory path to archives or libraries OR BioProject accession: full path of the directory where to find the sequence libraries or archives. If the raw sequences are publicly available at the European Nucleotide Archive (https://www.ebi.ac.uk/ena), only provide the BioProject accession number. The libraries will then automatically be retrieved from the provided URLs in the BARCODE section. The default is /home/$USER.
Tab separated list of archives: filename of archives containing raw libraries, one per column. The default is “no” for no archive.
Libraries already demultiplexed: “yes” or “no”. If the libraries are not demultiplexed, one fastq/sff file per sample in each library will be produced. An archive containing all demultiplexed libraries and labeled with "raw_fastq"/“raw_sff” will be created in the output directory. The default is “yes” for libraries already demultiplexed.
Strict barcode anchoring: “yes” or “no”. Wether the barcodes are strictly anchored at the 5'-end or partial match are also allowed (non-internal barcodes) in raw undemultiplexed libraries. The default is "yes".
Single barcode side: Illumina specific parameter, "both", "forward" or "reverse". For single-indexing (one barcode per library), provide the position of the barcode, "both" for both forward and reverse primers, "forward" for barcode only associated with the forward primer, "reverse" for barcode only associated with the reverse primer. The default is “both”.
Bind barcode to primer for demultiplexing: Illumina specific parameter, "yes" or "no". For dual-indexing, control binding of barcodes with the forward/reverse primer to increase demultiplexing accuracy. The first barcode in a pair is only search associated with the forward primer at 5'-end and the second barcode is only search associated with the reverse primer at 5'-end, in both paired libraries. The default is “no”.
Orientation threshold: Illumina specific parameter, a percentage of raw demultiplexed reads above which to keep using reads in a certain orientation (forward primer in R1 and reverse primer in R2, and/or reverse primer in R1 and forward primer in R2). The default is 1.
Target organisms: Latin name of the targeted group as labelled in the reference sequence database (eg. Fungi, Glomeromycota, Bacteria, Archea, Eukaryota), or protists (Eukaryota excluding Fungi, Metazoa and Streptophyta). The OTUs assigned to the target group will be extracted from the main OTU table and copied into an additional OTU table labeled with the provided name (see Outputs). The default is “protist”.
Target region: any name is accepted here, but specific workflows will be allowed with “16S”, “18S”, "28S", “ITS”, “COI” or "rbcl" respectively for prokaryotes, eukaryotes (18S and 28S), Fungi, Metazoa or plants workflows. The default is “18S”.
Forward primer name: the biological forward primer. The 5'-3' orientation of the forward primer must be the same than the reference database sequence orientation. The default is “TAReuk454FWD1”.
Forward primer sequence (5' to 3'): the sequence of the biological forward primer. The default is “CCAGCASCYGCGGTAATTCC”.
Reverse primer name: the biological reverse primer. The default is “TAReukREV3”.
Reverse primer sequence (5' to 3'): the sequence of the biological reverse primer. The default is “ACTTTCGTTCTTGATYRA”.
Sequencing direction: 454 specific parameter, “forward”, “reverse” or "both" for libraries (column 3 of SAMPLES section) predominently orientated in the forward or reverse primer direction or for bi-directional sequencing, respectively. The default is “forward”.
Remove primers at reads ends: "no", "3prime", "5prime" or "both". Using cutadapt, "3prime" remove non-anchored reverse-complemented reverse and forward primers from 3'-end of forward and reverse libraries, respectively; "5prime" remove non-anchored forward and reverse primers from 5'-end of forward and reverse libraries, respectively; "both" combined both previous removal using the linked adapter strategy with anchored 5' adapter; value "no" avoid primer clipping. For "3prime", "5prime" and "both" values, unmatched reads are checked against the opposite primer/linked-adapter to allow for biderectionnal sequencing strategy detection, and the reads matched in this second step are reverse-complemented and append to the reads match at the first step. The default is "5prime".
Anchored primers: “yes” or “no”. Wether the primers are anchored to the 5'-end or not in demultiplexed libraries. The default is "yes".
all Illumina-specific parameters
Pair-end algorithm: "vsearch", "ngmerge", "simple_bayesian", "ea_util", "flash", "pear", "rdp_mle", "stitch" or "uparse" . Either using the vsearch "--fastq_mergepairs" option, the NGmerge tool or any algorithm of PandaSeq. See the PandaSeq manual (or man pandaseq) for algorithm descriptions. The default is “simple_bayesian”.
Pair-end similarity threshold: a number between 0 and 1 setting a similarity threshold ratio on the alligned region from which a pair is conserved. This value is provided to the -t option of pandaseq. This value is substract to 1 and then multiply by 100 to be provided to the --fastq_maxdiffpct option of vsearch. The default is 0.6 (i.e. vsearch default is 40).
Minimum overlap: a number between 0 and ∞ provided to the --fastq_minovlen option of vsearch or the -o option of pandaseq. The default is 10.
Keep unmerged: “yes” or “no”. Wether to keep high-quality reads which cannot be pair-end assembled alongside with paired reads for further processing or not. A padding sequence of "NNNNN" is added between the unpaired reads. The default is "no".
Avoid pair-end: “yes” or “no”. In case of low quality reads or too long target fragment, the pair-end assembly procedure is skipped and paired sequences are simply joined with a padding sequence of "NNNNN". Use this option as a last resort in case all other parameter tunning did not provide suitable results. The default is "no".
Denoising: 454 specific parameter, "yes" or "no", whether to perform 454 flowgram denoising using FlowClus or nothing. The default is "no".
Minimum number of flows: 454 specific parameter, a number between 100 and 600, minimum length of flowgrams to be kept for denoising. The default is 360.
Number of mismatches allowed on the barcode sequence: a number between 0 and 2, or 'a' for automatic detection of maximum allowed mismatch on a barcode to avoid mislabeling of sequence inside one library. This value is used for demultiplexing only. The default is 1.
Number of mismatches allowed on the primer sequence: a number between 0 and 10. For Illumina libraries, this number will only be used to calculate allowed dissimilarity (number of mismatch / primer length) for primer detection and removal by cutadapt. The default is 6.
Maximum number of ambiguities allowed in the sequence: a number between 0 and ∞. The default is 0.
Maximum homopolymer length allowed: a number between 0 and 20. The default is 10.
Minimum sequence length: a number between 50 and the set Maximum sequence length. This parameter is used for filtering pair-end assembled reads in the case of Illumina libraries. The default is 50.
Maximum sequence length: a number between the set Minimum sequence length and 1000. This parameter is used for filtering pair-end assembled reads only in the case of Illumina libraries. The differrence between the Maximum sequence length and the Minimum sequence length added to the Minimum overlap defines the maximum overlap length allowed for PandaSeq based pair-end assembly. The default is 600.
Type of quality filtering: "average" or "maxee". Filter sequences either based on the average quality or based on the maximum expected error over the sequence, after any length truncation. Filtering based on maximum expected error is only possible for Illumina reads. The default is "average".
Truncation before pair-end: Illumina specific parameter, "yes" or "no", only for available for _Type of quality filtering__ set to "maxee". Perform length truncation on unpairred reads using a maximum expected error, and eventually a tuncation length in order to improve pair-end assembly efficiency. The default is "no".
Minimum average quality on the trimmed sequence length: a number between 20 and 30 (Phred score). Only consider when the parameter Type of quality filtering is set to "average". The read quality average is calculated after optimal sequence length trimming. The default is 20.
Maximum expected error on the trimmed length: Illumina specific parameter, a number between 0.5 and ∞, by step of 0.5. Only consider when the parameter Type of quality filtering is set to "maxee". The default is 2.
Minimum value for the maximum expected error on the trimmed length: Illumina specific parameter, a number between 0.5 and ∞, by step of 0.5. Only consider when the parameter Type of quality filtering is set to "maxee". Only use for trimming length and quality optimization The default is 0.5.
Increment for maximum expected error optimization: Illumina specific parameter, a number between 0.1 and 1, which is a value by which the Maximum expected error on the trimmed length is incremented for the optimization procedure. The default is 0.5.
Minimum length truncation of unpaired reads: Illumina specific parameter, "no" or a number between Minimum sequence length / 2 and ( Maximum sequence length + Minimum overlap ) / 2. When a length is provided, a length truncation is performed before pair-end join, using the length maximizing the length of the read while minimizing the maximum expected error on the truncated length, which allows to keep the read counts fixed by the Minimum number of trimmed reads per sample parameter. The truncation length and the maximum expected error are optimized for all combinations of libraries (R1 and R2) and directions (forward and reverse primers) separately, but are fixed for all reads in a combination. To allow for read truncation only based on maximum expected error (thus producing reads of variable sizes), set to "no". The default is "no".
Maximum length truncation of unpaired reads: Illumina specific parameter, "no" or a read length Minimum sequence length and Maximum sequence length , and higher than Minimum length truncation of unpaired reads + Increment for length truncation optimization .
Increment for length truncation optimization: Illumina specific parameter, a number between 1 and 10, which is the number of nucleotide by which the Minimum length truncation of unpaired reads is incremented for the optimization procedure. The default is 5.
Expected mean length of amplified barcode gene: Illumina specific parameter, "no" or a read length between Minimum sequence length and Maximum sequence length use to optimize truncation length and maxEE parameters, so that assembled pair-end reads reach at least this length
Reuse previous subproject optimized quality parameters: Illumina specific parameter, "no" or a path to a previous suproject execution directory. This will reuse the same optimize quality parameters for length and quality fitering truncation if the primers and the options Type of quality filtering, Truncation before pair-end and Minimum length truncation of unpaired reads are identical. The default is no.
Minimum number of trimmed reads per sample: a number between 0 and ∞. If the minimum of raw, raw with primers or trimmed reads is not reached in any of the samples, the pipeline will stop after the quality step and report these reads counts in the quality step .out logfile. If the value is between 0 and 1, no threshold is apply on raw reads and the value is used as ratio of reads in the previous filtration step for the tresholding of raw with primers, pair-end and trimmed reads (e.g. read count of trimmed reads have to be > to the provided ratio times read count of pair-end reads). The default is 2000.
Minimum number of pair-end reads per sample: Illumina specific parameter, a number between 0 and ∞. If the minimum number of pair-end reads is not reached in any of the samples, the pipeline will stop after the quality step and report these reads counts in the quality step .out logfile. If the value is between 0 and 1, no threshold is apply on raw reads and the value is used as ratio of reads in the previous filtration step for the tresholding. The default is 0.8.
Samples percentage allowed below the read count threshold: percentage of samples which are allowed to have their read count below the threshold set by the Minimum number of trimmed reads per sample parameter. These samples below the threshold are not considered for quality, maxee and length truncation optimization. A value of 100 percent disable thresholding and quality parameter optimizations and use the minimum (average quality, length truncation) or maximum (expected error) values provided for quality filtering. The default is 0.
ITSx region to extract: "no", “ITS1” or “ITS2” to skip or to extract either regions from fungal ITS reads using ITSx. The default is “no”.
Pre-clustering: "no", "cdhit454" or "mothur" to skip or to use cd-hit-454 or mothur pre.cluster algorithms to pre-cluster reads after trimming and before chimera removal. For the mothur based pre-clustering, and aligned version of a reference database is required (see below at Database prefix name). The default is "no".
Chimera removal: "before", "after" or "both", to check for chimera before OTU clustering only (in each sample separetedly), after OTU clustering only (among OTU representative sequences), or at both moments. De-novo chimera are detected with UCHIME and removed. For dada2 clustering, bimera are always removed using removeBimera. The default is "after".
Remove bimera after ASV calling and joining: Illumina specific parameter, "no" or "yes" to remove bimera when clustering with dada2. Independent from the Chimera removal parameter. The default is "no".
Subsampling: “yes” or “no”. Activating subsampling will randomly select the same number of reads in all samples according to the read count in the sample with the lowest number of reads, after the trimming step. The default is “no”.
Clustering algorithm: "cd-hit-est”, “vsearch”, “mcl”, “sumaclust”, “swarm” or “dada2“. The default is “vsearch”.
Clustering similarity threshold: a number between 80 and 100 (percent of similarity). Disregarded if the “swarm” or "dada2" clustering algorithms are used. The default is 97.
Single run: "yes" or "no", consider all libraries coming from the same run in order to build a single error model for dada2. The default is "no".
Cluster with previous subproject reference sequences: the accepted values are a full path or “no”. If the full path to a previous subproject output directory is provided, all amplicons of this previous subproject will be search for exact match with amplicons from the current subproject. If amplicons of one or multiple OTUs from the current subproject have match in a single OTUs of the previous subproject, these amplicons will be assign to the OTU index name used in the previous subproject. Remaining OTUs from the current subproject without match in the reference suproject have their OTU index names starting after the last OTU of the reference subproject. This option is experimental, use with caution. The default is “no”.
Cluster with previous subproject percent overlap: a percentage of length overlap between current and previous subproject amplicons. This is usefull to merge OTUs of two subproject when different primers were used to amplify the same region. The default is 95.
Minimum abundance of previous subproject amplicons to cluster with: the minimum abundance of any dereplicated and non-chimeric amplicon in the previous and the current subprojects to be included in the clustering with previous subproject approach. The default is 4.
Remove singletons before chimera re-check: the accepted values are “yes” or “no”. This option allow to remove singleton OTUs from sub-sequent analyses in order to reduce computation time and memory footprints, as high number of singletons could have critical effects on grid job with limited resources for both following chimera re-check and taxonomic identification steps. The default is “no”.
Taxonomic classifier: "bayesian" or "vsearch", to use the RDP naive bayesian classifier (Wang et al., 2007) as implemented in mothur or the usearch global alignment method (Edgard, 2010) as implemented in vsearch. The default is "bayesian".
Minimum identity: a rate between 0.0 and 1.0 defining the minimum pairwise identity between a query sequence and a reference sequence to assign taxonomy, only when using "vsearch" in Taxonomic classifier. The default is 0.8.
Directory path to database: the full path to the directory containing reference sequence database files. The default is /home/$USER .
Database prefix name: filename prefix identifying database files. A database have to be composed from a TAB-separated "prefix".txt file in which the second column describe the version, citation and reference of the database with the first column containing VERSION, CITATION and FULLCITATION, respectively (see e.g. <test/pr2_4.10.0.txt>) and either from a pair of "prefix".fasta and "prefix".taxonomy mothur's style database format for "bayesian" classifier, or from a "prefix".udb binary file produced by the command vsearch --makeudb_usearch for "vsearch" classifier. Optionnaly, a "prefix".align.fasta, containing an aligned version of the "prefix".fasta sequences, is needed for pre-clustering with "mothur". See the
Reduce database to amplified fragment: "yes" or "no", to cut the database reference sequences using either the provided primers or ITSx with the provided 'ITSx region to extract' or do nothing, respectively. If set to "yes", this will use the cut database for taxonomic assignment, while if set to "no" this will use the full database sequences for taxonomic assignment. For SILVA and UNITE databases, if a unique cut read is produced from multiple accessions with different taxonomic path, the cut read will be annotated with the least common ancestor. For PR2 database, cut reads are only dereplicated for each taxonomic path separately. If set to "yes" and the cut database is already present in the "Directory path to database", the database cutting is skipped. The default is "no".
Consensus assignment threshold: a number between 50 and 100. Consensus threshold percent to assign a taxonomic rank among the matches. For "bayesian" taxonomic calssifier the matches are the 100 bootstrap matches, for "vsearch" the matches are the best match(es) plus the match(es) in a similarity range of 10% below the best match(es) dissimilarity. The default is 60.
Assign all reads: the accepted values are “yes” or “no”. Activating this option will assign all dereplicated or pre-clustered reads to a taxonomy. A consensus assignment is then determined for each OTU at a threshold of 60 %. If set to “no”, only the most abundant read per OTU will be assigned to taxonomy. The default is “no”.
Full path to additional database prefix name: the full path to the filename prefix of additional reference sequences to add to the reference database (useful to integrate yet unpublished sequences). The fasta and taxonomy formats need to match with the main database. dub format for vsearch will be generated automatically if needed. The default is "no".
Assign putative function: "yes" or "no" to assign putative function to Fungi using the FUNGuild database (Nguyen et al., 2016).
Minimum number of sample for abundant OTUs: a number between 0 and ∞ (see Outputs). The default is 1.
Minimum number of reads for abundant OTUs: a number between 0 and ∞ (see Outputs). The default is 4.
454 libraries: the columns 1 to 3 have to be filled with barcode sequences, sample names and library filenames or URLs, respectively. If the same barcode was used to sequence in both forward and reverse directions, the library containing the reverse primer at its 5'-end have to be provided in the fourth column.
Illumina libraries: the columns 1 to 4 have to be filled with barcodes, sample names, the filename or URL of the forward libraries and the filename or URL of the reverse libraries, respectively. The column 1 can remain empty when libraries are already delmultiplexed. Dual index demultiplexing is turn on when a comma separated pair of barcodes is provided for every sample in column 1.
Columns 3 and 4 accept libraies in fastq or sff format with most kinds of compression (.gz, .tar, .tar.gz, .tgz, .bz2, .tar.bz2, .tbz2, .zip).
For ENA libraries, the column 2 have to match ENA "Submitter's sample name" field and column 3 to 4 have to match ENA full ftp or fasp (for aspera connect download) URLs of a run accession as listed in ENA fields “Submitted files (FTP)” or "FASTQ files (FTP)" or “Submitted files (Aspera)” or "FASTQ files (Aspera)".
Example configuration files 'configuration_xxx.tsv' are available in the test/ directory after installation with make.
To execute the full pipeline, run the following command in a terminal:
deltamp [path/]configuration_filereplacing [path/] by the path to your configuration file, and “configuration_file” by its filename. Avoid “path/” if you are already in the right directory.
For each DeltaMP execution, a "PROJECT” directory labeled with the project name will be created into the "Path to output location" (nothing done if this directory already exist). A “SUBPROJECT” directory will be created inside the PROJECT directory and will be labeled as follow: “Project name”_”Sequencing technology”_”Target region”_”unique identifier”. The unique identifier (cksum of date, time and configuration file) enables differentiation between each instance of the pipeline execution for the same project.
The SUBPROJECT directory is copied into the "Path to execution location" and all necessary jobs are submited to the queue.
Once all step's jobs are completed, the outputs are copied back into the "Path to output location"/PROJECT/SUBPROJECT directory.
output SUBPROJECT directory:
execution SUBPROJECT directory:
Begin with a dry run to check for correct configuration
deltamp -d [path/]configuration_filestart analyses with a "quality_only" workflow in order to first control read counts and quality:
deltamp -q [path/]configuration_fileThe output statistics will be then useful to check the proper handling of raw reads until trimming step (e.g. demultiplexing of 454 libraries or pair-end assembly of Illumina libraries), before starting more computational intensive OTU clustering and taxonomic identification steps.
test DeltaMP behaviors for checkpointing with multiple dry run on a continuously modified configuration files, avoiding the cleaning of the dry run directories:
deltamp -cd [path/]configuration_fileTo run the pipeline normally after such a dry checkpointing, first delete all directories created for dry runs from the output directory, or change the output directory in the configuration file.
In order to avoid to run the pipeline from the beginning if only one parameter is modified in the configuration file (e.g. clustering algorithm), DeltaMP allow re-use of previously produced files in the execution directory by checkpointing.
Checkpointing is turn on by default, which means that if any SUBPROJECT with the same target, same sequencing technology and the same list of samples is found in the "Path to output location/PROJECT" directory, all configuration parameters and option values will be compared.
Then come different situations:
if a previous SUBPROJECT have the same set of configuration parameters and option values than for the current deltamp call, and error message is issued and nothing appends to avoid any useless re-calculation. If the users really want to re-compute, he/she will have to first delete the previous SUBPROJECT, to change the "Path to output location" or to disable checkpointing with option -n.
if a previous SUBPROJECT have the same set of configuration parameters and option values than for the current deltamp call, but the SAMPLES section differs, and error message is issued and nothing appends to avoid checkpointing for different sets of samples/libraries. If the users really want to re-compute, he/she will have to first delete the previous SUBPROJECT, to change the "Path to output location" or to disable checkpointing with option -n.
if at least one configuration parameter or option value differs with any previous SUBPROJECT, a new SUBPROJECT will be created and the computation will start directly after the last common step, while all files and directories produced during the common steps of the previous SUBPROJECT are symlinked to the execution directory or hard copied to the output directory of the new SUBPROJECT. If multiple previous SUBPROJECT exist, the previous SUBPROJECT is the one with the highest amount of common steps (then the oldest if still multiple previous SUBPROJECT).
Relationship tree among successive checkpointed SUBPROJECTs are documented in the output directory of SUBPROJECTs with at least one previous SUBPROJECT at config/tree.summary.
Checkpointing could be repeated multiple times, so long at least one parameter differ with any of the previous SUBPROJECTs.
The previous SUBPROJECT could be hard set with the -p option, otherwise the previous SUBPROJECT with highest number of common steps will be taken as reference.
Checkpointing can be turn off by using the -n option.
Running the DeltaMP command will generate the directories and configuration files necessary to conduct the pipeline analysis as well as submitting the required steps to the queueing system.
The following steps are bash scripts available in the bin directory after build and are named following the scheme “xxx.sh”, xxx being the name of the step.
For 454 or Illumina specific steps, step script filenames follow “454_xxx.sh” and “Illumina_xxx.sh” schemes, respectively.
Each step job is waiting in the queue so long its preceeding step job is not completed, except for the cut_db step which do not have to wait on any job completion. For checkpointing, the init step job will wait on the completion of the last common step job from the previous SUBPROJECT.
SUBPROJECT.documentation.txt: The file describes the whole processing of the libraries through the pipeline in a human readable format. This file is not archived and will be directly outputted in the SUBPROJECT output directory.
The TAB-separated OTU matrices each contains a dense matrix of read counts per OTUs in each sample, each row corresponding to an OTU and each column to a sample, the second to last column containing the consensus taxonomic assignment (labeled as “taxonomy”), and the last column listing the representative sequence identifiers (labeled as “repseq”). Taxonomic rank with no consensus assignment are labeled with “unidentified”.
Job execution could be visualized with the grid engine native tool qstat (grid Engine) or squeue (SLURM).
If a job is waiting in error state or if it dependencies is never satisfied, it means that the job or its preceding job exit due to an error detected inside of the job.
If it concerns the quality step, the most common error is a number of reads below the set limit.
For any other jobs, it means that the previous step unexpectedly terminate before its end or that an error was issued by mothur in the previous step.
In all those cases, check the standard output (.out) and standard error (.err) log files of the respective step(s) and array(s), which are situated in the "Path to execution location"/SUBPROJECT/log directory.
To detect job terminated due to overpassing requested memory and/or time, compare the requested memory/time in the problematic step's script with the maximum used memory/runtime during job execution. To print job's record after execution, use qacct (grid Engine) or sacct (SLURM).
For unsolved issues, send an email to guillaume.lentendu@unine.ch, including the .out and .err log files as well as the output of the qacct command for the problematic job.