franciscozorrilla
commented
4 years ago Hi Joo-Young,
I believe that the problems you are experiencing are due to your misuse of the cut_up_fasta.py script:
$ cut_up_fasta.py -h
usage: cut_up_fasta.py [-h] [-c CHUNK_SIZE] [-o OVERLAP_SIZE] [-m]
[-b BEDFILE]
contigs [contigs ...]
Cut up fasta file in non-overlapping or overlapping parts of equal length.
Optionally creates a BED-file where the cutup contigs are specified in terms
of the original contigs. This can be used as input to concoct_coverage_table.py.
positional arguments:
contigs Fasta files with contigs
optional arguments:
-h, --help show this help message and exit
-c CHUNK_SIZE, --chunk_size CHUNK_SIZE
Chunk size
-o OVERLAP_SIZE, --overlap_size OVERLAP_SIZE
Overlap size
-m, --merge_last Concatenate final part to last contig
-b BEDFILE, --bedfile BEDFILE
BEDfile to be created with exact regions of the
original contigs corresponding to the newly created
contigsIn your code you wrote:
cut_up_fasta.py -c 10000 -m -b ./concoct_bins/SAMPLE_10k.bed > ./concoct_bins/SAMPLE_10k.fa ./input_megahit_contigs/SAMPLE.final.contigs.faHowever, your input contigs ./input_megahit_contigs/SAMPLE.final.contigs.fa are on the right hand side of the output redirect (>). In bash terms, this means that the output of what is generating by the left hand side will be stored in the file specified on the right hand side. The correct way to write this would be:
cut_up_fasta.py -c 10000 -o 0 -m ./input_megahit_contigs/SAMPLE.final.contigs.fa -b ./concoct_bins/SAMPLE_10k.bed > ./concoct_bins/SAMPLE_10k.fa Note that the above command should generate both the BED file (SAMPLE_10k.bed) and the cut up assembly (SAMPLE_10k.fa). Then generate the coverage table using the BED file + the sorted BAM file:
concoct_coverage_table.py contigs_10K.bed sorted.bam> coverage_table.tsvNow we can run CONCOCT:
concoct --coverage_file coverage_table.tsv \
--composition_file SAMPLE_10k.fa \
-b sample_ID Note that this approach assumes that you sorted bam file was generating by mapping your short reads against the ORIGINAL assembly, not the cut up assembly. Please also have a look at the CONCOCT help file to see additional useful parameters such as -c and -t:
$ concoct -h
cut_up usage: concoct [-h] [--coverage_file COVERAGE_FILE]
[--composition_file COMPOSITION_FILE] [-c CLUSTERS]
[-k KMER_LENGTH] [-t THREADS] [-l LENGTH_THRESHOLD]
[-r READ_LENGTH] [--total_percentage_pca TOTAL_PERCENTAGE_PCA]
[-b BASENAME] [-s SEED] [-i ITERATIONS]
[--no_cov_normalization] [--no_total_coverage]
[--no_original_data] [-o] [-d] [-v]
optional arguments:
-h, --help show this help message and exit
--coverage_file COVERAGE_FILE
specify the coverage file, containing a table where
each row correspond to a contig, and each column
correspond to a sample. The values are the average
coverage for this contig in that sample. All values
are separated with tabs.
--composition_file COMPOSITION_FILE
specify the composition file, containing sequences in
fasta format. It is named the composition file since
it is used to calculate the kmer composition (the
genomic signature) of each contig.
-c CLUSTERS, --clusters CLUSTERS
specify maximal number of clusters for VGMM, default
400.
-k KMER_LENGTH, --kmer_length KMER_LENGTH
specify kmer length, default 4.
-t THREADS, --threads THREADS
Number of threads to use
-l LENGTH_THRESHOLD, --length_threshold LENGTH_THRESHOLD
specify the sequence length threshold, contigs shorter
than this value will not be included. Defaults to
1000.
-r READ_LENGTH, --read_length READ_LENGTH
specify read length for coverage, default 100
--total_percentage_pca TOTAL_PERCENTAGE_PCA
The percentage of variance explained by the principal
components for the combined data.
-b BASENAME, --basename BASENAME
Specify the basename for files or directory where
outputwill be placed. Path to existing directory or
basenamewith a trailing '/' will be interpreted as a
directory.If not provided, current directory will be
used.
-s SEED, --seed SEED Specify an integer to use as seed for clustering. 0
gives a random seed, 1 is the default seed and any
other positive integer can be used. Other values give
ArgumentTypeError.
-i ITERATIONS, --iterations ITERATIONS
Specify maximum number of iterations for the VBGMM.
Default value is 500
--no_cov_normalization
By default the coverage is normalized with regards to
samples, then normalized with regards of contigs and
finally log transformed. By setting this flag you skip
the normalization and only do log transorm of the
coverage.
--no_total_coverage By default, the total coverage is added as a new
column in the coverage data matrix, independently of
coverage normalization but previous to log
transformation. Use this tag to escape this behaviour.
--no_original_data By default the original data is saved to disk. For big
datasets, especially when a large k is used for
compositional data, this file can become very large.
Use this tag if you don't want to save the original
data.
-o, --converge_out Write convergence info to files.
-d, --debug Debug parameters.
-v, --version show program's version number and exitAfter running CONCOCT you will probably be interested in looking at the scripts merge_cutup_clustering.py and extract_fasta_bins.py to extract your draft bins.
Hope this helps! Francisco
mutantjoo0
Hello CONCOCT team (@andand @alneberg @binnisb @inodb )
I am getting error while running
concoct_coverage_table.py. I am using CONCOCT on miniconda environment as follows:My contigs fa files were generated with megahit and mapped/sorted/indexed bam files were generated with bbmap.
First, following the workflow described in Basic Usage, I cut contigs by running
cut_up_fasta.py -c 10000 -m -b ./concoct_bins/SAMPLE_10k.bed > ./concoct_bins/SAMPLE_10k.fa ./input_megahit_contigs/SAMPLE.final.contigs.fa. Then, I got error in second step by runningconcoct_coverage_table.py. I tested with multiple samples, same error occurred repeatedly. I assumed that CONCOCT does not recognize columns in bed file resulted from first step,cut_up_fasta.py. I have attached example of my error as follows:I wonder which step I can fix this issue, for example, running
cut_up_fasta.pywith increased or decreased-csetting. Please enlighten me. Thank you for your time and support in advance.added on June 30, 2020
I tried a couple of different approaches to fix this issue. 1) use different parameters for step1. cut_up.fasta -c10k->20k; 2) create a new concoct environment and install concoct there; 3) install optional dependencies: bedtools, picard, samtools, bowtie2, gnu parallel, pysam in concoct environment.
The followings are corresponding results from each trial. Please note that I only copied and pasted commands and Traceback parts shown errors.
1) errors from running on concoct environment
2) errors from different parameter set in step 1
3) errors after installing optional dependencies
As you can see running concoct_coverage_table.py after installing optional dependencies reduced numbers of errors. However there are still errors regarding line 77 and line 28. Please help me to understand/fix this problem.
Stay healthy, Joo-Young