Closed maojn closed 4 years ago
Also running into the same problem while trying to set up my genome lib. Any luck?
Hi,
It may be that the cosmic data formatting has changed. We'll look into this and aim to have a new software release out by the end of the week.
On Sun, Oct 13, 2019 at 10:30 PM peter-yufan-zeng notifications@github.com wrote:
Also running into the same problem while trying to set up my genome lib. Any luck?
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/34?email_source=notifications&email_token=ABZRKX2DYAJB5SD43LIZ7GLQOPKUTA5CNFSM4IVX3NBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBDGWSI#issuecomment-541485897, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKX2AS4JWEZDP2GTTCZLQOPKUTANCNFSM4IVX3NBA .
Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas
Hi - the system is fully revised now and has a simpler setup. Please let me know if the new release gives you any trouble. The wiki is updated w/ pointers to the new ctat genome lib supplement for mutation calling too.
Hi Brian - thanks so much for your help.
After updating all my files, now I am getting an this error
Code
$ python mutation_lib_prep/ctat-mutation-lib-integration.py --genome_lib_dir GRCh38_gencode_v31_CTAT_lib_Oct012019.plug-n-play/ctat_genome_lib_build_dir/ --CosmicCodingMuts GRCh38_gencode_v31_CTAT_lib_Oct012019.plug-n-play/ctat_genome_lib_build_dir/ctat_mutation_lib/cosmic.vcf.gz --CosmicMutantExport GRCh38_gencode_v31_CTAT_lib_Oct012019.plug-n-play/CosmicMutantExport.tsv.gz
Error
2019-10-22 11:04:27,337: ref dict created for gatk use in pipe 2019-10-22 11:04:27,337: Capturing info from: GRCh38_gencode_v31_CTAT_lib_Oct012019.plug-n-play/CosmicMutantExport.tsv.gz Traceback (most recent call last): File "mutation_lib_prep/ctat-mutation-lib-integration.py", line 104, in <module> for row in mutant_reader: File "/home/ionadmin/anaconda3/lib/python3.7/csv.py", line 111, in __next__ self.fieldnames File "/home/ionadmin/anaconda3/lib/python3.7/csv.py", line 98, in fieldnames self._fieldnames = next(self.reader) _csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)
Hi Peter,
In the latest release, you don't need to run that step. The new ctat mutation lib supplement comes with what you need.
best,
~brian
On Tue, Oct 22, 2019 at 11:22 AM Peter Zeng notifications@github.com wrote:
Hi Brian - thanks so much for your help.
After updating all my files, now I am getting an this error
Code $ python mutation_lib_prep/ctat-mutation-lib-integration.py --genome_lib_dir GRCh38_gencode_v31_CTAT_lib_Oct012019.plug-n-play/ctat_genome_lib_build_dir/ --CosmicCodingMuts GRCh38_gencode_v31_CTAT_lib_Oct012019.plug-n-play/ctat_genome_lib_build_dir/ctat_mutation_lib/cosmic.vcf.gz --CosmicMutantExport GRCh38_gencode_v31_CTAT_lib_Oct012019.plug-n-play/CosmicMutantExport.tsv.gz
Error 2019-10-22 11:04:27,337: ref dict created for gatk use in pipe 2019-10-22 11:04:27,337: Capturing info from: GRCh38_gencode_v31_CTAT_lib_Oct012019.plug-n-play/CosmicMutantExport.tsv.gz Traceback (most recent call last): File "mutation_lib_prep/ctat-mutation-lib-integration.py", line 104, in
for row in mutant_reader: File "/home/ionadmin/anaconda3/lib/python3.7/csv.py", line 111, in next self.fieldnames File "/home/ionadmin/anaconda3/lib/python3.7/csv.py", line 98, in fieldnames self._fieldnames = next(self.reader) _csv.Error: iterator should return strings, not bytes (did you open the file in text mode?) — You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/34?email_source=notifications&email_token=ABZRKX55F3CRQSENEQJ64NDQP4LE5A5CNFSM4IVX3NBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEB6ERGI#issuecomment-545015961, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKX2E6J4YOJUNEHR3ALTQP4LE5ANCNFSM4IVX3NBA .
Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas
Hi Brian, Thanks again for your help. I have tried running ctat-mutation again, and it seems the folder now is lacking dbsnp_138.vcf.gz and radar.txt. I renamed dbsbp.vcf.gz to dbsnp_138.vcf.fz in the ctat_mutation_lib, but I am still missing the radar file.
Error code
FileNotFoundError: [Errno 2] No such file or directory: '/media/ionadmin/b68e1a06-1bb3-442c-bb3c-553aab1edfce/bioinformatics/ctat-mutations/GRCh38_gencode_v31_CTAT_lib_Oct012019.plug-n-play/ctat_genome_lib_build_dir/ctat_mutation_lib/radar.txt' Traceback (most recent call last): File "/media/ionadmin/b68e1a06-1bb3-442c-bb3c-553aab1edfce/bioinformatics/ctat-mutations/ctat_mutations", line 1649, in <module> pipeliner.run() File "/media/ionadmin/b68e1a06-1bb3-442c-bb3c-553aab1edfce/bioinformatics/ctat-mutations/PyLib/Pipeliner.py", line 53, in run run_cmd(cmd.get_cmd()) File "/media/ionadmin/b68e1a06-1bb3-442c-bb3c-553aab1edfce/bioinformatics/ctat-mutations/PyLib/Pipeliner.py", line 17, in run_cmd subprocess.check_call(cmd, shell=True) File "/home/ionadmin/anaconda3/lib/python3.7/subprocess.py", line 347, in check_call raise CalledProcessError(retcode, cmd)
Hi Peter,
Can you pull down the latest software: v2.3.0b https://github.com/NCIP/ctat-mutations/releases
I had to cut a new release recently, in case that's part of the problem you're experiencing here.
The CTAT mutation supplement https://github.com/NCIP/ctat-mutations/wiki/CTAT-mutations-installation
is the same as from the earlier faulty v2.3.0 release.
I'll put out a broader announcement about this.
Let me know if the latest gives you any trouble.
best,
~brian
On Thu, Oct 31, 2019 at 9:53 AM Peter Zeng notifications@github.com wrote:
Hi Brian, Thanks again for your help. I have tried running ctat-mutation again, and it seems the folder now is lacking dbsnp_138.vcf.gz and radar.txt. I renamed dbsbp.vcf.gz to dbsnp_138.vcf.fz in the ctat_mutation_lib, but I am still missing the radar file.
Error code FileNotFoundError: [Errno 2] No such file or directory: '/media/ionadmin/b68e1a06-1bb3-442c-bb3c-553aab1edfce/bioinformatics/ctat-mutations/GRCh38_gencode_v31_CTAT_lib_Oct012019.plug-n-play/ctat_genome_lib_build_dir/ctat_mutation_lib/radar.txt' Traceback (most recent call last): File "/media/ionadmin/b68e1a06-1bb3-442c-bb3c-553aab1edfce/bioinformatics/ctat-mutations/ctat_mutations", line 1649, in
pipeliner.run() File "/media/ionadmin/b68e1a06-1bb3-442c-bb3c-553aab1edfce/bioinformatics/ctat-mutations/PyLib/Pipeliner.py", line 53, in run run_cmd(cmd.get_cmd()) File "/media/ionadmin/b68e1a06-1bb3-442c-bb3c-553aab1edfce/bioinformatics/ctat-mutations/PyLib/Pipeliner.py", line 17, in run_cmd subprocess.check_call(cmd, shell=True) File "/home/ionadmin/anaconda3/lib/python3.7/subprocess.py", line 347, in check_call raise CalledProcessError(retcode, cmd) — You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/34?email_source=notifications&email_token=ABZRKX26WNVJBOYNCNMRPNDQRLPNRA5CNFSM4IVX3NBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECX3IOQ#issuecomment-548385850, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKX7XQSJBZTSVXKQZJHLQRLPNRANCNFSM4IVX3NBA .
Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas
Hi Brian, After updating ctat-mutation to the newest version, I no longer get an error like above.
However, I now encounter an error below. It seems to be missing the file ref_annot.splice_adj.bed.gz, which I cannot find in the mutation_lib file that I downloaded (the oct 18th version)
thanks in advance again for your help! I really appreciate it
CRITICAL:__main__:cannot locate required file: /media/ionadmin/b68e1a06-1bb3-442c-bb3c-553aab1edfce/bioinformatics/ctat-mutations/GRCh38_gencode_v31_CTAT_lib_Oct012019.plug-n-play/ctat_genome_lib_build_dir/ctat_mutation_lib/ref_annot.splice_adj.bed.gz 2019-11-02 17:57:26,400: ERROR Pipeliner.Pipeliner.run_cmd Error: Command '/media/ionadmin/b68e1a06-1bb3-442c-bb3c-553aab1edfce/bioinformatics/ctat-mutations/src/annotate_exon_splice_proximity.py --input_vcf /media/ionadmin/b68e1a06-1bb3-442c-bb3c-553aab1edfce/Peter/20190916_OICR_Data/190718_D00430_0371_ACDPHCANXX/104R/test/work_dir/HaplotypeCaller.raw_variants.vcf_snpeff.vcf_adj.vcf_RNAedit.vcf_PASSreads.vcf_Repeats.vcf_homopolymers.vcf.gz --ctat_mutation_lib_dir /media/ionadmin/b68e1a06-1bb3-442c-bb3c-553aab1edfce/bioinformatics/ctat-mutations/GRCh38_gencode_v31_CTAT_lib_Oct012019.plug-n-play/ctat_genome_lib_build_dir/ctat_mutation_lib --output_vcf /media/ionadmin/b68e1a06-1bb3-442c-bb3c-553aab1edfce/Peter/20190916_OICR_Data/190718_D00430_0371_ACDPHCANXX/104R/test/work_dir/HaplotypeCaller.raw_variants.vcf_snpeff.vcf_adj.vcf_RNAedit.vcf_PASSreads.vcf_Repeats.vcf_homopolymers.vcf_SpliceDist.vcf --tmpdir /media/ionadmin/b68e1a06-1bb3-442c-bb3c-553aab1edfce/Peter/20190916_OICR_Data/190718_D00430_0371_ACDPHCANXX/104R/test/work_dir' returned non-zero exit status 1., exit val: 1 2019-11-02 17:57:26,401: ERROR Pipeliner.Pipeliner.run Error, command: [ /media/ionadmin/b68e1a06-1bb3-442c-bb3c-553aab1edfce/bioinformatics/ctat-mutations/src/annotate_exon_splice_proximity.py --input_vcf /media/ionadmin/b68e1a06-1bb3-442c-bb3c-553aab1edfce/Peter/20190916_OICR_Data/190718_D00430_0371_ACDPHCANXX/104R/test/work_dir/HaplotypeCaller.raw_variants.vcf_snpeff.vcf_adj.vcf_RNAedit.vcf_PASSreads.vcf_Repeats.vcf_homopolymers.vcf.gz --ctat_mutation_lib_dir /media/ionadmin/b68e1a06-1bb3-442c-bb3c-553aab1edfce/bioinformatics/ctat-mutations/GRCh38_gencode_v31_CTAT_lib_Oct012019.plug-n-play/ctat_genome_lib_build_dir/ctat_mutation_lib --output_vcf /media/ionadmin/b68e1a06-1bb3-442c-bb3c-553aab1edfce/Peter/20190916_OICR_Data/190718_D00430_0371_ACDPHCANXX/104R/test/work_dir/HaplotypeCaller.raw_variants.vcf_snpeff.vcf_adj.vcf_RNAedit.vcf_PASSreads.vcf_Repeats.vcf_homopolymers.vcf_SpliceDist.vcf --tmpdir /media/ionadmin/b68e1a06-1bb3-442c-bb3c-553aab1edfce/Peter/20190916_OICR_Data/190718_D00430_0371_ACDPHCANXX/104R/test/work_dir ] failed, stack trace: [ st: file:/media/ionadmin/b68e1a06-1bb3-442c-bb3c-553aab1edfce/bioinformatics/ctat-mutations/PyLib/Pipeliner.py, lineno:86
Hi Peter,
The ctat mutation lib integration step should be generating that file. Did you run through the full installation procedure for the latest release?
This involves running: ctat-mutations/mutation_lib_prep/ctat-mutation-lib-integration.py
as part of
https://github.com/NCIP/ctat-mutations/wiki/CTAT-mutations-installation
best,
~b
On Sat, Nov 2, 2019 at 11:16 PM Peter Zeng notifications@github.com wrote:
Hi Brian, After updating ctat-mutation to the newest version, I no longer get an error like above.
However, I now encounter an error below. It seems to be missing the file ref_annot.splice_adj.bed.gz, which I cannot find in the mutation_lib file that I downloaded (the oct 18th version)
thanks in advance again for your help! I really appreciate it
CRITICAL:main:cannot locate required file: /media/ionadmin/b68e1a06-1bb3-442c-bb3c-553aab1edfce/bioinformatics/ctat-mutations/GRCh38_gencode_v31_CTAT_lib_Oct012019.plug-n-play/ctat_genome_lib_build_dir/ctat_mutation_lib/ref_annot.splice_adj.bed.gz 2019-11-02 17:57:26,400: ERROR Pipeliner.Pipeliner.run_cmd Error: Command '/media/ionadmin/b68e1a06-1bb3-442c-bb3c-553aab1edfce/bioinformatics/ctat-mutations/src/annotate_exon_splice_proximity.py --input_vcf /media/ionadmin/b68e1a06-1bb3-442c-bb3c-553aab1edfce/Peter/20190916_OICR_Data/190718_D00430_0371_ACDPHCANXX/104R/test/work_dir/HaplotypeCaller.raw_variants.vcf_snpeff.vcf_adj.vcf_RNAedit.vcf_PASSreads.vcf_Repeats.vcf_homopolymers.vcf.gz --ctat_mutation_lib_dir /media/ionadmin/b68e1a06-1bb3-442c-bb3c-553aab1edfce/bioinformatics/ctat-mutations/GRCh38_gencode_v31_CTAT_lib_Oct012019.plug-n-play/ctat_genome_lib_build_dir/ctat_mutation_lib --output_vcf /media/ionadmin/b68e1a06-1bb3-442c-bb3c-553aab1edfce/Peter/20190916_OICR_Data/190718_D00430_0371_ACDPHCANXX/104R/test/work_dir/HaplotypeCaller.raw_variants.vcf_snpeff.vcf_adj.vcf_RNAedit.vcf_PASSreads.vcf_Repeats.vcf_homopolymers.vcf_SpliceDist.vcf --tmpdir /media/ionadmin/b68e1a06-1bb3-442c-bb3c-553aab1edfce/Peter/20190916_OICR_Data/190718_D00430_0371_ACDPHCANXX/104R/test/work_dir' returned non-zero exit status 1., exit val: 1 2019-11-02 17:57:26,401: ERROR Pipeliner.Pipeliner.run Error, command: [ /media/ionadmin/b68e1a06-1bb3-442c-bb3c-553aab1edfce/bioinformatics/ctat-mutations/src/annotate_exon_splice_proximity.py --input_vcf /media/ionadmin/b68e1a06-1bb3-442c-bb3c-553aab1edfce/Peter/20190916_OICR_Data/190718_D00430_0371_ACDPHCANXX/104R/test/work_dir/HaplotypeCaller.raw_variants.vcf_snpeff.vcf_adj.vcf_RNAedit.vcf_PASSreads.vcf_Repeats.vcf_homopolymers.vcf.gz --ctat_mutation_lib_dir /media/ionadmin/b68e1a06-1bb3-442c-bb3c-553aab1edfce/bioinformatics/ctat-mutations/GRCh38_gencode_v31_CTAT_lib_Oct012019.plug-n-play/ctat_genome_lib_build_dir/ctat_mutation_lib --output_vcf /media/ionadmin/b68e1a06-1bb3-442c-bb3c-553aab1edfce/Peter/20190916_OICR_Data/190718_D00430_0371_ACDPHCANXX/104R/test/work_dir/HaplotypeCaller.raw_variants.vcf_snpeff.vcf_adj.vcf_RNAedit.vcf_PASSreads.vcf_Repeats.vcf_homopolymers.vcf_SpliceDist.vcf --tmpdir /media/ionadmin/b68e1a06-1bb3-442c-bb3c-553aab1edfce/Peter/20190916_OICR_Data/190718_D00430_0371_ACDPHCANXX/104R/test/work_dir ] failed, stack trace: [ st: file:/media/ionadmin/b68e1a06-1bb3-442c-bb3c-553aab1edfce/bioinformatics/ctat-mutations/PyLib/Pipeliner.py, lineno:86
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/34?email_source=notifications&email_token=ABZRKXY3OWMKMJ4SLOM2EY3QRY7CJA5CNFSM4IVX3NBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC5J6UI#issuecomment-549101393, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKXYNL7LBXUC2MMUMZELQRY7CJANCNFSM4IVX3NBA .
Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas
Hi Brian it works now! Thanks so much for your help! 👍
Great to hear!
We're still making improvements, so stay tuned to our google forum for updates.
best,
~brian
On Tue, Nov 5, 2019 at 12:45 PM Peter Zeng notifications@github.com wrote:
Hi Brian it works now! Thanks so much for your help! 👍
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/NCIP/ctat-mutations/issues/34?email_source=notifications&email_token=ABZRKX4KFUEXZGC6ZHYGUZ3QSGWKDA5CNFSM4IVX3NBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDDVCFQ#issuecomment-549933334, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKXZMYUIKTJ4MN2ZORITQSGWKDANCNFSM4IVX3NBA .
Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas
Hi,
I ran into problem during Genome lib setup step.
$ export PICARD_HOME=/usr/local/Anaconda/envs_app/ctatmutaions/2.0.1/share/picard-2.18.14-0/
$ python /usr/local/Anaconda/envs_app/ctatmutaions/2.0.1/share/ctat-mutations-2.0.1-5/mutation_lib_prep/ctat-mutation-lib-integration.py \ --CosmicCodingMuts /usr/local/apps/ctatmutations/GRCh37/CosmicCodingMuts.vcf.gz \ --CosmicMutantExport /usr/local/apps/ctatmutations/GRCh37/CosmicMutantExport.tsv.gz \ --genome_lib_dir /usr/local/apps/ctatmutations/GRCh37/GRCh37_gencode_v19_CTAT_lib_Aug152019.plug-n-play/ctat_genome_lib_build_dir/
2019-09-11 14:59:46,738: Uncompressing mutation lib in genome_lib_dir ctat_mutation_lib/ ctat_mutation_lib/rediportal.txt ctat_mutation_lib/header/ ctat_mutation_lib/header/cravat_annotation.txt ctat_mutation_lib/dbsnp_138.vcf.gz.tbi ctat_mutation_lib/dbsnp_138.vcf.gz ctat_mutation_lib/radar.txt ctat_mutation_lib/refGene.sort.bed ctat_mutation_lib/dbsnp_138.vcf.gz.csi ctat_mutation_lib/properties.json 2019-09-11 15:00:09,081: Generating /usr/local/apps/ctatmutations/GRCh37/GRCh37_gencode_v19_CTAT_lib_Aug152019.plug-n-play/ctat_genome_l ib_build_dir/ref_genome.dict INFO 2019-09-11 15:00:10 CreateSequenceDictionary
** NOTE: Picard's command line syntax is changing.
** For more information, please see: ** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
** The command line looks like this in the new syntax:
** CreateSequenceDictionary -R /usr/local/apps/ctatmutations/GRCh37/GRCh37_gencode_v19_CTAT_lib_Aug152019.plug-n-play/ctat_genome_lib_build_dir/ref_genome.fa -O /usr/local/apps/ctatmutations/GRCh37/GRCh37_gencode_v19_CTAT_lib_Aug152019.plug-n-play/ctat_genome_lib_build_dir/ref_genome.dict -VALIDATION_STRINGENCY LENIENT
15:00:10.932 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/usr/local/Anaconda/envs_app/ctatmutaions/2.0.1/share/picard-2.18.14-0/picard.jar!/com/intel/gkl/native/libgkl_compression.so [Wed Sep 11 15:00:10 EDT 2019] CreateSequenceDictionary OUTPUT=/usr/local/apps/ctatmutations/GRCh37/GRCh37_gencode_v19_CTAT_lib_Aug152019.plug-n-play/ctat_genome_lib_build_dir/ref_genome.dict VALIDATION_STRINGENCY=LENIENT REFERENCE=/usr/local/apps/ctatmutations/GRCh37/GRCh37_gencode_v19_CTAT_lib_Aug152019.plug-n-play/ctat_genome_lib_build_dir/ref_genome.fa TRUNCATE_NAMES_AT_WHITESPACE=true NUM_SEQUENCES=2147483647 VERBOSITY=INFO QUIET=false COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false [Wed Sep 11 15:00:10 EDT 2019] Executing as maoj@cn3097 on Linux 3.10.0-862.14.4.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_152-release-1056-b12; Deflater: Intel; Inflater: Intel; Provider GCS is not available; Picard version: 2.18.14-SNAPSHOT [Wed Sep 11 15:00:35 EDT 2019] picard.sam.CreateSequenceDictionary done. Elapsed time: 0.41 minutes. Runtime.totalMemory()=3670540288 2019-09-11 15:00:35,463: ref dict created for gatk use in pipe 2019-09-11 15:00:35,464: Capturing info from: /usr/local/apps/ctatmutations/GRCh37/CosmicMutantExport.tsv.gz Traceback (most recent call last): File "/usr/local/Anaconda/envs_app/ctatmutaions/2.0.1/share/ctat-mutations-2.0.1-5/mutation_lib_prep/ctat-mutation-lib-integration.py", line 116, in
mutant_dict_necessary_info[row["Mutation ID"]]=info
KeyError: u'Mutation ID'
Can someone help?