bash master.sh window.t0X errors

[nicholasblackburn]

Hi there,

I've seen this cross posted a few places without response and have also put it on the google group.

Haven't been able to get gemini to work yet at all.

Ran gemini update, works fine. Ran bash master-test.sh (results below)

Fails at query.t22...\c and then again for all window.t0X...\c tests.

Regards,

Nick

Nicholass-MacBook-Pro:gemini-master Nicholas$ gemini update /usr/local/share/gemini/anaconda/bin/conda Fetching package metadata: .. Solving package specifications: . Package plan for installation in environment /usr/local/share/gemini/anaconda:

The following packages will be UPDATED:

numpy: 1.8.2-py27_0 --> 1.9.0-py27_0

Unlinking packages ... [ COMPLETE ] |##############################################################################################################################################################################| 100% Linking packages ... [ COMPLETE ] |##############################################################################################################################################################################| 100% Fetching package metadata: ... Solving package specifications: ............... Package plan for installation in environment /usr/local/share/gemini/anaconda:

The following packages will be DOWNGRADED:

numpy: 1.9.0-py27_0 --> 1.8.2-py27_0

Unlinking packages ... [ COMPLETE ] |##############################################################################################################################################################################| 100% Linking packages ... [ COMPLETE ] |##############################################################################################################################################################################| 100% Requirement already satisfied (use --upgrade to upgrade): numpy>=1.7.1 in /usr/local/share/gemini/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/arq5x/gemini/master/requirements.txt (line 1)) Requirement already satisfied (use --upgrade to upgrade): pyparsing>=1.5.6,<=1.5.7 in /usr/local/share/gemini/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/arq5x/gemini/master/requirements.txt (line 2)) Requirement already satisfied (use --upgrade to upgrade): pysam>=0.6 in /usr/local/share/gemini/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/arq5x/gemini/master/requirements.txt (line 3)) Requirement already satisfied (use --upgrade to upgrade): cyvcf>=0.1.10 in /usr/local/share/gemini/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/arq5x/gemini/master/requirements.txt (line 4)) Requirement already satisfied (use --upgrade to upgrade): PyYAML>=3.10 in /usr/local/share/gemini/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/arq5x/gemini/master/requirements.txt (line 5)) Requirement already satisfied (use --upgrade to upgrade): pybedtools>=0.6.2 in /usr/local/share/gemini/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/arq5x/gemini/master/requirements.txt (line 6)) Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7.1 in /usr/local/share/gemini/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/arq5x/gemini/master/requirements.txt (line 7)) Requirement already satisfied (use --upgrade to upgrade): python-graph-core>=1.8.2 in /usr/local/share/gemini/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/arq5x/gemini/master/requirements.txt (line 8)) Requirement already satisfied (use --upgrade to upgrade): python-graph-dot>=1.8.2 in /usr/local/share/gemini/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/arq5x/gemini/master/requirements.txt (line 9)) Requirement already satisfied (use --upgrade to upgrade): bottle>=0.11.6 in /usr/local/share/gemini/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/arq5x/gemini/master/requirements.txt (line 10)) Requirement already satisfied (use --upgrade to upgrade): ipython-cluster-helper>=0.2.23 in /usr/local/share/gemini/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/arq5x/gemini/master/requirements.txt (line 11)) Requirement already satisfied (use --upgrade to upgrade): bx-python>=0.7.1 in /usr/local/share/gemini/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/arq5x/gemini/master/requirements.txt (line 12)) Requirement already satisfied (use --upgrade to upgrade): pandas>=0.11.0 in /usr/local/share/gemini/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/arq5x/gemini/master/requirements.txt (line 13)) Requirement already satisfied (use --upgrade to upgrade): openpyxl>=1.6.1,<2.0.0 in /usr/local/share/gemini/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/arq5x/gemini/master/requirements.txt (line 14)) Requirement already satisfied (use --upgrade to upgrade): scipy>=0.12.0 in /usr/local/share/gemini/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/arq5x/gemini/master/requirements.txt (line 15)) Requirement already satisfied (use --upgrade to upgrade): Unidecode>=0.04.14 in /usr/local/share/gemini/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/arq5x/gemini/master/requirements.txt (line 16)) Downloading/unpacking gemini==0.10.1 (from -r https://raw.githubusercontent.com/arq5x/gemini/master/requirements.txt (line 17)) Downloading gemini-0.10.1.tar.gz (15.9MB): 15.9MB downloaded Running setup.py (path:/private/var/folders/7g/nqfggqvj7p13bfny9n2krt5c0000gn/T/pip_build_Nicholas/gemini/setup.py) egg_info for package gemini

Requirement already satisfied (use --upgrade to upgrade): pydot in /usr/local/share/gemini/anaconda/lib/python2.7/site-packages (from python-graph-dot>=1.8.2->-r https://raw.githubusercontent.com/arq5x/gemini/master/requirements.txt (line 9)) Requirement already satisfied (use --upgrade to upgrade): pyzmq>=2.1.11 in /usr/local/share/gemini/anaconda/lib/python2.7/site-packages (from ipython-cluster-helper>=0.2.23->-r https://raw.githubusercontent.com/arq5x/gemini/master/requirements.txt (line 11)) Requirement already satisfied (use --upgrade to upgrade): ipython>=1.1.0 in /usr/local/share/gemini/anaconda/lib/python2.7/site-packages (from ipython-cluster-helper>=0.2.23->-r https://raw.githubusercontent.com/arq5x/gemini/master/requirements.txt (line 11)) Requirement already satisfied (use --upgrade to upgrade): netifaces>=0.10.3 in /usr/local/share/gemini/anaconda/lib/python2.7/site-packages (from ipython-cluster-helper>=0.2.23->-r https://raw.githubusercontent.com/arq5x/gemini/master/requirements.txt (line 11)) Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/local/share/gemini/anaconda/lib/python2.7/site-packages/setuptools-7.0-py2.7.egg (from pydot->python-graph-dot>=1.8.2->-r https://raw.githubusercontent.com/arq5x/gemini/master/requirements.txt (line 9)) Installing collected packages: gemini Found existing installation: gemini 0.11.0a Uninstalling gemini: Successfully uninstalled gemini Running setup.py install for gemini

changing mode of build/scripts-2.7/gemini from 644 to 755
changing mode of /usr/local/share/gemini/anaconda/bin/gemini to 755

Successfully installed gemini Cleaning up... Gemini upgraded to latest version Checking required dependencies... curl found

• downloading https://s3.amazonaws.com/gemini-annotations/dbsnp.hg19.b141.vcf.gz to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/dbsnp.hg19.b141.vcf.gz.tbi to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/clinvar_20140807.vcf.gz to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/clinvar_20140807.vcf.gz.tbi to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/29way_pi_lods_elements_12mers.chr_specific.fdr_0.1_with_scores.txt.hg19.merged.bed.gz to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/29way_pi_lods_elements_12mers.chr_specific.fdr_0.1_with_scores.txt.hg19.merged.bed.gz.tbi to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/hg19.CpG.bed.gz to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/hg19.CpG.bed.gz.tbi to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/hg19.pfam.ucscgenes.bed.gz to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/hg19.pfam.ucscgenes.bed.gz.tbi to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/hg19.gerp.elements.bed.gz to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/hg19.gerp.elements.bed.gz.tbi to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/hg19.cytoband.bed.gz to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/hg19.cytoband.bed.gz.tbi to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/hg19.dgv.bed.gz to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/hg19.dgv.bed.gz.tbi to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/hg19.gwas.bed.gz to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/hg19.gwas.bed.gz.tbi to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/hg19.rmsk.bed.gz to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/hg19.rmsk.bed.gz.tbi to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/hg19.segdup.bed.gz to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/hg19.segdup.bed.gz.tbi to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/ESP6500SI.all.snps_indels.vcf.gz to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/ESP6500SI.all.snps_indels.vcf.gz.tbi to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/ALL.wgs.integrated_phase1_v3.20101123.snps_indels_sv.sites.2012Oct12.vcf.gz to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/ALL.wgs.integrated_phase1_v3.20101123.snps_indels_sv.sites.2012Oct12.vcf.gz.tbi to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/genetic_map_HapMapII_GRCh37.gz to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/genetic_map_HapMapII_GRCh37.gz.tbi to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/wgEncodeRegTfbsClusteredV2.cell_count.20130213.bed.gz to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/wgEncodeRegTfbsClusteredV2.cell_count.20130213.bed.gz.tbi to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/encode.6celltypes.consensus.bedg.gz to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/encode.6celltypes.consensus.bedg.gz.tbi to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/stam.125cells.dnaseI.hg19.bed.gz to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/stam.125cells.dnaseI.hg19.bed.gz.tbi to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/GRCh37-gms-mappability.vcf.gz to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/GRCh37-gms-mappability.vcf.gz.tbi to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/GRC_patch_regions.bed.gz to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/GRC_patch_regions.bed.gz.tbi to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/kegg_pathways_ensembl66 to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/kegg_pathways_ensembl67 to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/kegg_pathways_ensembl68 to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/kegg_pathways_ensembl69 to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/kegg_pathways_ensembl70 to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/kegg_pathways_ensembl71 to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/hprd_interaction_graph to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/cse-hiseq-8_4-2013-02-20.bed.gz to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/cse-hiseq-8_4-2013-02-20.bed.gz.tbi to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/hg19.vista.enhancers.20131108.bed.gz to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/hg19.vista.enhancers.20131108.bed.gz.tbi to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/cosmic-v68-GRCh37.vcf.gz to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/cosmic-v68-GRCh37.vcf.gz.tbi to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/detailed_gene_table_v75 to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/summary_gene_table_v75 to /usr/local/share/gemini/data
• downloading https://s3.amazonaws.com/gemini-annotations/cancer_gene_census.20140120.tsv to /usr/local/share/gemini/data
• downloading http://compgen.bscb.cornell.edu/fitCons/0downloads/tracks/V1.01/i6/scores/fc-i6-0.bw to /usr/local/share/gemini/data

Gemini data files updated remote: Counting objects: 6, done. remote: Compressing objects: 100% (6/6), done. remote: Total 6 (delta 1), reused 0 (delta 0) Unpacking objects: 100% (6/6), done. From https://github.com/arq5x/gemini

• branch master -> FETCH_HEAD d735735..b15129e master -> origin/master Updating 7b24eb0..b15129e Fast-forward docs/content/database_schema.rst | 27 +++++++++++----------- docs/content/history.rst | 19 ++++++++++++---- docs/images/overview.png | Bin 0 -> 237447 bytes docs/index.rst | 9 ++++++-- gemini/GeminiQuery.py | 28 +++++++++++++++-------- gemini/annotation_provenance/make-1000G.sh | 11 ++++++--- gemini/annotations.py | 12 +++++----- gemini/database.py | 25 +++++++++++++++++---- gemini/gemini_annotate.py | 10 ++++++--- gemini/gemini_load_chunk.py | 117 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---------------------------- gemini/gemini_main.py | 54 +++++++++++++++++++++++++++++++++++++------- gemini/gemini_set_somatic.py | 46 +++++++++++++++++++++----------------- gemini/infotag.py | 8 ++++++- gemini/install-data.py | 3 ++- gemini/structural_variants.py | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ gemini/tool_burden_tests.py | 4 ++-- gemini/tool_fusions.py | 188 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ gemini/version.py | 2 +- requirements.txt | 2 +- test/test-interactions.sh | 148 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------------------------------------------------------ test/test-pop.sh | 21 ++++++++--------- test/test-query.sh | 14 ++++++------ 22 files changed, 619 insertions(+), 202 deletions(-) create mode 100644 docs/images/overview.png mode change 100644 => 100755 gemini/annotation_provenance/make-1000G.sh create mode 100644 gemini/structural_variants.py create mode 100644 gemini/toolfusions.py Previous HEAD position was b15129e... Merge branch 'master' of github.com:arq5x/gemini HEAD is now at 7b24eb0... v0.10.1 Run test suite with: cd /usr/local/share/gemini/gemini && bash master-test.sh Nicholass-MacBook-Pro:gemini-master Nicholas$ cd /usr/local/share/gemini/gemini Nicholass-MacBook-Pro:gemini Nicholas$ bash master-test.sh rm: ./.db: No such file or directory pid 27723: 4 variants processed. pid 27726: 18 variants processed. pid 27729: 9 variants processed. pid 27732: 10 variants processed. pid 27735: 7 variants processed. pid 27738: 7 variants processed. updated 7 variants pid 27743: 9 variants processed. pid 27747: 8 variants processed. pid 27750: 7 variants processed. pid 27753: 9 variants processed. pid 27756: 9 variants processed. updated 9 variants pid 27761: 9 variants processed. pid 27764: 9 variants processed. updated 9 variants pid 27770: 879 variants processed. pid 27773: 879 variants processed. updated 879 variants Bgzipping test.query.vcf into test.query.vcf.gz. Indexing test.query.vcf.gz with grabix. Loading 879 variants. Breaking test.query.vcf.gz into 2 chunks. Loading chunk 0. Loading chunk 1. pid 27789: 440 variants processed. pid 27787: 439 variants processed. Done loading 879 variants in 2 chunks. 2014-10-30 10:37:52 merging 2 chunks. 2014-10-30 10:37:55 indexing final database. pid 27797: 9 variants processed. updated 9 variants pid 27802: 9 variants processed. updated 9 variants pid 27807: 8 variants processed. pid 27810: 2 variants processed. pid 27813: 5 variants processed. pid 27816: 5 variants processed. pid 27819: 5 variants processed. pid 27822: 5 variants processed. pid 27825: 5 variants processed. pid 27828: 5 variants processed. pid 27831: 5 variants processed. pid 27834: 5 variants processed. pid 27837: 5 variants processed. pid 27840: 5 variants processed. pid 27843: 5 variants processed. pid 27846: 5 variants processed. pid 27852: 5 variants processed. pid 27855: 9 variants processed. pid 27858: 7 variants processed. pid 27861: 7 variants processed. pid 27864: 9 variants processed. region.t01...\c ok region.t02...\c ok region.t03...\c ok region.t04...\c ok region.t05...\c ok amend.t01...\c ok query.t01...\c ok query.t02...\c ok query.t03...\c ok query.t04...\c ok query.t05...\c ok query.t06...\c ok query.t07...\c ok query.t08...\c ok query.t09...\c ok query.t10...\c ok query.t11...\c ok query.t12...\c ok query.t13...\c ok query.t14...\c ok query.t15...\c ok query.t16...\c ok query.t17...\c ok query.t18...\c ok query.t19...\c ok query.t20...\c ok query.t21...\c ok query.t22...\c 1c1 < 10 10:1142207-1142208:T|C:1 0 1142207 C C C C C C C C

  10 10:1142207-1142208:C:1 0 1142207 C C C C C C C C 3,8c3,8 < 10 10:52004314-52004315:T|C:3 0 52004314 0 0 0 0 C C C C < 10 10:52497528-52497529:G|C:4 0 52497528 0 0 C C C C 0 0 < 10 10:126678091-126678092:G|A:5 0 126678091 G G G G G G G A < 10 10:135210790-135210791:T|C:6 0 135210790 T T C C C C T T < 10 10:135336655-135336656:G|A:7 0 135336655 0 0 A A 0 0 A A < 10 10:135369531-135369532:T|C:8 0 135369531 T T T C T C T T

  10 10:52004314-52004315:C:3 0 52004314 0 0 0 0 C C C C 10 10:52497528-52497529:C:4 0 52497528 0 0 C C C C 0 0 10 10:126678091-126678092:A|G:5 0 126678091 G G G G G G G A 10 10:135210790-135210791:C|T:6 0 135210790 T T C C C C T T 10 10:135336655-135336656:A:7 0 135336655 0 0 A A 0 0 A A 10 10:135369531-135369532:C|T:8 0 135369531 T T T C T C T T fail query.t23...\c ok query.t24...\c ok query.t25...\c ok query.t26...\c ok query.t27...\c ok query.t28...\c ok query.t29...\c ok query.t30...\c ok query.t31...\c ok query.t32...\c ok query.t33...\c ok query.t34...\c ok query.t35...\c ok query.t36...\c ok query.t37...\c ok query.t38...\c ok query.t39...\c ok query.t40...\c ok dump.t01...\c ok burden.t1...\c ok burden.t2...\c ok burden.t3...\c ok columns.t01...\c ok columns.t02...\c ok columns.t03...\c ok columns.t04...\c ok columns.t05...\c ok columns.t06...\c ok columns.t07...\c ok columns.t08...\c ok columns.t09...\c ok columns.t10...\c ok columns.t11...\c ok columns.t12...\c ok columns.t13...\c ok columns.t14...\c ok cadd.t01...\c ok cadd.t02...\c ok cosmic.t01...\c ok pid 28095: 41 variants processed. load.t1...\c ok pid 28101: 38 variants processed. load.t2...\c ok pid 28107: 9 variants processed. load.t3...\c ok pid 28113: 1 variants processed. pid 28113: 1 skipped due to having the FILTER field set. load.t4...\c ok genotypes.t01...\c ok genotypes.t02...\c ok genotypes.t03...\c ok genotypes.t04...\c ok genotypes.t05...\c ok genotypes.t06...\c ok genotypes.t07...\c ok genotypes.t08...\c ok clinvar.t01...\c ok pop_metrics.t1...\c ok pop_metrics.t2...\c ok pop_metrics.t3...\c ok map.t1...\c ok map.t2...\c ok genome.t1...\c ok genome.t2...\c ok genome.t3...\c ok genome.t4...\c ok genome.t5...\c ok genome.t6...\c ok encode.t1...\c ok encode.t2...\c ok encode.t3...\c ok effstring.t01...\c ok effstring.t02...\c ok effstring.t03...\c ok effstring.t04...\c ok annotate-tool.t1...\c updated 10 variants ok annotate-tool.t2...\c updated 10 variants ok annotate-tool.t3...\c updated 2 variants ok annotate-tool.t4...\c updated 2 variants ok annotate-tool.t5...\c updated 2 variants ok annotate-tool.t6...\c updated 2 variants ok annotate-tool.t7...\c updated 2 variants ok annotate-tool.t8...\c updated 2 variants ok annotate-tool.t9...\c updated 2 variants ok annotate-tool.t10...\c updated 2 variants ok annotate-tool.t11...\c updated 2 variants ok annotate-tool.t12...\c updated 2 variants ok annotate-tool.t13...\c updated 2 variants ok annotate-tool.t14...\c updated 2 variants ok annotate-tool.t15...\c ok annotate-tool.t16...\c ok annotate-tool.t17...\c ok comp_het.t1...\c ok comp_het.t2...\c ok comp_het.t3...\c ok path.t01...\c ok path.t02...\c ok path.t03...\c ok path.t04...\c ok interact.t01...\c ok interact.t02...\c ok interact.t03...\c ok interact.t04...\c ok interact.t05...\c ok interact.t06...\c ok interact.t07...\c ok interact.t08...\c ok sieve.t01...\c ok sieve.t02...\c ok stat.t01...\c ok stat.t02...\c ok stat.t03...\c ok stat.t04...\c ok stat.t05...\c ok stat.t06...\c ok stat.t07...\c ok stat.t08...\c ok stat.t09...\c ok window.t01...\c ^CTraceback (most recent call last): File "/usr/local/bin/gemini", line 6, in gemini.gemini_main.main() File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 1056, in main args.func(parser, args) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 550, in windower_fn gemini_windower.windower(parser, args) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_windower.py", line 81, in windower make_windows(c, args, temp_file) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_windower.py", line 50, in make_windows s=args.step_size) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/pybedtools/bedtool.py", line 664, in decorated result = method(self, args, _kwargs) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/pybedtools/bedtool.py", line 243, in wrapped check_stderr=check_stderr) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/pybedtools/helpers.py", line 410, in call_bedtools stdout, stderr = p.communicate() File "/usr/local/share/gemini/anaconda/lib/python2.7/subprocess.py", line 794, in communicate stderr = _eintr_retry_call(self.stderr.read) File "/usr/local/share/gemini/anaconda/lib/python2.7/subprocess.py", line 476, in _eintr_retry_call return func(_args) KeyboardInterrupt 0a1,2 chr1 30000 40000 0.856 chr1 60000 70000 0.7865 fail window.t03...\c ^CTraceback (most recent call last): File "/usr/local/bin/gemini", line 6, in gemini.gemini_main.main() File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 1056, in main args.func(parser, args) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 550, in windower_fn gemini_windower.windower(parser, args) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_windower.py", line 81, in windower make_windows(c, args, temp_file) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_windower.py", line 50, in make_windows s=args.step_size) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/pybedtools/bedtool.py", line 664, in decorated result = method(self, args, _kwargs) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/pybedtools/bedtool.py", line 243, in wrapped check_stderr=check_stderr) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/pybedtools/helpers.py", line 410, in call_bedtools stdout, stderr = p.communicate() File "/usr/local/share/gemini/anaconda/lib/python2.7/subprocess.py", line 794, in communicate stderr = _eintr_retry_call(self.stderr.read) File "/usr/local/share/gemini/anaconda/lib/python2.7/subprocess.py", line 476, in _eintr_retry_call return func(_args) KeyboardInterrupt 0a1,2 chr1 30000 40000 0.505 chr1 60000 70000 0.1573 fail window.t04...\c ^CTraceback (most recent call last): File "/usr/local/bin/gemini", line 6, in gemini.gemini_main.main() File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 1056, in main args.func(parser, args) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 550, in windower_fn gemini_windower.windower(parser, args) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_windower.py", line 81, in windower make_windows(c, args, temp_file) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_windower.py", line 50, in make_windows s=args.step_size) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/pybedtools/bedtool.py", line 664, in decorated result = method(self, args, _kwargs) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/pybedtools/bedtool.py", line 243, in wrapped check_stderr=check_stderr) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/pybedtools/helpers.py", line 410, in call_bedtools stdout, stderr = p.communicate() File "/usr/local/share/gemini/anaconda/lib/python2.7/subprocess.py", line 794, in communicate stderr = _eintr_retry_call(self.stderr.read) File "/usr/local/share/gemini/anaconda/lib/python2.7/subprocess.py", line 476, in _eintr_retry_call return func(_args) KeyboardInterrupt 0a1,2 chr1 30000 40000 1.0 chr1 60000 70000 1.0 fail window.t05...\c ^CTraceback (most recent call last): File "/usr/local/bin/gemini", line 6, in gemini.gemini_main.main() File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 1056, in main args.func(parser, args) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 550, in windower_fn gemini_windower.windower(parser, args) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_windower.py", line 81, in windower make_windows(c, args, temp_file) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_windower.py", line 50, in make_windows s=args.step_size) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/pybedtools/bedtool.py", line 664, in decorated result = method(self, args, _kwargs) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/pybedtools/bedtool.py", line 243, in wrapped check_stderr=check_stderr) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/pybedtools/helpers.py", line 410, in call_bedtools stdout, stderr = p.communicate() File "/usr/local/share/gemini/anaconda/lib/python2.7/subprocess.py", line 794, in communicate stderr = _eintr_retry_call(self.stderr.read) File "/usr/local/share/gemini/anaconda/lib/python2.7/subprocess.py", line 476, in _eintr_retry_call return func(_args) KeyboardInterrupt 0a1,2 chr1 30000 40000 1.000000,1.000000,0.775097,0.504985,1.000000 chr1 60000 70000 1.000000,1.000000,0.157299,0.775097,1.000000 fail window.t06...\c ^CTraceback (most recent call last): File "/usr/local/bin/gemini", line 6, in gemini.gemini_main.main() File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 1056, in main args.func(parser, args) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 550, in windower_fn gemini_windower.windower(parser, args) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_windower.py", line 81, in windower make_windows(c, args, temp_file) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_windower.py", line 50, in make_windows s=args.step_size) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/pybedtools/bedtool.py", line 664, in decorated result = method(self, args, _kwargs) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/pybedtools/bedtool.py", line 243, in wrapped check_stderr=check_stderr) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/pybedtools/helpers.py", line 410, in call_bedtools stdout, stderr = p.communicate() File "/usr/local/share/gemini/anaconda/lib/python2.7/subprocess.py", line 794, in communicate stderr = _eintr_retry_call(self.stderr.read) File "/usr/local/share/gemini/anaconda/lib/python2.7/subprocess.py", line 476, in _eintr_retry_call return func(_args) KeyboardInterrupt 0a1,2 chr1 30000 31000 0.1357 chr1 69000 70000 0.1833 fail window.t08...\c ^CTraceback (most recent call last): File "/usr/local/bin/gemini", line 6, in gemini.gemini_main.main() File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 1056, in main args.func(parser, args) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 550, in windower_fn gemini_windower.windower(parser, args) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_windower.py", line 81, in windower make_windows(c, args, temp_file) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_windower.py", line 50, in make_windows s=args.step_size) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/pybedtools/bedtool.py", line 664, in decorated result = method(self, args, _kwargs) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/pybedtools/bedtool.py", line 243, in wrapped check_stderr=check_stderr) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/pybedtools/helpers.py", line 410, in call_bedtools stdout, stderr = p.communicate() File "/usr/local/share/gemini/anaconda/lib/python2.7/subprocess.py", line 794, in communicate stderr = _eintr_retry_call(self.stderr.read) File "/usr/local/share/gemini/anaconda/lib/python2.7/subprocess.py", line 476, in _eintr_retry_call return func(_args) KeyboardInterrupt 0a1,2 chr1 30000 31000 0.0 chr1 69000 70000 0.0 fail window.t09...\c ^CTraceback (most recent call last): File "/usr/local/bin/gemini", line 6, in gemini.gemini_main.main() File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 1056, in main args.func(parser, args) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 550, in windower_fn gemini_windower.windower(parser, args) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_windower.py", line 81, in windower make_windows(c, args, temp_file) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_windower.py", line 50, in make_windows s=args.step_size) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/pybedtools/bedtool.py", line 664, in decorated result = method(self, args, _kwargs) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/pybedtools/bedtool.py", line 243, in wrapped check_stderr=check_stderr) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/pybedtools/helpers.py", line 410, in call_bedtools stdout, stderr = p.communicate() File "/usr/local/share/gemini/anaconda/lib/python2.7/subprocess.py", line 794, in communicate stderr = _eintr_retry_call(self.stderr.read) File "/usr/local/share/gemini/anaconda/lib/python2.7/subprocess.py", line 476, in _eintr_retry_call return func(_args) KeyboardInterrupt 0a1,2 chr1 30000 31000 0.4286 chr1 69000 70000 0.6667 fail window.t10...\c ^CTraceback (most recent call last): File "/usr/local/bin/gemini", line 6, in gemini.gemini_main.main() File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 1056, in main args.func(parser, args) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 550, in windower_fn gemini_windower.windower(parser, args) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_windower.py", line 81, in windower make_windows(c, args, temp_file) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_windower.py", line 50, in make_windows s=args.step_size) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/pybedtools/bedtool.py", line 664, in decorated result = method(self, args, _kwargs) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/pybedtools/bedtool.py", line 243, in wrapped check_stderr=check_stderr) File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/pybedtools/helpers.py", line 410, in call_bedtools stdout, stderr = p.communicate() File "/usr/local/share/gemini/anaconda/lib/python2.7/subprocess.py", line 794, in communicate stderr = _eintr_retry_call(self.stderr.read) File "/usr/local/share/gemini/anaconda/lib/python2.7/subprocess.py", line 476, in _eintr_retry_call return func(*args) KeyboardInterrupt 0a1,2 chr1 30000 31000 0.000000,0.000000,0.250000,0.428571,0.000000 chr1 69000 70000 0.000000,0.000000,0.666667,0.250000,0.000000 fail domain.t01...\c ok gerp.t1...\c ok gerp.t2...\c ok auto_dom.t1...\c ok auto_dom.t2...\c ok auto_dom.t3...\c ok auto_dom.t4...\c ok auto_dom.t5...\c ok auto_dom.t6...\c ok auto_dom.t7...\c ok auto_dom.t8...\c ok auto_dom.t9...\c ok auto_dom.t10...\c ok auto_dom.t11...\c ok auto_dom.t12...\c ok auto_dom.t13...\c ok auto_dom.t14...\c ok auto_dom.t15...\c ok auto_dom.t16...\c ok auto_rec.t1...\c ok auto_rec.t2...\c ok auto_rec.t3...\c ok auto_rec.t4...\c ok auto_rec.t5...\c ok auto_rec.t6...\c ok auto_rec.t7...\c ok auto_rec.t8...\c ok auto_rec.t9...\c ok auto_rec.t10...\c ok auto_rec.t11...\c ok auto_rec.t12...\c ok auto_rec.t13...\c ok auto_rec.t14...\c ok auto_rec.t15...\c ok auto_rec.t16...\c ok de_novo.t1...\c ok de_novo.t2...\c ok de_novo.t3...\c ok wildcard.t1...\c ok wildcard.t2...\c ok wildcard.t3...\c ok wildcard.t4...\c ok wildcard.t5...\c ok wildcard.t6...\c ok wildcard.t7...\c ok wildcard.t8...\c ok wildcard.t9...\c ok wildcard.t10...\c ok wildcard.t11...\c ok wildcard.t12...\c ok wildcard.t13...\c ok wildcard.t14...\c ok wildcard.t15...\c ok wildcard.t16...\c ok wildcard.t17...\c ok wildcard.t18...\c ok wildcard.t19...\c ok wildcard.t20...\c ok wildcard.t21...\c ok wildcard.t22...\c ok roh.t1...\c LOG: Querying and ordering variants by chromosomal position. ok roh.t2...\c LOG: Querying and ordering variants by chromosomal position. ok roh.t3...\c LOG: Querying and ordering variants by chromosomal position. ok roh.t4...\c LOG: Querying and ordering variants by chromosomal position. ok roh.t5...\c LOG: Querying and ordering variants by chromosomal position. ok roh.t6...\c LOG: Querying and ordering variants by chromosomal position. ok roh.t7...\c LOG: Querying and ordering variants by chromosomal position. ok

[arq5x]

Sorry for the trouble. This is odd, as I am not able to reproduce it. It looks as though possibly bedtools is not installed on your system though it should be automatically installed by the gemini installer.

Can you check that you have bedtools installed?

[nicholasblackburn]

Thanks for responding, sure do: Nicholass-MacBook-Pro:gemini Nicholas$ bedtools --version bedtools v2.17.0

Nicholass-MacBook-Pro:gemini Nicholas$ bedtools bedtools: flexible tools for genome arithmetic and DNA sequence analysis. usage: bedtools [options]

The bedtools sub-commands include:

[ Genome arithmetic ] intersect Find overlapping intervals in various ways. window Find overlapping intervals within a window around an interval. closest Find the closest, potentially non-overlapping interval. coverage Compute the coverage over defined intervals. map Apply a function to a column for each overlapping interval. genomecov Compute the coverage over an entire genome. merge Combine overlapping/nearby intervals into a single interval. cluster Cluster (but don't merge) overlapping/nearby intervals. complement Extract intervals not represented by an interval file. subtract Remove intervals based on overlaps b/w two files. slop Adjust the size of intervals. flank Create new intervals from the flanks of existing intervals. sort Order the intervals in a file. random Generate random intervals in a genome. shuffle Randomly redistrubute intervals in a genome. annotate Annotate coverage of features from multiple files.

[ Multi-way file comparisons ] multiinter Identifies common intervals among multiple interval files. unionbedg Combines coverage intervals from multiple BEDGRAPH files.

[ Paired-end manipulation ] pairtobed Find pairs that overlap intervals in various ways. pairtopair Find pairs that overlap other pairs in various ways.

[ Format conversion ] bamtobed Convert BAM alignments to BED (& other) formats. bedtobam Convert intervals to BAM records. bamtofastq Convert BAM records to FASTQ records. bedpetobam Convert BEDPE intervals to BAM records. bed12tobed6 Breaks BED12 intervals into discrete BED6 intervals.

[ Fasta manipulation ] getfasta Use intervals to extract sequences from a FASTA file. maskfasta Use intervals to mask sequences from a FASTA file. nuc Profile the nucleotide content of intervals in a FASTA file.

[ BAM focused tools ] multicov Counts coverage from multiple BAMs at specific intervals. tag Tag BAM alignments based on overlaps with interval files.

[ Statistics tools ] jaccard Calculates the Jaccard statistic b/w two sets of intervals.

[ Miscellaneous tools ] overlap Computes the amount of overlap from two intervals. igv Create an IGV snapshot batch script. links Create a HTML page of links to UCSC locations. makewindows Make interval "windows" across a genome. groupby Group by common cols. & summarize oth. cols. (~ SQL "groupBy") expand Replicate lines based on lists of values in columns.

[ General help ] --help Print this help menu. --version What version of bedtools are you using?. --contact Feature requests, bugs, mailing lists, etc.

[nicholasblackburn]

I've tried on two VMs on the cluster that I have access to, using the install without root access approach. Same result.

[arq5x]

Okay, I can now replicate this after upgrading pybedtools on my system to version 0.6.8, which would be the version of pybedtools that would be automatically installed by the recent installations of gemini. This looks like it might be a new bug in pybedtools that is causing this. We will have to explore the cause. In the interim, I think you can safely ignore the errors and use GEMINI so long as you avoid the window tool.

cc @udp3f

[nicholasblackburn]

Awesome, thank you. So definitely shouldn't affect vcf file load?

(also, basically I've just got 96K variants VCF pre-filtered, i just want to add the ENCODE annotations, do I need to do VEP?)

[arq5x]

This will not affect the load at all. If you just want ENCODE annotations, you can skip VEP. Just know that doing so will also prevent having gene annotations.

[nicholasblackburn]

Thank you Aaron! Shall let you know if I hit problems, this is end stage of the analysis post ANNOVAR filtering, when I don't have a deadline looming i'll redo everything with GEMINI. Is best place to contact here?

[arq5x]

Sure, here is fine or the GEMINI mailing list.

[arq5x]

I just pushed a fix for this. All of the window tests should pass if you do:

$ gemini update --devel

[nicholasblackburn]

Works perfectly now - thanks!