search

vpc-ccg / pamir

Discovery and Genotyping of Novel Sequence Insertions in Many Sequenced Individuals
BSD 3-Clause "New" or "Revised" License
8 stars 4 forks source link

InputFunctionException #40

Closed christinafliege closed 4 years ago

christinafliege commented 4 years ago

I'm getting the following python error when attempting to run pamir. Thanks!

InputFunctionException in line 1424 of path/to/Snakefile: KeyError: 'DataFilePartialName' Wildcards: sample=DataFilePartialName

f0t1h commented 4 years ago

Can you please post the config file you've used and full log of the failed run.

christinafliege commented 4 years ago

Thank you, my config file is:

path: /projects/mgc/Project_2/HLHS_BasilAniseVC/Pamir raw-data: /matt_k_inputs reference: /projects/mgc/Project_2/HLHS_BasilAniseVC/Pamir/inputs/Basilexamples/ref.1.fa population: basilexamp input: "basil":

The full error file is:

/usr/local/apps/bioapps/python/Python-3.6.1/bin/snakemake ... Exists :) /usr/local/apps/bioapps/samtools/samtools-1.9/samtools ... Exists :) /usr/local/apps/bioapps/bedtools/bedtools-2.28.0/bin/bedtools ... Exists :) /usr/local/apps/bioapps/mrsfast/mrsfast-3.4.1//mrsfast ... Exists :) /usr/local/apps/bioapps/bwa/bwa-0.7.17-compiler_intel-16.0/bwa ... Exists :) Snakemake version...OK Samtools version...OK MrsFAST version...OK Bedtools version...OK Building DAG of jobs... InputFunctionException in line 1424 of /usr/local/apps/bioapps/pamir/pamir/pamir/Snakefile: KeyError: 'Basilexamps' Wildcards: sample=Basilexamps

Additionally, when I remove the slash before the raw data field in the config file I get the following error file:

/usr/local/apps/bioapps/python/Python-3.6.1/bin/snakemake ... Exists :) /usr/local/apps/bioapps/samtools/samtools-1.9/samtools ... Exists :) /usr/local/apps/bioapps/bedtools/bedtools-2.28.0/bin/bedtools ... Exists :) /usr/local/apps/bioapps/mrsfast/mrsfast-3.4.1//mrsfast ... Exists :) /usr/local/apps/bioapps/bwa/bwa-0.7.17-compiler_intel-16.0/bwa ... Exists :) Snakemake version...OK Samtools version...OK MrsFAST version...OK Bedtools version...OK Building DAG of jobs... MissingInputException in line 1344 of /usr/local/apps/bioapps/pamir/pamir/pamir/Snakefile: Missing input files for rule link_bam: /projects/mgc/Project_2/HLHS_BasilAniseVC/Pamirmatt_k_inputs/Basilexamps.bam

f0t1h commented 4 years ago

We added bam support and many other updates to pamir with 2.1.0 release. Can you try again with a clean install of the 2.1.0? You can also see the small example in the README.

mortunco commented 4 years ago

Dear @christinafliege,

Could you use insert your config.yaml or output from the command line in to ``` hello ``` so it wont be formatted by markdown?

for ex:

hello

Thanks.

christinafliege commented 4 years ago 

  /projects/mgc/Project_2/HLHS_BasilAniseVC/Pamir
raw-data:
  /matt_k_inputs
reference:
  /projects/mgc/Project_2/HLHS_BasilAniseVC/Pamir/inputs/Basilexamples/ref.1.fa
population:
  basilexamp
input:
  "basil":
  - Basilexamps.bam
centromeres:
  /projects/mgc/Project_2/HLHS_BasilAniseVC/Pamir/inputs/centromeres.txt

/usr/local/apps/bioapps/python/Python-3.6.1/bin/snakemake
... Exists :)
/usr/local/apps/bioapps/samtools/samtools-1.9/samtools
... Exists :)
/usr/local/apps/bioapps/bedtools/bedtools-2.28.0/bin/bedtools
... Exists :)
/usr/local/apps/bioapps/mrsfast/mrsfast-3.4.1//mrsfast
... Exists :)
/usr/local/apps/bioapps/bwa/bwa-0.7.17-compiler_intel-16.0/bwa
... Exists :)
Snakemake version...OK
Samtools version...OK
MrsFAST version...OK
Bedtools version...OK
Building DAG of jobs...
InputFunctionException in line 1424 of /usr/local/apps/bioapps/pamir/pamir/pamir/Snakefile:
KeyError: 'Basilexamps'
Wildcards:
sample=Basilexamps
christinafliege commented 4 years ago

We added bam support and many other updates to pamir with 2.1.0 release. Can you try again with a clean install of the 2.1.0? You can also see the small example in the README.

we are currently running 5.17 which was freshly installed for this project. Thanks!

f0t1h commented 4 years ago

I mean the pamir version.

joshfactorial commented 4 years ago

I'm one of the people who installed Pamir for Christina. I found version 2.1.0 and will try a fresh install.

joshfactorial commented 4 years ago

Right, so I found version 2.1.0 in releases, downloaded it to our cluster, extracted and ran make and got the following:

$ make 
make[1]: Entering directory '/usr/local/apps/bioapps/pamir/pamir-2.1.0'
Please clone the repository with --recursive option!
Makefile:114: recipe for target 'ext/edlib/edlib/include/edlib.h' failed
make[1]: *** [ext/edlib/edlib/include/edlib.h] Error 1
make[1]: Leaving directory '/usr/local/apps/bioapps/pamir/pamir-2.1.0'
Makefile:60: recipe for target 'release' failed
make: *** [release] Error 2

Originally I just cloned the repo and did it that way. I tried looking for ext/edlib/edlib/include/edlib.h and it does not appear to be present

fhach commented 4 years ago

I assume you downloaded the tarball. Please clone recursively so it can download its submodules. 2.1.0 is currently master but you can easily add it to recursive command.

git clone https://github.com/vpc-ccg/pamir.git --branch 2.1.0 --recursive
joshfactorial commented 4 years ago

I reinstalled using git clone https://github.com/vpc-ccg/pamir.git --branch 2.1.0 --recursive and it is still giving is the InputFunctionException error.

f0t1h commented 4 years ago

Can you confirm the input bam exists at the path:

 /projects/mgc/Project_2/HLHS_BasilAniseVC/Pamir/matt_k_inputs/Basilexamps.bam
joshfactorial commented 4 years ago

Yes, that is the location and samtools view shows that it has reads.

f0t1h commented 4 years ago

Can you post the latest error log?

joshfactorial commented 4 years ago 
/usr/local/apps/bioapps/python/Python-3.6.1/bin/snakemake
... Exists :)
/usr/local/apps/bioapps/samtools/samtools-1.9/samtools
... Exists :)
/usr/local/apps/bioapps/bedtools/bedtools-2.28.0/bin/bedtools
... Exists :)
/usr/local/apps/bioapps/mrsfast/mrsfast-3.4.1//mrsfast
... Exists :)
/usr/local/apps/bioapps/bwa/bwa-0.7.17-compiler_intel-16.0/bwa
... Exists :)
Snakemake version...OK
Samtools version...OK
MrsFAST version...OK
Bedtools version...OK
Building DAG of jobs...
InputFunctionException in line 1428 of /usr/local/apps/bioapps/pamir/pamir-2.1.0/pamir/Snakefile:
KeyError: 'Basilexamps'
Wildcards:
sample=Basilexamps
fhach commented 4 years ago

I think we managed to replicate the error and we narrowed down it to the following line. Can you please replace this in your config file. We would like to verify this while we fixing the code to work with your original config file.

input:
  "Basilexamps":
    - Basilexamps.bam
fhach commented 4 years ago

@joshfactorial any update on this? We have a fix on our dev branch that we would like to merge to master to address this issue more broadly. I would like to confirm that this config line was causing the crash.

christinafliege commented 4 years ago

When we changed the config file inputs like above it appeared to start working, however then we ran into other issues. It does appear that this is correct and this input change fixed that issue.

fhach commented 4 years ago

That is good. We will merge the new snakemake rules into the master branch today. Please open up another issue with your log and we can quickly check it out. We will close this issue upon merging to master.

joshfactorial commented 4 years ago

We're making progress now on troubleshooting. Will post if we hit any other roadblocks. Thanks for the help!