search

dib-lab / dammit

just annotate it, dammit!
http://dib-lab.github.io/dammit/
Other
88 stars 28 forks source link

TaskError - taskid:remap_hmmer:longest_orfs.pep.pfam.tbl #46

Closed KField-Bucknell closed 8 years ago

KField-Bucknell commented 8 years ago

I am trying to run dammit on a clean Ubuntu install within VirtualBox on Mac OS 10.11. I am afraid that one of the tools didn't install properly -- I remember a lot of error codes being thrown. But unfortunately, I don't have them logged and I don't remember which step it was. I am going to try reinstalling Python and pandas to see if that helps. But I thought I would check here to see if someone else has another idea. Ken

--- Running annotate!

      Transcriptome file: Trinity-GG.fasta

      Output directory: /home/kfield/Documents/Trinity-GG.fasta.dammit

      [x] Trinity-GG.fasta

      [x] transcriptome_stats:Trinity-GG.fasta

      [ ] busco:Trinity-GG.fasta-eukaryota

      [ ] TransDecoder.LongOrfs:Trinity-GG.fasta

      [ ] hmmscan:longest_orfs.pep.x.Pfam-A.hmm

      [ ] remap_hmmer:longest_orfs.pep.pfam.tbl

      [ ] TransDecoder.Predict:Trinity-GG.fasta

      [ ] cmscan:Trinity-GG.fasta.x.Rfam.cm

      [x] lastal:Trinity-GG.fasta.x.orthodb.maf

      [ ] maf_best_hits:Trinity-GG.fasta.x.orthodb.maf-Trinity-
      GG.fasta.x.orthodb.maf.best.csv

      [ ] cmscan-gff3:Trinity-GG.fasta.rfam.tbl.gff3

Some tasks failed![dammit.annotate:ERROR] TaskError - taskid:remap_hmmer:longest_orfs.pep.pfam.tbl[dammit.annotate:ERROR] PythonAction Error Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/doit/action.py", line 383, in execute returned_value = self.py_callable(_self.args, _kwargs) File "/usr/local/lib/python2.7/dist-packages/dammit/tasks.py", line 472, in cmd assert(all(hmmer_df.env_coord_to <= merged_df.end)) AssertionError [dammit.annotate:ERROR] Some tasks failed![dammit.annotate:ERROR] TaskError - taskid:maf_best_hits:Trinity-GG.fasta.x.orthodb.maf-Trinity-GG.fasta.x.orthodb.maf.best.csv[dammit.annotate:ERROR] PythonAction Error Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/doit/action.py", line 383, in execute returned_value = self.py_callable(_self.args, _kwargs) File "/usr/local/lib/python2.7/dist-packages/dammit/tasks.py", line 305, in cmd parsers.maf_to_df_iter(maf_fn)]) File "/usr/local/lib/python2.7/dist-packages/pandas/tools/merge.py", line 812, in concat copy=copy) File "/usr/local/lib/python2.7/dist-packages/pandas/tools/merge.py", line 845, in init raise ValueError('No objects to concatenate') ValueError: No objects to concatenate [dammit.annotate:ERROR] Some tasks failed![dammit.annotate:ERROR] UnmetDependency - taskid:maf-gff3:Trinity-GG.fasta.x.orthodb.maf.gff3[dammit.annotate:ERROR] maf_best_hits:Trinity-GG.fasta.x.orthodb.maf-Trinity-GG.fasta.x.orthodb.maf.best.csv [dammit.annotate:ERROR] Some tasks failed![dammit.annotate:ERROR] UnmetDependency - taskid:hmmscan-gff3:Trinity-GG.fasta.pfam.csv.gff3[dammit.annotate:ERROR] remap_hmmer:longest_orfs.pep.pfam.tbl [dammit.annotate:ERROR] Some tasks failed![dammit.annotate:ERROR] UnmetDependency - taskid:gff3-merge:Trinity-GG.fasta.dammit.gff3[dammit.annotate:ERROR] hmmscan-gff3:Trinity-GG.fasta.pfam.csv.gff3 maf-gff3:Trinity-GG.fasta.x.orthodb.maf.gff3 [dammit.annotate:ERROR] Some tasks failed![dammit.annotate:ERROR] UnmetDependency - taskid:fasta-annotate:Trinity-GG.fasta.dammit.fasta[dammit.annotate:ERROR] gff3-merge:Trinity-GG.fasta.dammit.gff3 [dammit.annotate:ERROR]

abshah commented 8 years ago

Same Error here with the example datasets! `~~~~~~ submodule: annotate


--- Checking PATH for dependencies

          [x] TransDecoder

          [x] LAST

          [x] HMMER

          [x] Infernal

          [x] crb-blast

          [x] BLAST+

          [x] BUSCO

--- Dependency results

          All dependencies satisfied!

--- Checking for database prep (dir: /home/abhijeet/.dammit/databases)

          [x] download_and_gunzip:Pfam-A.hmm

          [x] hmmpress:Pfam-A.hmm

          [x] download_and_gunzip:Rfam.cm

          [x] cmpress:Rfam.cm

          [x] download_and_gunzip:aa_seq_euk.fasta

          [x] lastdb:aa_seq_euk.fasta.db

          [x] download_and_gunzip:ODB8_EukOGs_genes_ALL_levels.txt

          [x] download_and_untar:databases-eukaryota

--- Database results

          All databases prepared!

--- Running annotate!

          Transcriptome file: cdna_nointrons_utrs.fa

          Output directory: /home/abhijeet/dammit_test/cdna_nointrons_utrs.fa.dammit

          [x] cdna_nointrons_utrs.fa

          [x] transcriptome_stats:cdna_nointrons_utrs.fa

          [x] busco:cdna_nointrons_utrs.fa-eukaryota

          [x] transeq:cdna_nointrons_utrs.fa

          [x] TransDecoder.LongOrfs:cdna_nointrons_utrs.fa

          [x] hmmscan:longest_orfs.pep.x.Pfam-A.hmm

          [ ] remap_hmmer:longest_orfs.pep.pfam.tbl

          [x] TransDecoder.Predict:cdna_nointrons_utrs.fa

          [x] cmscan:cdna_nointrons_utrs.fa.x.Rfam.cm

          [x] lastal:cdna_nointrons_utrs.fa.x.orthodb.maf

          [x] sanitize_fasta:pep.fa

          [x] crb-blast:cdna_nointrons_utrs.fa.x.pep.fa

          [x] maf_best_hits:cdna_nointrons_utrs.fa.x.orthodb.maf-
          cdna_nointrons_utrs.fa.x.orthodb.maf.best.csv

          [x] maf-gff3:cdna_nointrons_utrs.fa.x.orthodb.maf.gff3

          [x] cmscan-gff3:cdna_nointrons_utrs.fa.rfam.tbl.gff3

          [ ] crbb-gff3:cdna_nointrons_utrs.fa.x.pep.fa.crbb.tsv.gff3

Some tasks failed![dammit.annotate:ERROR]
TaskError - taskid:remap_hmmer:longest_orfs.pep.pfam.tbl[dammit.annotate:ERROR]
PythonAction Error
Traceback (most recent call last):
  File "/home/abhijeet/anaconda2/lib/python2.7/site-packages/doit/action.py", line 383, in execute
    returned_value = self.py_callable(*self.args, **kwargs)
  File "/home/abhijeet/anaconda2/lib/python2.7/site-packages/dammit/tasks.py", line 464, in cmd
    parsers.hmmscan_to_df_iter(hmmer_filename)])
  File "/home/abhijeet/anaconda2/lib/python2.7/site-packages/dammit/parsers.py", line 358, in hmmscan_to_df_iter
    yield build_df(data)
  File "/home/abhijeet/anaconda2/lib/python2.7/site-packages/dammit/parsers.py", line 339, in build_df
    convert_dtypes(df, dict(hmmscan_cols))
  File "/home/abhijeet/anaconda2/lib/python2.7/site-packages/dammit/parsers.py", line 103, in convert_dtypes
    df[c] = df[c].astype(dtypes[c])
  File "/home/abhijeet/anaconda2/lib/python2.7/site-packages/pandas/core/generic.py", line 2518, in astype
    dtype=dtype, copy=copy, raise_on_error=raise_on_error, **kwargs)
  File "/home/abhijeet/anaconda2/lib/python2.7/site-packages/pandas/core/internals.py", line 2755, in astype
    return self.apply('astype', dtype=dtype, **kwargs)
  File "/home/abhijeet/anaconda2/lib/python2.7/site-packages/pandas/core/internals.py", line 2710, in apply
    applied = getattr(b, f)(**kwargs)
  File "/home/abhijeet/anaconda2/lib/python2.7/site-packages/pandas/core/internals.py", line 409, in astype
    values=values, **kwargs)
  File "/home/abhijeet/anaconda2/lib/python2.7/site-packages/pandas/core/internals.py", line 443, in _astype
    values = com._astype_nansafe(values.ravel(), dtype, copy=True)
  File "/home/abhijeet/anaconda2/lib/python2.7/site-packages/pandas/core/common.py", line 2773, in _astype_nansafe
    return lib.astype_intsafe(arr.ravel(), dtype).reshape(arr.shape)
  File "pandas/lib.pyx", line 910, in pandas.lib.astype_intsafe (pandas/lib.c:16221)
  File "pandas/src/util.pxd", line 60, in util.set_value_at (pandas/lib.c:67123)
TypeError: long() argument must be a string or a number, not 'NoneType'
[dammit.annotate:ERROR]
Some tasks failed![dammit.annotate:ERROR]
UnmetDependency - taskid:hmmscan-gff3:cdna_nointrons_utrs.fa.pfam.csv.gff3[dammit.annotate:ERROR]
remap_hmmer:longest_orfs.pep.pfam.tbl
[dammit.annotate:ERROR]
Some tasks failed![dammit.annotate:ERROR]
TaskError - taskid:crbb-gff3:cdna_nointrons_utrs.fa.x.pep.fa.crbb.tsv.gff3[dammit.annotate:ERROR]
PythonAction Error
Traceback (most recent call last):
  File "/home/abhijeet/anaconda2/lib/python2.7/site-packages/doit/action.py", line 383, in execute
    returned_value = self.py_callable(*self.args, **kwargs)
  File "/home/abhijeet/anaconda2/lib/python2.7/site-packages/dammit/tasks.py", line 541, in cmd
    remap=True):
  File "/home/abhijeet/anaconda2/lib/python2.7/site-packages/dammit/parsers.py", line 263, in crb_to_df_iter
    remap_blast(group)
  File "/home/abhijeet/anaconda2/lib/python2.7/site-packages/dammit/blast.py", line 68, in remap_blast_coords_df
    df['sstrand'] = coords[coords:,4]
IndexError: invalid slice
[dammit.annotate:ERROR]
Some tasks failed![dammit.annotate:ERROR]
UnmetDependency - taskid:gff3-merge:cdna_nointrons_utrs.fa.dammit.gff3[dammit.annotate:ERROR]
hmmscan-gff3:cdna_nointrons_utrs.fa.pfam.csv.gff3 crbb-gff3:cdna_nointrons_utrs.fa.x.pep.fa.crbb.tsv.gff3
[dammit.annotate:ERROR]
Some tasks failed![dammit.annotate:ERROR]
UnmetDependency - taskid:fasta-annotate:cdna_nointrons_utrs.fa.dammit.fasta[dammit.annotate:ERROR]
gff3-merge:cdna_nointrons_utrs.fa.dammit.gff3
[dammit.annotate:ERROR]`
macmanes commented 8 years ago

same here.

I failed using .2.6, updated to .2.6.1 still failing. 

--- Running annotate!

...
          Transcriptome file: Rimi.v1.02.fasta

          Output directory: /mnt/data3/macmanes/imitator/Jan16/imidammit

          [x] Rimi.v1.02.fasta

          [x] transcriptome_stats:Rimi.v1.02.fasta

          [x] busco:Rimi.v1.02.fasta-vertebrata

          [x] hmmscan:longest_orfs.pep.x.Pfam-A.hmm

          [ ] remap_hmmer:longest_orfs.pep.pfam.tbl

          [x] TransDecoder.Predict:Rimi.v1.02.fasta

          [x] cmscan:Rimi.v1.02.fasta.x.Rfam.cm

          [x] lastal:Rimi.v1.02.fasta.x.orthodb.maf

          [x] lastal:Rimi.v1.02.fasta.x.uniref.maf

          [x] maf_best_hits:Rimi.v1.02.fasta.x.orthodb.maf-
          Rimi.v1.02.fasta.x.orthodb.maf.best.csv

          [x] maf-gff3:Rimi.v1.02.fasta.x.orthodb.maf.gff3

          [x] maf_best_hits:Rimi.v1.02.fasta.x.uniref.maf-
          Rimi.v1.02.fasta.x.uniref.maf.best.csv

          [x] maf-gff3:Rimi.v1.02.fasta.x.uniref.maf.best.csv.gff3

          [x] cmscan-gff3:Rimi.v1.02.fasta.rfam.tbl.gff3

Some tasks failed![dammit.annotate:ERROR]
TaskError - taskid:remap_hmmer:longest_orfs.pep.pfam.tbl[dammit.annotate:ERROR]
PythonAction Error
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/doit/action.py", line 383, in execute
    returned_value = self.py_callable(*self.args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/dammit/tasks.py", line 472, in cmd
    assert(all(hmmer_df.env_coord_to <= merged_df.end))
AssertionError
[dammit.annotate:ERROR]
Some tasks failed![dammit.annotate:ERROR]
UnmetDependency - taskid:hmmscan-gff3:Rimi.v1.02.fasta.pfam.csv.gff3[dammit.annotate:ERROR]
remap_hmmer:longest_orfs.pep.pfam.tbl
[dammit.annotate:ERROR]
Some tasks failed![dammit.annotate:ERROR]
UnmetDependency - taskid:gff3-merge:Rimi.v1.02.fasta.dammit.gff3[dammit.annotate:ERROR]
hmmscan-gff3:Rimi.v1.02.fasta.pfam.csv.gff3
[dammit.annotate:ERROR]
Some tasks failed![dammit.annotate:ERROR]
UnmetDependency - taskid:fasta-annotate:Rimi.v1.02.fasta.dammit.fasta[dammit.annotate:ERROR]
gff3-merge:Rimi.v1.02.fasta.dammit.gff3
[dammit.annotate:ERROR]
camillescott commented 8 years ago

sigh it appears my closing the previous issue was premature. Wrapping this problem up once and for all is on my to-do for the next two days. Thanks for filing issues folks!

macmanes commented 8 years ago

Thanks Camille. I'm poised to fire up a run as soon as fixed!

camillescott commented 8 years ago

Alright folks. This issue should be finally, officially, resolved! As per usual:

pip install --upgrade --no-deps dammit

For further reading, refer to https://github.com/camillescott/dammit/blob/master/dammit/parsers.py#L10-L16 for this developers opinion of bioinformatics coordinate systems.

abshah commented 8 years ago

Hi Camille, I just tried running the example datasets and ended up with a slightly different error!

`--- Running annotate!

      Transcriptome file: cdna_nointrons_utrs.fa

      Output directory: /home/abhijeet/dammit_test/cdna_nointrons_utrs.fa.dammit

      [ ] cdna_nointrons_utrs.fa

      [ ] transcriptome_stats:cdna_nointrons_utrs.fa

      [ ] busco:cdna_nointrons_utrs.fa-eukaryota

      [ ] TransDecoder.LongOrfs:cdna_nointrons_utrs.fa

      [ ] hmmscan:longest_orfs.pep.x.Pfam-A.hmm

      [ ] remap_hmmer:longest_orfs.pep.pfam.tbl

      [ ] TransDecoder.Predict:cdna_nointrons_utrs.fa

      [ ] cmscan:cdna_nointrons_utrs.fa.x.Rfam.cm

      [ ] lastal:cdna_nointrons_utrs.fa.x.orthodb.maf

      [ ] sanitize_fasta:pep.fa

      [ ] crb-blast:cdna_nointrons_utrs.fa.x.pep.fa

      [ ] maf_best_hits:cdna_nointrons_utrs.fa.x.orthodb.maf-
      cdna_nointrons_utrs.fa.x.orthodb.maf.best.csv

      [ ] maf-gff3:cdna_nointrons_utrs.fa.x.orthodb.maf.gff3

      [ ] hmmscan-gff3:cdna_nointrons_utrs.fa.pfam.csv.gff3

      [ ] cmscan-gff3:cdna_nointrons_utrs.fa.rfam.tbl.gff3

      [ ] crbb-gff3:cdna_nointrons_utrs.fa.x.pep.fa.crbb.tsv.gff3

Some tasks failed![dammit.annotate:ERROR] TaskError - taskid:crbb-gff3:cdna_nointrons_utrs.fa.x.pep.fa.crbb.tsv.gff3[dammit.annotate:ERROR] PythonAction Error Traceback (most recent call last): File "/home/abhijeet/anaconda2/lib/python2.7/site-packages/doit/action.py", line 383, in execute returned_value = self.py_callable(_self.args, *_kwargs) File "/home/abhijeet/anaconda2/lib/python2.7/site-packages/dammit/tasks.py", line 539, in cmd remap=True): File "/home/abhijeet/anaconda2/lib/python2.7/site-packages/dammit/parsers.py", line 284, in crb_to_df_iter remap_blast(group) File "/home/abhijeet/anaconda2/lib/python2.7/site-packages/dammit/blast.py", line 68, in remap_blast_coords_df df['sstrand'] = coords[coords:,4] IndexError: invalid slice [dammit.annotate:ERROR] Some tasks failed![dammit.annotate:ERROR] UnmetDependency - taskid:gff3-merge:cdna_nointrons_utrs.fa.dammit.gff3[dammit.annotate:ERROR] crbb-gff3:cdna_nointrons_utrs.fa.x.pep.fa.crbb.tsv.gff3 [dammit.annotate:ERROR] Some tasks failed![dammit.annotate:ERROR] UnmetDependency - taskid:fasta-annotate:cdna_nointrons_utrs.fa.dammit.fasta[dammit.annotate:ERROR] gff3-merge:cdna_nointrons_utrs.fa.dammit.gff3 [dammit.annotate:ERROR]`

macmanes commented 8 years ago

FWIW, I had a successful run post update..

camillescott commented 8 years ago

@abshah: this is a different problem, so I've open a new issue for it over at #51

@macmanes: good to hear -- you should find the annotation intervals to be a tad truthier as well ;)