KeyError: 'score' during lift over

elcortegano commented 3 years ago

I got the following error when running TELR. It seems to occur during a lift over step. However, it is not clear to me what can I do to fix it.

...
Successfully created the directory /home/elcortegano/TELR/test/intermediate_files/telr_reads 
Starting lift over workflow...
Generating flanking sequences...
Align flanking sequence to reference...
Parsing flanking sequence alignments...
Traceback (most recent call last):
  File "/home/elcortegano/anaconda3/envs/TELR_env/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2646, in get_loc
    return self._engine.get_loc(key)
  File "pandas/_libs/index.pyx", line 111, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1622, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'score'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./telr.py", line 149, in <module>
    main()
  File "./telr.py", line 134, in main
    loci_eval,
  File "/home/elcortegano/TELR/TELR_te.py", line 570, in find_te
    loci_eval=loci_eval,
  File "/home/elcortegano/TELR/TELR_liftover.py", line 303, in annotation_liftover
    | (new_df["score"] == 0.5)
  File "/home/elcortegano/anaconda3/envs/TELR_env/lib/python3.6/site-packages/pandas/core/frame.py", line 2800, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/home/elcortegano/anaconda3/envs/TELR_env/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2648, in get_loc
    return self._engine.get_loc(self._maybe_cast_indexer(key))
  File "pandas/_libs/index.pyx", line 111, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1622, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'score'

The script was run as:

./telr.py -o test -i file.bam -r reference.fa -l TEs.fa

shunhuahan commented 3 years ago

@elcortegano Thanks for reporting this issue. This should be a bug related to error handing when TELR makes zero predictions. I will make an update to fix the issue shortly and notify you on that.
In the meantime, you might want to run telr with -k option to keep intermediate files and check intermediate_files/$PREFIX.vcf. This file includes all the SVs that are detected by Sniffles (https://github.com/fritzsedlazeck/Sniffles). You can also check intermediate_files/$PREFIX.vcf_filtered.tsv (if present), which should contain all the Sniffles insertions that include TE consensus sequences in your library.

shunhuahan commented 3 years ago

@elcortegano I've made an update on TELR, which now should be able to handle the KeyError issue properly. Basically, TELR would report No non-reference TE insertion found if no prediction can be made instead of throwing out an error. Let me know if the new update works for you.

elcortegano commented 3 years ago

Thank you for the update! using the latest code (commit a097776d23b5866bbe6a012995813d0fc1579c9a) the script finish running successfully.

I have a doubt about the meaning of this No non-reference TE insertion found message, and the absence of regular output files (e.g. the output VCF). What can be the cause of this? we have some evidence of TE activity in our data (originally found with sniffles), and these TEs are included in the library.

shunhuahan commented 3 years ago

@elcortegano The zero prediction by TELR could be caused by issues such as: 1) low coverage (ep. average sequencing coverage less than 10X). 2) insertions in highly repetitive regions or SV regions. 4) low quality TE consensus sequences. 5) low quality reference genome (or reference too diverged from the sequenced sample).
It's hard to determine the cause of this issue and figure out ways to improve sensitivity without checking some details in your run. Could you send over the intermediate_files/$prefix.vcf (Sniffles VCF file), intermediate_files/$contigs.fa (assembled local contigs), TE library and reference genome you were using? My email address is shhan@uga.edu. You could also post the accession ID if the raw dataset is public. Thanks.

cbergman commented 2 years ago

@elcortegano: did you resolve this issue? If no, could you please do a clean install and try again? If yes, I'd like to close this issue. Thanks!

elcortegano commented 2 years ago

Yes!, it works, thanks for the support.

bergmanlab / TELR

KeyError: 'score' during lift over #3