search

PacificBiosciences / kineticsTools

Tools for detecting DNA modifications from single molecule, real-time sequencing data
19 stars 21 forks source link

ipdSummary: UnboundLocalError: local variable 'mods' referenced before assignment #77

Open nick-youngblut opened 4 years ago

nick-youngblut commented 4 years ago

I am running ipdSummary as such:

ipdSummary --identify m6A,m4C \
  --methylFraction    --numWorkers 8   --maxCoverage 100    \
  --log-file tests/ipdSummary/subreads/cov100/MT21.log \
  --gff tests/pdSummary/subreads/cov100/MT21.gff \
  --csv tests/ipdSummary/subreads/cov100/MT21.csv \
  --reference reference.fasta           
  MT21.bam

and am getting the following uninformative error: 

Traceback (most recent call last):
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/KineticWorker.py", line 141, in onChunk
    perSiteResults, (start, end))
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/KineticWorker.py", line 317, in _decodePositiveControl
    for (r, mod) in decoder.decode().items():
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ModificationDecode.py", line 49, in decode
    return self.scoreMods(modCalls)
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ModificationDecode.py", line 257, in scoreMods
    pos - self.post, pos + self.pre, modSeq)
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/MultiSiteCommon.py", line 150, in getContextMeans
    meanVector.append(self.gbmModel.getPredictions([ctx]))
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ipdModel.py", line 273, in getPredictions
    modBits = baseToCode[s[i]]
KeyError: 67
> /ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ipdModel.py(273)getPredictions()
-> modBits = baseToCode[s[i]]
(Pdb)
Process KineticWorkerProcess-5:
Traceback (most recent call last):
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/WorkerProcess.py", line 151, in run
    self._run()
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/WorkerProcess.py", line 128, in _run
    datum)  # pylint: disable=assignment-from-none
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/KineticWorker.py", line 155, in onChunk
    modDict = dict((x['tpl'], x) for x in mods if start <=
UnboundLocalError: local variable 'mods' referenced before assignment
Traceback (most recent call last):
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/KineticWorker.py", line 141, in onChunk
    perSiteResults, (start, end))
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/KineticWorker.py", line 339, in _decodePositiveControl
    for (r, mod) in decoder.decode().items():
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ModificationDecode.py", line 49, in decode
    return self.scoreMods(modCalls)
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ModificationDecode.py", line 257, in scoreMods
    pos - self.post, pos + self.pre, modSeq)
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/MultiSiteCommon.py", line 150, in getContextMeans
    meanVector.append(self.gbmModel.getPredictions([ctx]))
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ipdModel.py", line 273, in getPredictions
    modBits = baseToCode[s[i]]
KeyError: 67
> /ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ipdModel.py(273)getPredictions()
-> modBits = baseToCode[s[i]]
(Pdb)
Process KineticWorkerProcess-2:
Traceback (most recent call last):
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/WorkerProcess.py", line 151, in run
    self._run()
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/WorkerProcess.py", line 128, in _run
    datum)  # pylint: disable=assignment-from-none
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/KineticWorker.py", line 155, in onChunk
    modDict = dict((x['tpl'], x) for x in mods if start <=
UnboundLocalError: local variable 'mods' referenced before assignment
Traceback (most recent call last):
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/KineticWorker.py", line 141, in onChunk
    perSiteResults, (start, end))
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/KineticWorker.py", line 317, in _decodePositiveControl
    for (r, mod) in decoder.decode().items():
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ModificationDecode.py", line 49, in decode
    return self.scoreMods(modCalls)
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ModificationDecode.py", line 257, in scoreMods
    pos - self.post, pos + self.pre, modSeq)
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/MultiSiteCommon.py", line 150, in getContextMeans
    meanVector.append(self.gbmModel.getPredictions([ctx]))
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ipdModel.py", line 273, in getPredictions
    modBits = baseToCode[s[i]]
KeyError: 65
Traceback (most recent call last):
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/KineticWorker.py", line 141, in onChunk
    perSiteResults, (start, end))
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/KineticWorker.py", line 317, in _decodePositiveControl
    for (r, mod) in decoder.decode().items():
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ModificationDecode.py", line 49, in decode
    return self.scoreMods(modCalls)
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ModificationDecode.py", line 257, in scoreMods
    pos - self.post, pos + self.pre, modSeq)
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/MultiSiteCommon.py", line 150, in getContextMeans
    meanVector.append(self.gbmModel.getPredictions([ctx]))
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ipdModel.py", line 273, in getPredictions
    modBits = baseToCode[s[i]]
KeyError: 84
> /ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ipdModel.py(273)getPredictions()
-> modBits = baseToCode[s[i]]
(Pdb) > /ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ipdModel.py(273)getPredictions()
-> modBits = baseToCode[s[i]]
(Pdb)
Process KineticWorkerProcess-8:
Traceback (most recent call last):
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/WorkerProcess.py", line 151, in run
    self._run()
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/WorkerProcess.py", line 128, in _run
    datum)  # pylint: disable=assignment-from-none
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/KineticWorker.py", line 155, in onChunk
    modDict = dict((x['tpl'], x) for x in mods if start <=
UnboundLocalError: local variable 'mods' referenced before assignment

Process KineticWorkerProcess-4:
Traceback (most recent call last):
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/WorkerProcess.py", line 151, in run
    self._run()
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/WorkerProcess.py", line 128, in _run
    datum)  # pylint: disable=assignment-from-none
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/KineticWorker.py", line 155, in onChunk
    modDict = dict((x['tpl'], x) for x in mods if start <=
UnboundLocalError: local variable 'mods' referenced before assignment
Traceback (most recent call last):
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/KineticWorker.py", line 141, in onChunk
    perSiteResults, (start, end))
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/KineticWorker.py", line 339, in _decodePositiveControl
    for (r, mod) in decoder.decode().items():
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ModificationDecode.py", line 49, in decode
    return self.scoreMods(modCalls)
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ModificationDecode.py", line 257, in scoreMods
    pos - self.post, pos + self.pre, modSeq)
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/MultiSiteCommon.py", line 150, in getContextMeans
    meanVector.append(self.gbmModel.getPredictions([ctx]))
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ipdModel.py", line 273, in getPredictions
    modBits = baseToCode[s[i]]
KeyError: 67
> /ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ipdModel.py(273)getPredictions()
-> modBits = baseToCode[s[i]]
(Pdb)
Process KineticWorkerProcess-7:
Traceback (most recent call last):
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/WorkerProcess.py", line 151, in run
    self._run()
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/WorkerProcess.py", line 128, in _run
    datum)  # pylint: disable=assignment-from-none
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/KineticWorker.py", line 155, in onChunk
    modDict = dict((x['tpl'], x) for x in mods if start <=
UnboundLocalError: local variable 'mods' referenced before assignment
Child process exited with exitcode=1.  Aborting.

I've installed ipdSummary via:

wget https://downloads.pacbcloud.com/public/software/installers/smrtlink_9.0.0.92188.zip && unzip *.zip
 ./smrtlink_9.0.0.92188.run --rootdir  /path/to/install/location/ --install --smrttools-only

The amount of input sequence data is fairly low, which might be causing the error, but still, the error should then be more informative if that is the case.

natechols commented 4 years ago

Yeah, I suspect it's ending up with an uninitialized variable because of insufficient coverage, but still a bug in any case. I'll take a look.

GDelevoye commented 4 years ago

Quick suggestion here: The standard deviation can only be computed if coverage >= 3

nick-youngblut commented 4 years ago

@GDelevoye that doesn't explain why I get the error when using --maxCoverage 100 but not when using --maxCoverage 1

natechols commented 4 years ago

@nick-youngblut Looks like the --methylFraction option got broken in the process of the py3 porting. Sorry! I'll post a diff here once I have a fix.

natechols commented 4 years ago

@nick-youngblut Try this patch:

diff --git a/kineticsTools/MultiSiteCommon.py b/kineticsTools/MultiSiteCommon.py
index 96d06e8..aed1350 100644
--- a/kineticsTools/MultiSiteCommon.py
+++ b/kineticsTools/MultiSiteCommon.py
@@ -143,7 +143,7 @@ class MultiSiteCommon(object):
     def getContextMeans(self, start, end, sequence):
         meanVector = []
         for pos in range(start, end + 1):
-            ctx = sequence[(pos - self.pre):(pos + self.post + 1)].tostring()
+            ctx = sequence[(pos - self.pre):(pos + self.post + 1)].tostring().decode("ascii")
             if ctx in self.contextMeanTable:
                 meanVector.append(self.contextMeanTable[ctx])
             else:

That's in the file /ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/MultiSiteCommon.py. (It fixes the crash using test inputs, but I haven't tried with real data yet.)