eseraygun / python-alignment

Native Python library for generic sequence alignment
BSD 3-Clause "New" or "Revised" License
55 stars 14 forks source link

Unlimited backtrace is impractical #9

Open bertsky opened 5 years ago

bertsky commented 5 years ago

When globally aligning sequences that deviate much, combinatory explosion can quickly leed to excessive runtime memory consumption in the current implementation. And it is not always easy to detect those cases by score heuristics in a prior backtrace=False pass.

I believe these should be added:

  1. a package-exposed variable with a default limit (perhaps relative to the sequences' length)
  2. an optional parameter with an override limit to be able to control the quality-performance trade-off.

(The limit could be based on stack depth or number of alternatives, for example.)

Example: I am trying to align OCRed images of German Fraktur script with their corresponding ground truth text. Sometimes the OCR fails miserably like so: Mitreden andrer 274. Günſtiger Eindruck der Staatsrathsſitzungen 274. (original line) *0obe-ondrer '? '-änſiger Eindrue der Torerotheflgg,, (OCR result) In this case, using StrictGlobalSequenceAligner tries to take more than 20 GB RSS (at which point I quit).

bertsky commented 5 years ago

A workaround is to insert a limit into the current number of alignments as a second (non-terminal) case in backtraceFrom(): for example,

        elif len(alignments) > 100*max(f.shape):
            return # enough is enough

But surely there must be a cleaner way…