mengyao / Complete-Striped-Smith-Waterman-Library

298 stars 113 forks source link

gap extension penalty is ignored if extension penalty > opening penalty #75

Closed stuckyb closed 2 years ago

stuckyb commented 4 years ago

As best as I can tell by testing with the Python wrapper, the gap extension penalty is not actually used when calculating alignment scores. Instead, the "opening penalty" is assessed against each gap position. Another way of saying this is that gaps are assessed using linear penalties instead of an affine penalties.

stuckyb commented 4 years ago

Okay, I did some more investigating and can now confirm that affine gap penalties are scored as expected as long as extension penalty <= opening penalty. If extension penalty > opening penalty, the extension penalty is essentially ignored and the opening penalty is assessed against every gap position. I realize that the parameter space that triggers the bug is rare for biological sequence alignments, but this is still a bug if viewed as a general implementation of the Smith-Waterman algorithm.