Describe the bug
Segmenter will hang if it encounters unfinished html ( unescaped html attribute with unfinished hml tag).
The reason is the regex taht can be found in rules.py, class HTML:
HTMLTagRule = Rule(r"<\/?\w+((\s+\w+(\s*=\s*(?:\".*?\"|'.*?'|[\^'\">\s]+))?)+\s*|\s*)\/?>", '')
To Reproduce
Steps to reproduce the behavior:
sentencer = pysbd.Segmenter(language="en", clean=True)
txt = '<iframe width="100%" height="166" scrolling="no" frameborder="no" allow="autoplay" src="url Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum:<iframe width="100%" height="166" scrolling="no" frameborder="no" allow="autoplay" src="url Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum:<iframe width="100%" height="166" scrolling="no" frameborder="no" allow="autoplay" src="url Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum:<iframe width="100%" height="166" scrolling="no" frameborder="no" allow="autoplay" src="url Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum'
sentences = sentencer.segment(txt)
Expected behavior
The segmenter/regex should not hang if it encounters unfinished html.
Possible solution
We could replace the regex by a simplified one:
HTMLTagRule = Rule(r"(<([^>]+)>)", '')
that will do the same job (remove tags and keep the text) without the nested quantifiers that lead almost always to catastrophic backtracking.
Describe the bug Segmenter will hang if it encounters unfinished html ( unescaped html attribute with unfinished hml tag). The reason is the regex taht can be found in rules.py, class HTML:
HTMLTagRule = Rule(r"<\/?\w+((\s+\w+(\s*=\s*(?:\".*?\"|'.*?'|[\^'\">\s]+))?)+\s*|\s*)\/?>", '')To Reproduce Steps to reproduce the behavior:
You can test it here, please see the link: https://regex101.com/r/dGyZqj/1/
Expected behavior The segmenter/regex should not hang if it encounters unfinished html.
Possible solution We could replace the regex by a simplified one:
HTMLTagRule = Rule(r"(<([^>]+)>)", '')that will do the same job (remove tags and keep the text) without the nested quantifiers that lead almost always to catastrophic backtracking.