Closed lolipopshock closed 2 years ago
Hi @lolipopshock, and thanks for submitting this PR! I really appreciate the clear explanation and examples. The CI pipeline is getting caught on the linting step. If you could reformat your code with psf/black — just run make format
from the repository root — and commit/push, that'd be great. In the meantime, I'll take a closer look at the proposed changes.
@jsvine thank you for your prompt response -- just updated the files!
Thanks again, @lolipopshock. This seems like a worthwhile addition to the library, and I really appreciate you providing such a thorough PR. I'm going to merge into develop
. I might also fiddle with the implementation slightly (merging this new logic with the keep_blank_chars
logic), but nothing radical.
In this PR, we add an optional argument to the
extract_words
function that allows to enforce breaking tokens at punctuations.Usage
More visualization
The red boxes denote the detected tokens![image](https://user-images.githubusercontent.com/22512825/177434450-74bab47e-87b5-4ea8-a355-fe95e6640c97.png)