Closed gokceuludogan closed 11 months ago
This can be solved by perplexity filtering.
The mark_items() function was introduced in f2c10d2 to detect captions and correct the preceding or following lines, considering the surrounding lines and their average token length and digit ratio. Note that this function significantly increases the time required to process a document as it processes the lines in a sliding window way to correct the anomalies.
The correction step in the script should be performed considering the surrounding lines. This adjustment will help eliminate any lines that are kept between lines intended to be dropped, resulting in more accurate filtering.