In the pdfplumber preprocess pipeline I've found that duplicated tokens may exist. Specifically in the obtain_word_tokens of the pdfplumber.py file, one should put a .drop_duplicates before converting the dataframe to list.
It can happen in some cases that the tokens from a PAWLS pdf structure appear duplicated and this messes up things a bit when indexing from the annotation file.
In the pdfplumber preprocess pipeline I've found that duplicated tokens may exist. Specifically in the
obtain_word_tokens
of thepdfplumber.py
file, one should put a.drop_duplicates
before converting the dataframe to list.It can happen in some cases that the tokens from a PAWLS pdf structure appear duplicated and this messes up things a bit when indexing from the annotation file.