search

allenai / vila

Incorporating VIsual LAyout Structures for Scientific Text Classification
Apache License 2.0
173 stars 16 forks source link

Replace Certain Unicode Characters for the input #23

Closed lolipopshock closed 2 years ago

lolipopshock commented 2 years ago

HF tokenizer will replace certain unicode characters with a space ' '. Therefore, the token-level prediction will become shorter than the input, which can cause mis-matched sequences. This PR tries to fix this issue via enabling replacing such unicode characters with the [UNK] tokens. We replace unicode characters in certain "categories", namely, ["Cc", "Cf", "Co", "Cs", "Mn", "Zl", "Zp", "Zs"], as specified by the rules in the corresponding HF tokenizer:

Usage:

df_predictor.predict(pdf_data, page_size, replace_empty_unicode=False)

A future update could be just replacing the unicode characters in the cached file examples/find-empty-unicode-chars/zero-length-unicode-chars.txt, which we've tested and confirmed that has zero tokenization lengths.

lolipopshock commented 2 years ago

replace_bad_unicode_with_unk