poloclub / unitable

UniTable: Towards a Unified Table Foundation Model
https://arxiv.org/abs/2403.04822
MIT License
276 stars 15 forks source link

Consistency between prediction and VALID_HTML_TOKEN #18

Open ygzhu opened 5 days ago

ygzhu commented 5 days ago

Thanks for your contributions! I noticed that the VALID_HTML_TOKEN is composed of [""] and HTML_TOKENS in constant.py. VALID_HTML_TOKEN is used to filter the prediction during inference phase. But the annotation of "[" and "]" are not included in VALID_HTML_TOKEN. Does this omission affect the final prediction during the inference phase?