any tool to extract start-end bytes?

alontalmor / MultiQA

138 stars 23 forks source link

any tool to extract start-end bytes? #17

Open vgopinath opened 4 years ago

vgopinath commented 4 years ago

Hi, In creating custom data for QA, took Squad 1.1 and adding my own sentences with QA. Is there a tool to calculate text tokens (& start byte) ? Pls help.

alontalmor commented 4 years ago

Hi,

Unfortunately there is no specific tool for that, but this can be achieved by using the relevant tokenizer to your model.