Open vgopinath opened 4 years ago
Hi, In creating custom data for QA, took Squad 1.1 and adding my own sentences with QA. Is there a tool to calculate text tokens (& start byte) ? Pls help.
Hi,
Unfortunately there is no specific tool for that, but this can be achieved by using the relevant tokenizer to your model.
Hi, In creating custom data for QA, took Squad 1.1 and adding my own sentences with QA. Is there a tool to calculate text tokens (& start byte) ? Pls help.