According to the exception message, it seems this issue possibly is caused by the missing implementation of Rust-based tokenization, while calling tokenize() method to tokenize batch of sequences and access the result of batch_tokenized with slice(not str like input_ids or attention_mask, etc.) . Does anyone could help with this? Thanks!
According to the exception message, it seems this issue possibly is caused by the missing implementation of Rust-based tokenization, while calling
tokenize()
method to tokenize batch of sequences and access the result ofbatch_tokenized
withslice
(not str likeinput_ids
orattention_mask
, etc.) . Does anyone could help with this? Thanks!