Open jihan-yin opened 1 year ago
This is because tokenizer.cache['toString'] = {function toString()}
. Not sure how this is happening
Currently resolving by adding
// @ts-ignore
tokenizer.cache['toString'] = 'toString';
The bpe still returns the javascript native function instead of token ids though, as tokenizer.encodings['toString']
is set to the function for some reason
will fail. This is because
tokenizer.bpe('toString')
returns the javascript functiontoString()
instead of an actual string representing the token.