Use tokenizer_encode given as input to PromptDataset (bug fix)

lucidrains / toolformer-pytorch

Implementation of Toolformer, Language Models That Can Use Tools, by MetaAI

MIT License

1.94k stars 124 forks source link

Closed murthyn closed 11 months ago

murthyn commented 11 months ago

Summary: Use tokenizer_encode given as input to PromptDataset, instead of default tokenizer (small bug fix).

lucidrains commented 11 months ago

thanks!