search

openedtech / edubot

GNU General Public License v3.0
16 stars 5 forks source link

Use tiktoken library to estimate tokens #24

Closed pravintargaryen closed 1 year ago

pravintargaryen commented 1 year ago

Updated the tokens function with tiktoken tokenizer library