dave-gray101
commented
1 year ago - Adds a new function TokenizeString. To make things simple to implement across backends, it takes similar parameters to Predict and pulls the string to tokenize from the Prompt parameter.
- This will initially be used for returning the TokenUsage of the prompts, but I'm hopeful it'll find other uses as well.
- Please review the CGo! The way that I'm boxing ints between 32 and 64 bits seems... wrong.
- Minor cleanup to makefiles and gitignore
- Minor test refactoring to allow for a new TokenizeString test case.