Sorry for probably stupid question, but my task is to count tokens before sending text for vector embedding using cl100k_base - that one seems to be used with text-embedding-3-large model.
I've not seen such an example in the readme and wondering whether this model is supported by this module and how to use it properly for this task.
I mean that I have only one text, should I put it into 'system' or 'user' role field?
And which model I have to specify?
Thanks in advance!
Sorry for probably stupid question, but my task is to count tokens before sending text for vector embedding using
cl100k_base- that one seems to be used withtext-embedding-3-largemodel. I've not seen such an example in the readme and wondering whether this model is supported by this module and how to use it properly for this task. I mean that I have only one text, should I put it into 'system' or 'user' role field? And which model I have to specify? Thanks in advance!