Previously we would not normalise the input text before encoding which resulted in discrepancies for certain special characters, e.g. ™. These cases should now match the Python SDK.
This PR also:
Imports Tiktoken from the /lite sub-module to reduce the bundle size.
Previously we would not normalise the input text before encoding which resulted in discrepancies for certain special characters, e.g. ™. These cases should now match the Python SDK.
This PR also:
/litesub-module to reduce the bundle size.<EOT>