switch tokenizer implementation with pure js and more compatible js-tiktoken

[masterkain]

closes https://github.com/transitive-bullshit/chatgpt-api/issues/570

[socket-security[bot]]

New dependency changes detected. Learn more about Socket for GitHub ↗︎

---

👍 No new dependency issues detected in pull request

Bot Commands

To ignore an alert, reply with a comment starting with @SocketSecurity ignore followed by a space separated list of package-name@version specifiers. e.g. @SocketSecurity ignore foo@1.0.0 bar@* or ignore all packages with @SocketSecurity ignore-all

Pull request alert summary

Issue	Status
Install scripts	✅ 0 issues
Native code	✅ 0 issues
Bin script shell injection	✅ 0 issues
Unresolved require	✅ 0 issues
Invalid package.json	✅ 0 issues
HTTP dependency	✅ 0 issues
Git dependency	✅ 0 issues
Potential typo squat	✅ 0 issues
Known Malware	✅ 0 issues
Telemetry	✅ 0 issues
Protestware/Troll package	✅ 0 issues

---

📊 Modified Dependency Overview:

➕ Added Package	Capability Access	+/- Transitive Count	Publisher
js-tiktoken@1.0.5	None	+0	davidduong

🚮 Removed packages: @dqbd/tiktoken@1.0.7

[masterkain]

tested, works

[BLipscomb]

When will this get merged?

[transitive-bullshit]

Thanks @masterkain

It seems @dqbd has multiple version of his port, and I wasn't able to find the differences clearly documented anywhere. (@dqbd/tiktoken, js-tiktoken, and tiktoken). Langchainjs seems to be using js-tiktoken, so we'll go with that for now.

Would love for this to be clearer, though.