Implement `tiktoken` in our own Rust binding

toeverything / AFFiNE

There can be more than Notion and Miro. AFFiNE(pronounced [ə‘fain]) is a next-gen knowledge base that brings planning, sorting and creating all together. Privacy first, open-source, customizable and ready to use.

https://affine.pro

Other

35.39k stars 2.33k forks source link

Implement `tiktoken` in our own Rust binding #6679

Open Brooooooklyn opened 2 months ago

Brooooooklyn commented 2 months ago

Currently we are using a slow wasm implementation: https://www.npmjs.com/package/tiktoken in https://github.com/toeverything/AFFiNE/blob/v0.14.0-canary.9/packages/backend/server/package.json#L90

### Tasks
- [ ] https://github.com/toeverything/AFFiNE/issues/6680

affine-issue-bot[bot] commented 2 months ago

Issue Status: 🆕 *Untriaged

*🆕 Untriaged**

The team has not yet reviewed the issue. We usually do it within one business day. Docs: https://github.com/toeverything/AFFiNE/blob/canary/docs/issue-triaging.md

^{_{This is an automatic reply by the bot.}}