Openai's Tiktoken implementation written in Swift. This is basic implementation from ordinary encode/decode.
Supports vocab:
And also supports asian characters and emojis.
Stars are welcome 😊.
let encoder = try await Tiktoken.shared.getEncoding("gpt-4")
let encoded = encoder?.encode(value: "這個算法真的太棒了")
print(encoded)
let decoded = encoder?.decode(value: encoded)
print(decoded)