search

SimonWang9610 / gpt_tokenizer

BPE tokenizer used for Dart/Flutter applications when calling ChatGPT APIs
https://pub.dev/packages/flutter_gpt_tokenizer
8 stars 6 forks source link

Crashes when using count on IOS #4

Open youngjun625 opened 1 year ago

youngjun625 commented 1 year ago

Whenever I call 'await Tokenizer().count(prompt, modelName)' the entire app crashes. I checked that invoking Tokenizer() does fine, but only crashes on calling the count method It happened in

SimonWang9610 commented 1 year ago

Whenever I call 'await Tokenizer().count(prompt, modelName)' the entire app crashes.

I checked that invoking Tokenizer() does fine, but only crashes on calling the count method

It happened in

  • iPhone 11, IOS version 16.4

  • Simulator: iPad Pro (12.9-inch) (6th generation)

Does Tokenizer().encode(...) work well?

youngjun625 commented 1 year ago

hello, I tried .encode(...) as well to get the length of it but it crashes there as well :(

SimonWang9610 commented 1 year ago

hello, I tried .encode(...) as well to get the length of it but it crashes there as well :(

I will investigate and try to fix it. Thanks for your report.