libgpt_tokenizer.so not found

[alinemati-uwm]

Hi I used this lib but when I run my app I get this error: [ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: Invalid argument(s): Failed to load dynamic library 'libgpt_tokenizer.so': dlopen failed: library "libgpt_tokenizer.so" not found E/flutter ( 4258): #0 _open (dart:ffi-patch/ffi_dynamic_library_patch.dart:11:43) E/flutter ( 4258): #1 new DynamicLibrary.open (dart:ffi-patch/ffi_dynamic_library_patch.dart:22:12) E/flutter ( 4258): #2 createLibraryImpl (package:flutter_gpt_tokenizer/src/ffi/io.dart:12:27) E/flutter ( 4258): #3 _loadTokenizerLib (package:flutter_gpt_tokenizer/src/tokenizer.dart:12:34) E/flutter ( 4258): #4 _loadTokenizer (package:flutter_gpt_tokenizer/src/tokenizer.dart:23:13) E/flutter ( 4258): #5 Tokenizer._getTokenizer (package:flutter_gpt_tokenizer/src/tokenizer.dart:149:31) E/flutter ( 4258): E/flutter ( 4258): #6 Tokenizer.encode (package:flutter_gpt_tokenizer/src/tokenizer.dart:54:23) E/flutter ( 4258): E/flutter ( 4258): #7 GPTTokenizer.tokenize (package:nerd_studio/business_logic/services/GPTTokenizer.dart:34:27) E/flutter ( 4258): E/flutter ( 4258): #8 GPTTokenizer.checkIfThisInputTextCanBeSent (package:nerd_studio/business_logic/services/GPTTokenizer.dart:66:43) E/flutter ( 4258): E/flutter ( 4258): #9 PromptCreator.addPromptsAndResponse (package:nerd_studio/business_logic/services/PromptCreator.dart:132:40) E/flutter ( 4258): E/flutter ( 4258): #10 ChatWidgetState._sendMessageToEngine (package:nerd_studio/presentation/pages/conversation/ChatWidget.dart:556:18) E/flutter ( 4258): E/flutter ( 4258):

this is my code that use this lib Future tokenize(inputText) async { int maxLengthTokenizerIndex = 0, maxLength = 0; List maxEncodedText = [];

//this loop encodes the input text and find the maximum length
for (var i = 0; i < models.length; i++) {
  var encodedTokens = await Tokenizer().encode(
    inputText,
    modelName: models[i],
  );

  if (encodedTokens.length > maxLength) {
    maxLength = encodedTokens.length;
    maxLengthTokenizerIndex = i;
    maxEncodedText = encodedTokens;
  }
}
return TokenizerReturnBean(
    encodedDataLength: maxEncodedText.length,
    modelName: models[maxLengthTokenizerIndex],
    encodedData: maxEncodedText,
    prompt: inputText);

}

and this is my flutter doctor Doctor summary (to see all details, run flutter doctor -v): [√] Flutter (Channel stable, 3.10.6, on Microsoft Windows [Version 10.0.22621.2134], locale en-US) [√] Windows Version (Installed version of Windows is version 10 or higher) [√] Android toolchain - develop for Android devices (Android SDK version 34.0.0) [√] Chrome - develop for the web [√] Visual Studio - develop for Windows (Visual Studio Community 2019 16.11.27) [√] Android Studio (version 2022.3) [√] VS Code (version 1.81.1) [√] Connected device (5 available) [√] Network resources

• No issues found!

any one can help me ?

[esfandiaripgdp]

I have this problem too

[SimonWang9610]

It is kind of like you have not downloaded the compiled library yet. However, it is wired because you should download it from Git Hub when you run flutter pub get.

I will try to fix it.

[SimonWang9610]

you could check if libgpt_tokenizer.so could be found in /path/to/.pub-cache/hosted/pub.dev/flutter_gpt_tokenizer-0.1.0/android

[marcglasberg]

In Windows:

[ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: Invalid argument(s): Failed to load dynamic library 'gpt_tokenizer.dll': The specified module could not be found.

 (error code: 126)
#0      _open (dart:ffi-patch/ffi_dynamic_library_patch.dart:11:43)
#1      new DynamicLibrary.open (dart:ffi-patch/ffi_dynamic_library_patch.dart:22:12)
#2      createLibraryImpl (package:flutter_gpt_tokenizer/src/ffi/io.dart:10:27)
#3      _loadTokenizerLib (package:flutter_gpt_tokenizer/src/tokenizer.dart:12:34)
#4      _loadTokenizer (package:flutter_gpt_tokenizer/src/tokenizer.dart:23:13)
#5      Tokenizer._getTokenizer (package:flutter_gpt_tokenizer/src/tokenizer.dart:161:31)
<asynchronous suspension>
#6      Tokenizer.count (package:flutter_gpt_tokenizer/src/tokenizer.dart:94:23)
<asynchronous suspension>