search

firebase / flutterfire

🔥 A collection of Firebase plugins for Flutter apps.
https://firebase.google.com/docs/flutter/setup
BSD 3-Clause "New" or "Revised" License
8.7k stars 3.97k forks source link

[firebase_vertexai]: embedContent calls result in 404 #13269

Open MrCsabaToth opened 1 month ago

MrCsabaToth commented 1 month ago

Is there an existing issue for this?

Which plugins are affected?

Other

Which platforms are affected?

Android

Description

I'm trying to obtain emnbeddings for short text content payloads. Neither the text-embedding-004 (which works with google_generative_ai) nor the text-multilingual-embedding-002 work, both fail with a FormatException: Unexpected character (at offset 0) (kinda like #13221) because the JSON interpretation goes side ways due to the return payload being bodyBytes length 38 [0x3c, 0x68, 0x31, 0x3e, 0x4e, 0x6f, 0x74, 0x20, 0x46, 0x6f, 0x75, 0x6e, 0x64, 0x3c, 0x2f, 0x68, 0x31, 0x3e, 0xa, 0x3c, 0x68, 0x32, 0x3e, 0x45, 0x72, 0x72, 0x6f, 0x72, 0x20, 0x34, 0x30, 0x34, 0x3c, 0x2f, 0x68, 0x32, 0x3e, 0xa] which UTF decodes to:

<h1>Not Found</h1>
<h2>Error 404</h2>

The interpreter code doesn't care that the HTTP return code is 404 and tries to interpret the first < character as a number and then tumbles down the rabbit hole.

I tried Firebase Vertex AI, because I'd like to use text-multilingual-embedding-002 and google_generative_ai only supports text-embedding-004 https://github.com/google-gemini/generative-ai-dart/issues/209.

As far as I see Firebase Vertex AI uses Google AI API under the hood (see https://github.com/firebase/flutterfire/blob/46b13044f1e16700797180ca11e807ed81f70413/packages/firebase_vertexai/firebase_vertexai/pubspec.yaml#L22 and https://github.com/firebase/flutterfire/blob/46b13044f1e16700797180ca11e807ed81f70413/packages/firebase_vertexai/firebase_vertexai/lib/src/vertex_model.dart#L75), however I've seen miracle this year when Firebase cured incurable nagging GCP Cloud Function CORS errors simply calling over through Firebase endpoints, so I'm hopeful there can be some solution.

I tried to supply taskType: TaskType.retrievalQuery with the embedContent call, however the task embedContent in the URL is independent from that, and ultimately I need to find what's the right URL.

Google AI API uses v1beta in the call URL whereas Firebase Vertex AI uses v2beta. Not sure if that matters, the domain names are different in the two calls:

  1. https://firebaseml.googleapis.com/v2beta/projects/[MY_PROJECT_ID]/locations/us-central1/publishers/google/models/text-embedding-004:embedContent
  2. https://generativelanguage.googleapis.com/v1beta/models/text-embedding-004:embedContent

Reproducing the issue

  1. I assume that the installation steps are all done such as activating CLI, initializing Firebase via the CLI, firebase.json firebase_options.dart google-services.json generated, Android app is created under the corresponding Firebase project and google-services.json updated, Firebase.initializeApp called, GCP APIs are turned on, service accounts are suited up etc.
  2. Instantiate an embedding model and then try to call embedContent on it.

Example:

    final model = FirebaseVertexAI.instance.generativeModel(
      model: 'text-embedding-004', // 'text-multilingual-embedding-002',
    );
    final content = Content.text(prompt);
    final embeddingResult = await model.embedContent(content);

Firebase Core version

3.4.0

Flutter Version

3.24.1

Relevant Log Output

[log] FormatException: Unexpected character (at offset 0)
      #0      _ChunkedJsonParser.fail (dart:convert-patch/convert_patch.dart:1465:5)
      #1      _ChunkedJsonParser.parseNumber (dart:convert-patch/convert_patch.dart:1332:9)
      #2      _ChunkedJsonParser.parse (dart:convert-patch/convert_patch.dart:933:22)
      #3      _JsonUtf8Parser.parseChunk (dart:convert-patch/convert_patch.dart:1601:5)
      #4      _JsonUtf8Decoder.convert (dart:convert-patch/convert_patch.dart:63:12)
      #5      Codec.decode (dart:convert/codec.dart:30:34)
      #6      HttpApiClient.makeRequest (package:google_generative_ai/src/client.dart:64:22)
      <asynchronous suspension>
      #7      parseEmbedContentResponse (package:google_generative_ai/src/api.dart:618:1)
      <asynchronous suspension>
      #8      GenerativeModel.embedContent.<anonymous closure> (package:firebase_vertexai/src/vertex_model.dart:219:15)
      <asynchronous suspension>
      #9      EmbeddingMixin.obtainEmbedding (package:inspector_gadget/ai/service/embedding_mixin.dart:34:29)
      <asynchronous suspension>
      #10     PersonalizationPageState.embedAndPersistPersonalization (package:inspector_gadget/database/view/personalization_page.dart:83:25)
      <asynchronous suspension>
      #11     PersonalizationPageState._processDeferredActionQueue (package:inspector_gadget/database/view/personalization_page.dart:117:13)
      <asynchronous suspension>

Flutter dependencies

Expand Flutter dependencies snippet
```yaml Dart SDK 3.5.1 Flutter SDK 3.24.1 inspector_gadget 1.0.1+1 dependencies: - adaptive_dialog 2.2.0 [animations collection dynamic_color flutter intersperse macos_ui meta] - assorted_layout_widgets 9.0.2 [clock flutter matrix4_transform] - camera 0.11.0+2 [camera_android_camerax camera_avfoundation camera_platform_interface camera_web flutter flutter_plugin_android_lifecycle] - collection 1.19.0 - country_flags 3.0.0 [collection flutter jovial_svg] - dart_helper_utils 2.5.2 [intl mime] - daylight 3.1.0 [equatable] - file_picker 8.1.2 [flutter flutter_web_plugins flutter_plugin_android_lifecycle plugin_platform_interface ffi path win32 cross_file web] - firebase_core 3.4.0 [firebase_core_platform_interface firebase_core_web flutter meta] - firebase_vertexai 0.2.3+1 [firebase_app_check firebase_auth firebase_core firebase_core_platform_interface flutter google_generative_ai] - fl_location 4.1.0 [flutter fl_location_platform_interface fl_location_web] - flex_color_scheme 7.3.1 [flex_seed_scheme flutter meta] - flutter 0.0.0 [characters collection material_color_utilities meta vector_math sky_engine] - flutter_easy_animations 0.0.2 [flutter] - flutter_layout_grid 2.0.7 [flutter collection quiver] - flutter_localizations 0.0.0 [flutter intl characters clock collection material_color_utilities meta path vector_math] - flutter_localized_locales 2.0.5 [flutter intl] - flutter_tts 4.0.2 [flutter flutter_web_plugins] - get_it 7.7.0 [async collection meta] - google_generative_ai 0.4.5 [http] - heart_rate_flutter 0.0.2 [flutter plugin_platform_interface] - http 1.2.2 [async http_parser meta web] - intl 0.19.0 [clock meta path] - listview_utils_plus 0.0.3 [flutter http] - media_kit 1.1.11 [meta path image uri_parser collection synchronized uuid http safe_local_storage universal_platform js] - media_kit_libs_audio 1.0.5 [media_kit_native_event_loop media_kit_libs_android_audio media_kit_libs_ios_audio media_kit_libs_macos_audio media_kit_libs_windows_audio media_kit_libs_linux] - mime 1.0.6 - number_selector 1.0.7 [flutter] - objectbox 4.0.2 [collection flat_buffers ffi meta path] - objectbox_flutter_libs 4.0.2 [flutter objectbox path_provider] - path 1.9.0 - path_provider 2.1.4 [flutter path_provider_android path_provider_foundation path_provider_linux path_provider_platform_interface path_provider_windows] - pref 2.8.0 [flutter logging shared_preferences] - record 5.1.2 [flutter uuid record_platform_interface record_web record_windows record_linux record_android record_darwin] - speech_to_text 7.0.0 [flutter speech_to_text_platform_interface json_annotation clock pedantic flutter_web_plugins meta web] - statemachine 3.3.1 [meta] - strings 3.1.2 [characters unicode] - translator 1.0.0 [http] - tuple 2.0.2 - video_player 2.9.1 [flutter html video_player_android video_player_avfoundation video_player_platform_interface video_player_web] - watch_it 1.4.2 [get_it flutter functional_listener] dev dependencies: - build_runner 2.4.12 [analyzer args async build build_config build_daemon build_resolvers build_runner_core code_builder collection crypto dart_style frontend_server_client glob graphs http_multi_server io js logging meta mime package_config path pool pub_semver pubspec_parse shelf shelf_web_socket stack_trace stream_transform timing watcher web_socket_channel yaml] - flutter_test 0.0.0 [flutter test_api matcher path fake_async clock stack_trace vector_math leak_tracker_flutter_testing async boolean_selector characters collection leak_tracker leak_tracker_testing material_color_utilities meta source_span stream_channel string_scanner term_glyph vm_service] - mocktail 1.0.4 [collection matcher test_api] - objectbox_generator 4.0.2 [objectbox analyzer build collection dart_style glob path source_gen pubspec_parse yaml http cryptography] - very_good_analysis 6.0.0 dependency overrides: - collection 1.19.0 - js 0.7.1 - web 1.0.0 transitive dependencies: - _fe_analyzer_shared 73.0.0 [meta] - _flutterfire_internals 1.3.41 [collection firebase_core firebase_core_platform_interface flutter meta] - _macros 0.3.2 - analyzer 6.8.0 [_fe_analyzer_shared collection convert crypto glob macros meta package_config path pub_semver source_span watcher yaml] - animations 2.0.11 [flutter] - appkit_ui_element_colors 1.0.0 [equatable flutter plugin_platform_interface] - archive 3.6.1 [crypto path] - args 2.5.0 - async 2.11.0 [collection meta] - boolean_selector 2.1.1 [source_span string_scanner] - build 2.4.1 [analyzer async convert crypto glob logging meta package_config path] - build_config 1.1.1 [checked_yaml json_annotation path pubspec_parse yaml] - build_daemon 4.0.2 [built_collection built_value crypto http_multi_server logging path pool shelf shelf_web_socket stream_transform watcher web_socket_channel] - build_resolvers 2.4.2 [analyzer async build collection convert crypto graphs logging package_config path pool pub_semver stream_transform yaml] - build_runner_core 7.3.2 [async build build_config build_resolvers collection convert crypto glob graphs json_annotation logging meta package_config path pool timing watcher yaml] - built_collection 5.1.1 - built_value 8.9.2 [built_collection collection fixnum meta] - camera_android_camerax 0.6.8+3 [async camera_platform_interface flutter meta stream_transform] - camera_avfoundation 0.9.17+3 [camera_platform_interface flutter stream_transform] - camera_platform_interface 2.8.0 [cross_file flutter plugin_platform_interface stream_transform] - camera_web 0.3.5 [camera_platform_interface flutter flutter_web_plugins stream_transform web] - characters 1.3.0 - checked_yaml 2.0.3 [json_annotation source_span yaml] - clock 1.1.1 - code_builder 4.10.0 [built_collection built_value collection matcher meta] - convert 3.1.1 [typed_data] - cross_file 0.3.4+2 [meta web] - crypto 3.0.5 [typed_data] - cryptography 2.7.0 [collection crypto ffi js meta typed_data] - csslib 1.0.0 [source_span] - dart_style 2.3.6 [analyzer args collection path pub_semver source_span] - dynamic_color 1.7.0 [flutter flutter_test material_color_utilities] - equatable 2.0.5 [collection meta] - fake_async 1.3.1 [clock collection] - ffi 2.1.3 - file 7.0.0 [meta path] - firebase_app_check 0.3.1 [firebase_app_check_platform_interface firebase_app_check_web firebase_core firebase_core_platform_interface flutter] - firebase_app_check_platform_interface 0.1.0+35 [_flutterfire_internals firebase_core flutter meta plugin_platform_interface] - firebase_app_check_web 0.1.2+13 [_flutterfire_internals firebase_app_check_platform_interface firebase_core firebase_core_web flutter flutter_web_plugins web] - firebase_auth 5.2.0 [firebase_auth_platform_interface firebase_auth_web firebase_core firebase_core_platform_interface flutter meta] - firebase_auth_platform_interface 7.4.4 [_flutterfire_internals collection firebase_core flutter meta plugin_platform_interface] - firebase_auth_web 5.12.6 [firebase_auth_platform_interface firebase_core firebase_core_web flutter flutter_web_plugins http_parser meta web] - firebase_core_platform_interface 5.2.1 [collection flutter flutter_test meta plugin_platform_interface] - firebase_core_web 2.17.5 [firebase_core_platform_interface flutter flutter_web_plugins meta web] - fixnum 1.1.0 - fl_location_platform_interface 5.0.0 [flutter meta vector_math plugin_platform_interface] - fl_location_web 4.1.0 [flutter flutter_web_plugins fl_location_platform_interface] - flat_buffers 23.5.26 - flex_seed_scheme 1.5.0 [collection flutter meta] - flutter_plugin_android_lifecycle 2.0.22 [flutter] - flutter_web_plugins 0.0.0 [flutter characters collection material_color_utilities meta vector_math] - frontend_server_client 4.0.0 [async path] - functional_listener 2.3.1 [flutter] - glob 2.1.2 [async collection file path string_scanner] - gradient_borders 1.0.1 [flutter] - graphs 2.3.2 [collection] - html 0.15.4 [csslib source_span] - http_multi_server 3.2.1 [async] - http_parser 4.1.0 [collection source_span string_scanner typed_data] - image 4.2.0 [archive meta xml] - intersperse 2.0.0 - io 1.0.4 [meta path string_scanner] - jovial_misc 0.8.5 [pointycastle async collection convert meta] - jovial_svg 1.1.22 [xml collection args vector_math jovial_misc http meta flutter] - json_annotation 4.9.0 [meta] - leak_tracker 10.0.5 [clock collection meta path vm_service] - leak_tracker_flutter_testing 3.0.5 [flutter leak_tracker leak_tracker_testing matcher meta] - leak_tracker_testing 3.0.1 [leak_tracker matcher meta] - lists 1.0.1 [meta] - logging 1.2.0 - macos_ui 2.0.8 [flutter macos_window_utils gradient_borders appkit_ui_element_colors equatable] - macos_window_utils 1.5.0 [flutter] - macros 0.1.2-main.4 [_macros] - matcher 0.12.16+1 [async meta stack_trace term_glyph test_api] - material_color_utilities 0.11.1 [collection] - matrix4_transform 3.0.1 [flutter] - media_kit_libs_android_audio 1.3.6 [flutter plugin_platform_interface] - media_kit_libs_ios_audio 1.1.4 [flutter plugin_platform_interface] - media_kit_libs_linux 1.1.3 [flutter] - media_kit_libs_macos_audio 1.1.4 [flutter plugin_platform_interface] - media_kit_libs_windows_audio 1.0.9 [flutter] - media_kit_native_event_loop 1.0.9 [flutter] - meta 1.15.0 - package_config 2.1.0 [path] - path_provider_android 2.2.10 [flutter path_provider_platform_interface] - path_provider_foundation 2.4.0 [flutter path_provider_platform_interface] - path_provider_linux 2.2.1 [ffi flutter path path_provider_platform_interface xdg_directories] - path_provider_platform_interface 2.1.2 [flutter platform plugin_platform_interface] - path_provider_windows 2.3.0 [ffi flutter path path_provider_platform_interface] - pedantic 1.11.1 - petitparser 6.0.2 [meta] - platform 3.1.5 - plugin_platform_interface 2.1.8 [meta] - pointycastle 3.9.1 [collection convert js] - pool 1.5.1 [async stack_trace] - pub_semver 2.1.4 [collection meta] - pubspec_parse 1.3.0 [checked_yaml collection json_annotation pub_semver yaml] - quiver 3.2.2 [matcher] - record_android 1.2.6 [flutter record_platform_interface] - record_darwin 1.1.2 [flutter record_platform_interface] - record_linux 0.7.2 [flutter record_platform_interface] - record_platform_interface 1.1.0 [flutter plugin_platform_interface] - record_web 1.1.3 [flutter flutter_web_plugins web record_platform_interface] - record_windows 1.0.3 [flutter record_platform_interface] - safe_local_storage 1.0.2 [path synchronized] - shared_preferences 2.3.2 [flutter shared_preferences_android shared_preferences_foundation shared_preferences_linux shared_preferences_platform_interface shared_preferences_web shared_preferences_windows] - shared_preferences_android 2.3.2 [flutter shared_preferences_platform_interface] - shared_preferences_foundation 2.5.2 [flutter shared_preferences_platform_interface] - shared_preferences_linux 2.4.1 [file flutter path path_provider_linux path_provider_platform_interface shared_preferences_platform_interface] - shared_preferences_platform_interface 2.4.1 [flutter plugin_platform_interface] - shared_preferences_web 2.4.2 [flutter flutter_web_plugins shared_preferences_platform_interface web] - shared_preferences_windows 2.4.1 [file flutter path path_provider_platform_interface path_provider_windows shared_preferences_platform_interface] - shelf 1.4.2 [async collection http_parser path stack_trace stream_channel] - shelf_web_socket 2.0.0 [shelf stream_channel web_socket_channel] - sky_engine 0.0.99 - source_gen 1.5.0 [analyzer async build dart_style glob path source_span yaml] - source_span 1.10.0 [collection path term_glyph] - speech_to_text_platform_interface 2.2.0 [flutter meta plugin_platform_interface] - sprintf 7.0.0 - stack_trace 1.11.1 [path] - stream_channel 2.1.2 [async] - stream_transform 2.1.0 - string_scanner 1.2.0 [source_span] - synchronized 3.2.0 - term_glyph 1.2.1 - test_api 0.7.2 [async boolean_selector collection meta source_span stack_trace stream_channel string_scanner term_glyph] - timing 1.0.1 [json_annotation] - typed_data 1.3.2 [collection] - unicode 0.3.1 [lists] - universal_platform 1.1.0 - uri_parser 2.0.2 [path safe_local_storage] - uuid 4.5.0 [crypto sprintf meta fixnum] - vector_math 2.1.4 - video_player_android 2.7.2 [flutter video_player_platform_interface] - video_player_avfoundation 2.6.1 [flutter video_player_platform_interface] - video_player_platform_interface 6.2.2 [flutter plugin_platform_interface] - video_player_web 2.3.2 [flutter flutter_web_plugins video_player_platform_interface web] - vm_service 14.2.5 - watcher 1.1.0 [async path] - web_socket 0.1.6 [web] - web_socket_channel 3.0.1 [async crypto stream_channel web web_socket] - win32 5.5.4 [ffi] - xdg_directories 1.0.4 [meta path] - xml 6.5.0 [collection meta petitparser] - yaml 3.1.2 [collection source_span string_scanner] ```

Additional context and comments

There might be other platforms affected besides Android. I hope you like this meme I generated: https://imgflip.com/i/92caxj

MrCsabaToth commented 1 month ago

Could this be the us-central1 location? Although usually that has most of what GCP and Vertex AI can offer. And text-embedding-004 works via google_generative_ai

csells commented 3 weeks ago

I'm seeing this exact same issue on macOS. And it's blocking me from doing RAG w/ vertex (even though it works just fine with the gemini SDK).

MrCsabaToth commented 3 weeks ago

A workaround could be (which I'll do) is to implement the embedding in a Firebase function.