Gemini integration: Image type "Image Network URL" does not work

[AndreSchm]

Can we access your project?

• [X] I give permission for members of the FlutterFlow team to access and test my project for the sole purpose of investigating this issue.

Current Behavior

Using the Gemini Integration I tried to use the Text from Image feature by uploading an image to a container using "Upload media to Firebase" This image URL is used for the Text from Image Feature of the Gemini Integration: but the action output is not updated after the action:

Expected Behavior

Action output "geminiResponse" should have the response of the Gemini API.

Using the local upload for the container action and providing the uploaded image as gemini input as "Uploaded Image File" works fine and I get an value for "geminiResponse".

Steps to Reproduce

1. Create a new page
2. Add a container and provide an action for image upload as "Upload media to Firebase"
3. Add button with action of gemini integration using Text from image and image type "Image Network URL" and action output "geminiResponse"
4. Run the app and debug "geminiResponse" after button click

Reproducible from Blank

• [X] The steps to reproduce above start from a blank project.

Bug Report Code (Required)

IT4kz8r6w5N2od9b1s/1bflVoDgUKUc3TLJEoetxc0gbfe/4PZkXPczBZRdWZ72zaH1mGEyZ/UcC+fzFv+HfJPEoaDGtG9lG+M0NWBPPbEOVeKayC6qKXnBRP9tUN0iP356nhRBQG/hvc3An6kOuOPOXTT6Cf9qOYwx5e6fDbOY=

Visual documentation

Environment

- FlutterFlow version: FlutterFlow v5.0.0+ released October 9, 2024
- Platform: Web
- Browser name and version: Google Chrome Version 129.0.6668.90 (Offizieller Build) (64-Bit)
- Operating system and version affected: Windows 11 Pro, 10.0.22631

Additional Information

No response

[ambience-kim]

I have been using FF to integrate OpenAI's API with my app. However, since the beginning of this week, I encountered an issue with the API integration that was previously working fine.

While there is no issue with text input and output, the API that accepts images via a URL (not as a file) returns an "invalid image" error message. To check if there was an issue with my code, I tested it with the version I released last week, which was working perfectly at the time. However, the same error message appears in that version as well.

This seems to be the same issue, and I believe it is a problem with the API tool.

[AndreSchm]

Today I tried Inspecting the network activity in the run mode. A CORS error is issued when running the gemini action. When installing the APK the Gemini integration works fine.

[ambience-kim]

1. OpenAI Error Message Details

I posted about this issue on OpenAI's forums as well. I'm not sure if it’s because of my post, but today, when I tested again, OpenAI started providing more detailed error messages. (For debugging purposes, I store the error messages returned by the OpenAI API in Firestore.)

[Previously]

• Error code: invalid_image
• Error type: invalid_request_error
• Error message: Invalid image.

[Currently]

• Error code: invalid_image_url
• Error type: invalid_request_error
• Error message: Timeout while downloading https://firebasestorage.googleapis.com/v0/b/zzan-prod.appspot.com/o/users%2FjJYiyCR65JbF3jwapT81cFV4qSy2%2Fuploads%2F1729134282375000.jpg?alt=media&token=15c86709-d398-47f6-bcbd-c334456628e1.

However, when I check the image path in a web browser, the image loads correctly. I’m not sure if the period "." at the end of the image URL is mistakenly added by the Cloud Function in FlutterFlow or if OpenAI adds it for grammatical reasons in the error message. Either way, whether the period is present or not, the image displays correctly in a web browser.

2. CORS Issue

I have already configured CORS settings in Firebase and have been using them without any issues. While my previous CORS settings didn’t include OpenAI’s API domain, my apps were functioning fine up until about two weeks ago. This could be because server-to-server communication generally doesn’t require CORS settings.

Nevertheless, to be thorough, I added https://api.openai.com to the CORS settings in Firebase today. I tested it again, but the same error still persists.

• Error code: invalid_image_url
• Error type: invalid_request_error
• Error message: Timeout while downloading https://firebasestorage.googleapis.com/v0/b/zzan-prod.appspot.com/o/users%2FjJYiyCR65JbF3jwapT81cFV4qSy2%2Fuploads%2F1729134282375000.jpg?alt=media&token=15c86709-d398-47f6-bcbd-c334456628e1.

3. Cause and Solution ?

Now that the issue's symptoms have been clearly identified, could the FlutterFlow team please help determine the cause of this issue and provide a solution?

[ambience-kim]

FF Team,

please respond to this issue.