Support blobs to be sent for google ai studio - gemini

haseeb-heaven commented 6 months ago

The Feature

Proposal: Implement multiple image upload modes in Google Gemini AI Studio

Description:

Enable users to upload images through three distinct methods:
1. URL: Provide a field for users to input a direct image URL.
2. Local storage: Allow users to browse and select images from their local device.
3. Online storage: Integrate with popular cloud storage services such as Google Drive and Dropbox, enabling users to seamlessly import images from their existing online repositories.

Enhancements:

User experience: Offer flexibility and convenience to users by accommodating various image sourcing preferences.
Project adaptability: Expand the range of projects and use cases that can effectively leverage image-based interactions within Gemini AI Studio.

Motivation, pitch

Motivation:

Address diverse image sourcing needs: Accommodate users who work with images from various origins, including web sources, local collections, and cloud-based storage.
Unleash visual creativity: Empower users to explore a broader spectrum of visual prompts and image-based interactions within Gemini AI Studio.
Streamline workflows: Reduce friction by allowing users to directly access and integrate images from their preferred sources, enhancing productivity and efficiency.
Align with cloud-centric practices: Foster seamless integration with existing cloud-based workflows and data management strategies.

Specific Problem:

The current restriction to image uploads via URL only limits the scope of visual experimentation and potential use cases within Gemini AI Studio.

Twitter / LinkedIn details

haseeb_heaven

krrishdholakia commented 6 months ago

@haseeb-heaven we already support passing images via URL + local storage for gemini ai studio.

The missing one is blob, but i don't any code examples of how blob data is processed / passed for ai studio.

Can you point me to a relevant code example on this?

haseeb-heaven commented 6 months ago

@haseeb-heaven we already support passing images via URL + local storage for gemini ai studio.

The missing one is blob, but i don't any code examples of how blob data is processed / passed for ai studio.

Can you point me to a relevant code example on this?

There is no documentation on how to access local files I checked this VertexAI Documentation couldn't find anything here.

It says this.

LiteLLM Supports the following image types passed in url

Images with Cloud Storage URIs - gs://cloud-samples-data/generative-ai/image/boats.jpeg Images with direct links - https://storage.googleapis.com/github-repo/img/gemini/intro/landmark3.jpg Videos with Cloud Storage URIs - https://storage.googleapis.com/github-repo/img/gemini/multimodality_usecases_overview/pixel8.mp4

No mention of local storage

krrishdholakia commented 6 months ago

Great point - added here https://docs.litellm.ai/docs/providers/gemini

haseeb-heaven commented 6 months ago

Great point - added here https://docs.litellm.ai/docs/providers/gemini

Thanks. But need more documentation on code we need different code for different modes offline files and online files how to access then via Gemini Vision

krrishdholakia commented 6 months ago

good idea - feel free to contribute any examples to docs 😄

haseeb-heaven commented 6 months ago

good idea - feel free to contribute any examples to docs 😄

@krrishdholakia @ishaan-jaff Check this PR https://github.com/BerriAI/litellm/pull/1370

BerriAI / litellm