search

google / generative-ai-docs

Documentation for Google's Gen AI site - including the Gemini API and Gemma
https://ai.google.dev
Apache License 2.0
1.69k stars 608 forks source link

Optimizing PaliGemma to TFLite / TorchScript #470

Open DarioPTWR opened 4 months ago

DarioPTWR commented 4 months ago

Description of the feature request:

Hi, I was wondering if anyone has a conversion script or method that can be used to convert PaliGemma from HuggingFace Transformers to TFLite or TorchScript? Am trying to use it in mobile devices but currently can't seem to find a method that works 100% for this model conversion. Any help is greatly appreciated, thanks!

What problem are you trying to solve with this feature?

No response

Any other information you'd like to share?

No response

haozha111 commented 3 months ago

hi @DarioPTWR ,

please take a look at the AI Edge Torch generative API : https://github.com/google-ai-edge/ai-edge-torch/tree/main/ai_edge_torch/generative

DarioPTWR commented 2 months ago

hi @haozha111 have taken a look and tried it on my own but seemed to face significant errors in converting the model's layers, are there any working solutions for PaliGemma available that i could reference / get more help in detail? also seems like the generative API now only has working examples for LLMs, will it work for VLMs as well? Thanks and greatly appreciate

haozha111 commented 2 months ago

hi @DarioPTWR

Yes we are actively working on adding multi-modal support for Generative API, and we aim to provide an update on this topic in the near future, thanks!

DarioPTWR commented 2 months ago

hi @haozha111 thanks for your update! looking forward to the added support, but also want to ask if you are familiar with / aware of any other existing methods to efficiently run PaliGemma (or any VLM) using on-device inference in Android? preferably in TFLite format. thanks!

yichunk commented 2 months ago

Hi @DarioPTWR

We are actively working on the image modality support for VLM. At the same time, you can also stay tuned for AiCore from Android https://developer.android.com/ai/aicore, which may provide VLM integration in the near future as well.