search

opea-project / GenAIExamples

Generative AI Examples is a collection of GenAI examples such as ChatQnA, Copilot, which illustrate the pipeline capabilities of the Open Platform for Enterprise AI (OPEA) project.
https://opea.dev
Apache License 2.0
274 stars 192 forks source link

Update VisualQnA example with Falcon VLM #561

Open arun-gupta opened 3 months ago

arun-gupta commented 3 months ago

Update VisualQnA example that uses Falcon VLM.

This would require to include Falcon as part of the validation at https://github.com/opea-project/GenAIComps/tree/main/comps/llms. And then create an updated VisualQnA that would use this microservice to use Falcon VLM.

lucasmelogithub commented 3 months ago

Supporting Falcon-11B would be great.

kevinintel commented 2 months ago

TGI-Gaudi did not support this model: https://huggingface.co/tiiuae/falcon-11B-vlm

we need to wait for TGI-Gaudi

chickenrae commented 2 weeks ago

@kevinintel can we move forward with this issue now?

kevinintel commented 2 weeks ago

we can't do anything unless tgi-gaudi supports it

arun-gupta commented 2 weeks ago

@kevinintel can this be done using a larger Xeon instance?

lucasmelogithub commented 2 weeks ago

@kevinintel can this be done using a larger Xeon instance? image

VisualQnA works great on Xeon, I tested with a AWS "c7i.24xlarge" instance (96 vCPU 4th Gen Xeon w/ Intel AMX) using the llava-hf/llava-v1.6-mistral-7b-hf Model.

But when using tiiuae/falcon-11B-vlm, the "tgi-llava-xeon-server" microservice errors out.

chickenrae commented 2 weeks ago

@kevinintel What are the next steps to resolve this issue?