search

EricLBuehler / mistral.rs

Blazingly fast LLM inference.
MIT License
4.41k stars 307 forks source link

Phi-3.5-vision-Instruct muliples images loading #795

Open Aveline67 opened 1 month ago

Aveline67 commented 1 month ago

How is it possible to load multiples images for Phi-3.5-vision-Instruct ?

And referencing them as Image ?

Maybe it is supported but now example to show how.

Aveline67 commented 1 month ago

By modifying code I was able to load 2 pictures and it seems to work

EricLBuehler commented 1 month ago

@Aveline67 can you please share the code? Phi 3.5 vision instruct can support multiple images, just add messages with the correlated image!

Aveline67 commented 1 month ago

I just did call multiple times .add_phiv_image_message() but I had to comment out the candle_core::bail!("Can only process one image per batch"); condition in mistralrs-core\src\vision_models\phi3_inputs_processor.rs

I am looking to create a proper PR, also some changes are needed in phi3.rs to ensure to pass all pictures dimensions

EricLBuehler commented 1 month ago

@Aveline67 I see! It looks like this should be a fix as well as what you mentioned. Please feel free to open a PR!

Aveline67 commented 1 month ago

I did try this one https://github.com/EricLBuehler/mistral.rs/pull/829

not sure about images sizes

kuladeephx commented 3 weeks ago

@Aveline67, were you able to use multiple Images? I f so pls let me know

Aveline67 commented 3 weeks ago

sure i did, i will publish the PR

kuladeephx commented 3 weeks ago

And I want to run multiple prompts at once, but not successful, if you have any idea regarding it, pls let me know