search

TIGER-AI-Lab / Mantis

Official code for Paper "Mantis: Multi-Image Instruction Tuning"
https://tiger-ai-lab.github.io/Mantis/
Apache License 2.0
177 stars 15 forks source link

I tried to deploy Mantis in my own server for some test. Do you have any suggestion about the tools which can deploy Mantis to run faster? #15

Closed BrenchCC closed 1 month ago

BrenchCC commented 2 months ago

I am trying to add Mantis to the supported model list in VLLM or Sglang

jdf-prog commented 2 months ago

For Mantis-Idefics2, it's literally the same as How Idefics2 is used. I found a existing issue in VLLM project that there are some people trying to implement Idefics2 in VLLM, I think probably you can try comment there to see what's the current progress.

For Mantis-CLIP and Mantis-SigLIP, they shall similar architectures with llava-1.5, which should be supported by VLLM now. You can try if that will work.

Inference acceleration is an important issue. I will also try to see if mantis can be integrated into VLLM in the near future.

BrenchCC commented 2 months ago

I had verified that Mantis-CLIP can be deployed by VLLM. But Mantis-SigLIP has a problem as follows: In the config of Mantis-SigLIP, image_size = 384 and patch_size = 14, but in VLLM , siglip.py requires a divisible relationship, and the 'LlavaForConditionalGeneration' in Mantis has difference compared with VLLM.