search

vllm-project / vllm

A high-throughput and memory-efficient inference and serving engine for LLMs
https://docs.vllm.ai
Apache License 2.0
29.69k stars 4.48k forks source link

[Usage]: set num_crops in LVLM #7861

Closed Liyan06 closed 1 month ago

Liyan06 commented 2 months ago

How to set up the num_crops for LVLMs? For example, when initializing the processor for Ph-3.5-vision-instruct, the hugging face code looks like the following: 

processor = AutoProcessor.from_pretrained(model_id, 
  trust_remote_code=True, 
  num_crops=4
)

But I didn't find a way to set num_crops in vllm.

I checked the pull request #7710, but I didn't find the solution.

DarkLight1337 commented 2 months ago

Currently there is no way to pass options to the processor directly. Any help with that is welcome!

alex-jw-brooks commented 2 months ago

Hi @Liyan06, are you planning to submit a PR for this? I'm interesting in being able to enable different image processor options for different models as well, happy to take a pass at adding this with num_crops for phi3v as an example if you aren't