search

haotian-liu / LLaVA

[NeurIPS'23 Oral] Visual Instruction Tuning (LLaVA) built towards GPT-4V level capabilities and beyond.
https://llava.hliu.cc
Apache License 2.0
19.17k stars 2.1k forks source link

[Question] Possibility for Multi-image input? #197

Open jsg921019 opened 1 year ago

jsg921019 commented 1 year ago

Question

I really enjoyed reading this paper and I have played with it few days and came up to this question: LLava Architecture is capable of having more than one image input. I tried give two images as an input, but the inference result was not good. That is probably the model was not trained with multiple image input. Have you tried training LLaVA with dataset that has more than one input??

haotian-liu commented 1 year ago

Hi @jsg921019

Thank you for your interest in our work. Due to the current way of training, we do not observe the model having very good capability referring to / comparing with multiple images. We are working on improving this aspect as well, stay tuned! You can also checkout some of the discussions here.

codybum commented 1 year ago

This is also of great interest to our group as well. We work with pathology images and it can take more than one image to describe a region of interest due to image size. In this case we don't need to compare images, but allow several images to represent one thing. This would be similar in concept to MIL modeling (https://github.com/Project-MONAI/tutorials/tree/main/pathology/multiple_instance_learning).

haotian-liu commented 1 year ago

@codybum Thanks for explaining this interesting direction! I am curious about if there is any plan in your group in working in this direction? Would be happy to integrate that into LLaVA :)

sskorol commented 10 months ago

I am also looking forward to seeing this feature soon. For instance, GPT-4V can take several images and find relationships between objects on different images. It's pretty cool and has a variety of use cases.

shure-dev commented 10 months ago

I have a strong interest in this topic I want to know if this is possible or not for the current version by customizing the code

unnikrishnanrnair commented 9 months ago

@haotian-liu In my understanding GPT4v slices higher resolution images into 512x512 images plus one context image and then tokenizes + collates those tokens. Have you tried something like this with the latest LLaVA model by any chance? Is it something worth trying? My use case is simlar to @codybum 's where I need to pass in higher resolution images.

shure-dev commented 4 months ago

https://tiger-ai-lab.github.io/Blog/mantis