LLaMA2-Adaptor x Region Demo setting

[erjui]

Hi, first of all, thanks for the nice work.

The demo of the multi-modal llama2-adaptor and region is really nice that it even finds bounding boxes of the given image. However, when I tried the same thing on the llama2-adaptor demo page, it seemed the model did not give a proper answer. (http://llama-adapter.opengvlab.com/)

So I wonder if there is a difference when training/fine-tuning the llama2-adaptor model on multi-modal visual instructions.

Thanks in advance :)

[Artanic30]

Hello, thank you for your interest in our work. At the moment, the demo model does not incorporate bounding boxes, which means it cannot generate location information as requested in the prompt. However, we are actively working on the llama2-adaptor with localization capabilities, and we will eagerly share the intriguing demo with you as soon as our research is complete.

[erjui]

Thanks a lot for the answer! However then, I'm very curious about how you incorporate bounding box localization from the prompt. Could you tell me how you integrated such localization ability by any chance??

[gaopengpjlab]

We will release the training pipeline for incorporating box into LLM as soon as possible. Stay tuned.

[erjui]

Thanks a lot for the answer! 👍

[gaopengpjlab]

Please try SPHINX which shows strong performance on object spotting.