🦦 Otter, a multi-modal model based on OpenFlamingo (open-sourced version of DeepMind's Flamingo), trained on MIMIC-IT and showcasing improved instruction-following and in-context learning ability.
I want to give some images to the model as an in-cotext, then input the video and ask questions about the video content.
(Specifically, I would like to teach the model the type of dogs as images and then have the model count the number of dogs in the video.)
The Otter-image model can be given an image as context, but no video can be input.
And, the Otter-video model cannot be given an image as context, but video can be input.
Is there an optimal implementation method or model for this type of situation?
I want to give some images to the model as an in-cotext, then input the video and ask questions about the video content. (Specifically, I would like to teach the model the type of dogs as images and then have the model count the number of dogs in the video.)![multimodal](https://github.com/Luodian/Otter/assets/43651940/b07b3810-5eb9-42d2-8ffb-00851477e284)
The Otter-image model can be given an image as context, but no video can be input. And, the Otter-video model cannot be given an image as context, but video can be input.
Is there an optimal implementation method or model for this type of situation?