search

Luodian / Otter

🦦 Otter, a multi-modal model based on OpenFlamingo (open-sourced version of DeepMind's Flamingo), trained on MIMIC-IT and showcasing improved instruction-following and in-context learning ability.
https://otter-ntu.github.io/
MIT License
3.56k stars 241 forks source link

Plans to release Flamingov2 model weights #223

Closed vishaal27 closed 1 year ago

vishaal27 commented 1 year ago

Hey, thanks for your great contributions. I saw a comment here (https://twitter.com/BoLi68567011/status/1684028048998559747) that you have trained Flamingo-Llama2-Chat7B and Flamingo-MPT7B models. However, unfortunately I couldn't find the corresponding weights for these models on HF: https://huggingface.co/luodian Are you planning to release them soon, or if they are already released please let me know where I can access them :)

Luodian commented 1 year ago

hi the flamingo one is there~ https://huggingface.co/luodian/Flamingo-Llama2-Chat7B-CC3M/tree/main

vishaal27 commented 1 year ago

Thanks for the prompt response, can we simply load it like this?

from flamingo.modeling_flamingo import FlamingoForConditionalGeneration
flamingo_model = FlamingoForConditionalGeneration.from_pretrained("luodian/Flamingo-Llama2-Chat7B-CC3M", device_map=auto)

Or would the model class still have to be OtterForConditionalGeneration?

Luodian commented 1 year ago

Yes, it's FlamingoForConditionalGeneration (but the two are compatible now), the prompt template is:

prompt = "<image>an image of"

or simply

prompt = "<image>"
vishaal27 commented 1 year ago

Awesome, thanks so much!