Closed AsadMir10 closed 11 months ago
Hi, this is a good point. This work has arbitrary modifications to the model architecture. I am not sure it can work smoothly with the standard Hugging Face model library. I may try it hopefully in the next two weeks.
Great, will be waiting for it to be on air soon. Anyhow I was also wondering about the role of Blip2 in the stack and from what I understood you are just using Blip2 for feature extraction and than creating a embedded dataset and meanwhile the real meat inside the architecture is using T5 multimodal, correct me if I'm wrong, and thanks for the update!
Yes. We use BLIP2 just as the feature extractor. A projection layer is used to adapt the vision features to the T5 encoder-decoder architecture.
Great, Understood.
I just wanted to use your model in the Hugging Face model library but I don't see any model usage definitions, will you be defining any usage instructions or model card any time soon?