search

kohjingyu / fromage

🧀 Code and models for the ICML 2023 paper "Grounding Language Models to Images for Multimodal Inputs and Outputs".
https://jykoh.com/fromage
Apache License 2.0
466 stars 34 forks source link

can you give me a pre-trained weight file not pruning? #28

Closed seungwoo-Jang closed 9 months ago

seungwoo-Jang commented 9 months ago

Hello, I am a student studying AI. While looking for multimodal, I was attracted to the fromage model, so I am studying. I want to check after fine-tuning the fromage model toward fashion, but the prune weight you provided does not proceed to resume and the result value is too low if I train from the beginning. Please check if you can share the weight file for fine-tuning due to the lack of gcp credits provided for training from cc3m!

kohjingyu commented 9 months ago

Hi, what's the issue with initializing from the pruned weights? The model should be exactly the same assuming you also restored the pretrained weights from OPT/CLIP-ViT, as we don't change any other model params outside of the embedding matrix and linear layers.

seungu1108 commented 9 months ago

Oh, there's no problem. I thought it would be better to learn more personal datasets through train resume based on models in fromage_model/fromage_vis4 or fromage_model that are not prune, but would it be similar to TRAIN based on the OPT 6.7 / CLIP vit large model shown in the paper?

kohjingyu commented 9 months ago

Yes, the weights should be exactly the same because we do not train the CLIP-ViT or OPT-6.7B model weights.

seungu1108 commented 9 months ago

Thanks your reply !

seungwoo-Jang commented 9 months ago

If I can make model with own dataset, I will show you! Thanks!