Closed ilovecv closed 1 month ago
Thank you for your interest!
For the released model, was it fine-tuned only with image tokens head or fully fine-tuned? From the paper, it looks like the first one.
Yes it is the first one.
in step 1: do we need to change modeling_chameleon.py?
Yes! You just have to commented out the code! The original modeling_chameleon.py disabled the image logits.
Thanks for your quick reply!
For fully fine-tuning, if we train both image head and text head, how to preserve the text generation and image understanding capability? Did you try any experiments or can you share any insights?
BTW, did you have results with 30B?
Thank you for your interest!
Fine-tuning Chameleon might be a bit tricky. Initially, we tried full parameter fine-tuning to facilitate image and multimodal generation from Chameleon, but it didn't really work. So, we decided to fine-tune only the image head, which did work. I suggest using Anole directly for further fine-tuning on your downstream tasks, as Anole already possesses full capabilities for multimodal understanding and generation.
We will be releasing the 30B version soon! Stay tuned!
Hi,
Thanks for sharing the awesome work!
For the released model, was it fine-tuned only with image tokens head or fully fine-tuned? From the paper, it looks like the first one.
For fully fine-tuning, https://github.com/GAIR-NLP/anole/blob/main/training/README.md in step 1: do we need to change modeling_chameleon.py?
I don't see any difference between original and modified.