search

OSU-NLP-Group / MagicBrush

[NeurIPS'23] "MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing".
https://osu-nlp-group.github.io/MagicBrush/
Other
313 stars 14 forks source link

ckpt in HuggingFace #22

Open mo230761 opened 1 month ago

mo230761 commented 1 month ago

I really like your work！ But I found that the results using the HuggingFace version are not as good as when using the source code. Upon further exploration, I discovered that the HuggingFace version might not be using EMA. Is that the case? If so, how can I use the EMA version of the checkpoint with HuggingFace?

drogozhang commented 4 weeks ago

We were using EMA. Please see fine-tune.yaml

mo230761 commented 4 weeks ago

Thank you for answering my questions！！ I would like to know how you convert a ckpt file into a format that can be used by Hugging Face.

drogozhang commented 4 weeks ago

That's actually a very good question haha. Amazing @vinesmsuic helped deploy it so I don't really know. Maybe @vinesmsuic can point us to somewhere?

mo230761 commented 4 weeks ago

Thank you again for your answer！！！ And I would like to ask if @vinesmsuic could provide some guidance? If possible, I would be extremely grateful.

vinesmsuic commented 4 weeks ago

Hi @mo230761 , I don't exactly remembered how did I convert the weights back then since it has been a while. It could be possible that I missed the EMA option. But IIRC diffusers port are performing a bit worse, even for InstructPix2Pix.

Would you mind helping me to test the performance of this? https://huggingface.co/vinesmsuic/magicbrush-ema

mo230761 commented 3 weeks ago

Of course! I’ve also run some data using the EMA version you provided. Its results are indeed slightly worse than the original, but it’s better than the previous ported version that didn’t use EMA. It’s a pity that you don’t exactly remember how you converted the weights back then.

vinesmsuic commented 3 weeks ago

@mo230761 it would be great if you can share some visualizations.

mo230761 commented 3 weeks ago

This might not be possible, as the data I’m testing is under a confidentiality agreement. My apologies.