search

salesforce / HIVE

Apache License 2.0
91 stars 11 forks source link

HIVE does not work as good as InstructPix2Pix #6

Open sanchit88 opened 1 year ago

sanchit88 commented 1 year ago

Hi Authors, recently I am doing a comparison between HIVE and InstructPix2Pix. To my surprise HIVE is not working that good as compared to InstructPix2Pix. It is even quite off compared to it. I am really wondering if this is really the case or am I missing something. I have tested both condition and weighted based models with SD 2.1 as backbone. Both are not good and weighted based is quite even worse than condition based. And, in the paper figure 20 it shows that they are not supposed to be too different. I am testing with simple instructions even. Can you please help me here what is going on? Thanks a lot!

shugerdou commented 1 year ago

@sanchit88 Thank you for your interest on our work. Could you please provide some examples so that we can take a deep look? From our released evaluation data, we obtained higher votes using Mturker. We did not observe what you said. We are updating the model as well, which will be released soon. We are always happy to improve our model. Thanks!

sanchit88 commented 1 year ago

Hi @shugerdou, thanks a lot for your reply. Sure, very happy to help you guys. Can you please provide me your email ID? I will rather send the data there and also correspondingly InstructPix2Pix for the better look. I would say approach that you guys followed (human in the feedback + its retraining) caught my attention. May be you need to evaluate HIVE from out of distribution (OOD) data to check its generalization?

shugerdou commented 1 year ago

@sanchit88 Sure. Please email to shu.zhang@salesforce.com. We will evaluate the performance on your data and improve the model. Thanks again for your interest in our work!