Open Johnson-yue opened 7 months ago
When I trained two loras, content_lora and style_lora. Then I train ziplora for compose those, but loss3 is very very diffcult for training. I have updated 'compute_mergers_similarity' function :
def compute_mergers_similarity(self): return (self.merger_1 * self.merger_2).abs().mean()
and fixed GPU memory issue. I can use 24G 3090 train 300 steps with train_ziplora.py but loss3 from 5.6 only decrease 5.54
In paper the loss3 maybe decrease to 0 , but this implement it will be ?
When I trained two loras, content_lora and style_lora. Then I train ziplora for compose those, but loss3 is very very diffcult for training. I have updated 'compute_mergers_similarity' function :
and fixed GPU memory issue. I can use 24G 3090 train 300 steps with train_ziplora.py but loss3 from 5.6 only decrease 5.54
In paper the loss3 maybe decrease to 0 , but this implement it will be ?