when Lora model passed to peft(0.10.0) ,peft dosen't solve to_out.0 this layer, so we can't train this layer. The details are, to_out is a Sequential, the code in peft try to set attribute 0 of the sequential, but actually, it only set an new attribute for this Sequential, but real "0"layer is in the layers attribute of sequential is not replaced by lora.
To solve this problem, I can only modify the peft to adapt the model passed by jdiffusion.
when Lora model passed to peft(0.10.0) ,peft dosen't solve to_out.0 this layer, so we can't train this layer. The details are, to_out is a Sequential, the code in peft try to set attribute 0 of the sequential, but actually, it only set an new attribute for this Sequential, but real "0"layer is in the layers attribute of sequential is not replaced by lora.
To solve this problem, I can only modify the peft to adapt the model passed by jdiffusion.