search

pureexe / sd-light-time

Adding light condition at time
0 stars 0 forks source link

Make code support SD inversion to preserve the detail instead of using controlnet #5

Open pureexe opened 2 months ago

pureexe commented 2 months ago

Since we discussed on #4
We can use inversion code to inverse network

pureexe commented 2 months ago

With the Inversion, we found that the condition is somehow reconstructed back to the dog, even though the network didn't feed the dog. Probably this is overfitting on training.

After the discussion we may try to use depth or normal to help network to disentangle the lighting data.

image

pureexe commented 2 months ago

I confirmed that this is not the input bug. The model already see the corrected input during DDIM inversion

image

pureexe commented 2 months ago

The previous comment is incorrect. It actually the input bug. but it bug on the prompting part

image

pureexe commented 2 months ago

Update: with fixed prompt: the image still doesn't relight (Source image show light coming from left direction while target light is coming from the right)

image

pureexe commented 2 months ago

Update: Our code is now already support the Inversion