Explore the SD inversion to preserve the detail instead of using controlnet

pureexe / sd-light-time

Adding light condition at time

0 stars 0 forks source link

Explore the SD inversion to preserve the detail instead of using controlnet #5

Open pureexe opened 3 weeks ago

pureexe commented 3 weeks ago

Since we discussed on #4
We can use inversion code to inverse network

pureexe commented 2 weeks ago

With the Inversion, we found that the condition is somehow reconstructed back to the dog, even though the network didn't feed the dog. Probably this is overfitting on training.

After the discussion we may try to use depth or normal to help network to disentangle the lighting data.

pureexe commented 2 weeks ago

I confirmed that this is not the input bug. The model already see the corrected input during DDIM inversion

pureexe commented 2 weeks ago

The previous comment is incorrect. It actually the input bug. but it bug on the prompting part

pureexe commented 2 weeks ago

Update: with fixed prompt: the image still doesn't relight (Source image show light coming from left direction while target light is coming from the right)