search

danielroich / PTI

Official Implementation for "Pivotal Tuning for Latent-based editing of Real Images" (ACM TOG 2022) https://arxiv.org/abs/2106.05744
MIT License
897 stars 112 forks source link

experiments with throwing many images at decoder (via ffmpeg) #27

Closed johndpope closed 2 years ago

johndpope commented 3 years ago

I was thinking this would be fit for ffmpeg to handle this in an image sequence. The code has very support for multiple images - this would be interesting to explore.

danielroich commented 3 years ago

Do you mean tuning the generator based on different images of the same person from a video?

johndpope commented 3 years ago

yes. - one person. or is that not going to work?

(I think I've worked out the pkl problem - I'm using the e4e generator to inject in - I need to swap in the pti generator and save that instead. I just haven't gotten around to it. I also have doubts with the g_ema findings....)

danielroich commented 2 years ago

The thing about StyleGAN's latent space which I have noticed while researching the inversion problem is that if you have the same identity, but with different backgrounds - the corresponding latent codes will reside in different places in the W space. Hence, I think that using the same person and the same background will result in even better reconstruction (maybe it will enable the tuning to be faster?) For the same person and different backgrounds, PTI will handle it like 2 different identities.

Most of the time in a video the background remains the same, so it seems PTI will be even better.

Notice one thing though - if you take the first frame and receive a new tuned StyleGAN using PTI. it will already contain the identity you want. Meaning the 10th frame will already be in the W space even if you never used PTI on it. You can imagine the difference between the 10th frame and the first frame to be an edit. This explains why every image of the same identity and background is already in the tuned generator and does not require new tuning.

Hope it answers your question, I have wondered a bit in my answer to explain even more stuff which might be helpful

johndpope commented 2 years ago

fyi - http://arxiv.org/abs/2201.08361