search

johndpope / MegaPortrait-hack

Using Claude Opus to reverse engineer code from MegaPortraits: One-shot Megapixel Neural Head Avatars
https://arxiv.org/abs/2207.07621
82 stars 8 forks source link

loss target #27

Closed ChenyangWang95 closed 5 months ago

ChenyangWang95 commented 6 months ago

Hi, thanks for the impressed work!

I have a question about the loss. why the losses are calculated between source_frame and output_frame.

image

It may be the difference between output_frame and driving_frame?

image

johndpope commented 6 months ago

i switch the loss to use driving in this branch - https://github.com/johndpope/MegaPortrait-hack/pull/28

johndpope commented 5 months ago

happy to look at a PR

in mean time - i switch this back because the generator seems to focus on driving video exclusively - disregarding the source video. Screenshot from 2024-06-02 06-55-22

Screenshot from 2024-06-02 06-57-01

ChenyangWang95 commented 5 months ago

It seems that the source frame and driving frame are sampled from the same video in the paper. If I understand correctly, maybe the perceptual and adversarial loss are still calculated between driving_frame and output_frame?

image image

Also, where can I download 512x512 video? I can only get 256x256 voxceleb2.

johndpope commented 5 months ago

i overhaul training https://github.com/johndpope/MegaPortrait-hack/issues/32

https://github.com/johndpope/MegaPortrait-hack/pull/33

30gb torrent here - https://academictorrents.com/details/843b5adb0358124d388c4e9836654c246b988ff4

ChenyangWang95 commented 5 months ago

got it! thx!

johndpope commented 5 months ago

@ChenyangWang95 - please check - logic has been updated - and there's a lot preprocessing work to warp and crop - but it's not converging...... PR https://github.com/johndpope/MegaPortrait-hack/pull/33

UPDATE - i think I make some progress... will update in a few hours once i get through some training epochs....

johndpope commented 5 months ago

i think i got it. training now. i merged to main.