johndpope / MegaPortrait-hack

Using Claude Opus to reverse engineer code from MegaPortraits: One-shot Megapixel Neural Head Avatars
https://arxiv.org/abs/2207.07621
82 stars 8 forks source link

How is the result now? #48

Closed JZArray closed 2 months ago

JZArray commented 5 months ago

same as the title

johndpope commented 5 months ago

i think the training strategy is erroneous - i'm limited to my single gpu - 3090 the idea i read in another more recent paper - which makes more sense to me just to skip the frames ahead - and have the neural net work out the warping. overfit to one model. get to rotate head / face / eyes - completely disentangle to one subject. i put a bunch of updates here - https://github.com/johndpope/MegaPortrait-hack/pull/47 but I'm more interested to create new architecture. the game is to disentangle - but seems like there were perhaps better ways from many years ago. https://arxiv.org/pdf/2008.02401

also the es - leakage - i haven't solved.

johndpope commented 2 months ago

i solved id leakage in this repo - https://github.com/johndpope/IMF

there can be a problem with the batch normalization where many images in batch contaminated things. https://wandb.ai/snoozie/IMF/runs/fyk2cfti?nw=nwusersnoozie

i create a seperate ticket.