search

johndpope / MegaPortrait-hack

Using Claude Opus to reverse engineer code from MegaPortraits: One-shot Megapixel Neural Head Avatars
https://arxiv.org/abs/2207.07621
68 stars 7 forks source link

Data prep - calculate 3d keypoints #3

Closed johndpope closed 4 months ago

johndpope commented 4 months ago

Since we predict 3D instead of 2D keypoints, We use an orthographic projection to project the keypoints to the image plane before computing the loss. https://ar5iv.labs.arxiv.org/html/2011.15126

Screenshot from 2024-05-13 14-19-12

https://github.com/1adrianb/face-alignment Screenshot from 2024-05-13 21-34-09

francqz31 commented 4 months ago

@johndpope have you tried implementing the paper using gpt4-o ? it looks better than opus

johndpope commented 4 months ago

Just playing now on iPhone I did through paper into prompt - it confidently spat out garbage. It might be great to hone in on aligning the diagrams to code (see reference folder - Claude got this wrong had to correct it) - will test later.

Also I did update training code with a face alignment - 2d points - can adjust later 3d - I need to start debugging the training but looks much better than prior work.

UPDATE it seems to truncate file uploaded - can't handle the context window like Claude - 200k waste of time. Screenshot from 2024-05-14 14-10-09