About 3D warping operation

gorgeousdays commented 1 year ago

Hi, thanks for your awesome work.

I am trying to reproduce the paper, but I have meet some problems with 3D warping operation.Can you provide a detailed explanation of the 3D warping operation.

Thanks.

johndpope commented 7 months ago

old ticket - but can you review this https://real3dportrait.github.io/ their code has these modules from modules.real3d.facev2v_warp.network I believe that this builds from this paper - let me know otherwise.

johndpope commented 6 months ago

@gorgeousdays - i completely rebuild some code (using Claude Opus) https://github.com/johndpope/MegaPortrait-hack

I got stuck on warping too (but I'm close) https://github.com/johndpope/MegaPortrait-hack/issues/4

@neeek2303 is the paper diagram actually accurate?

https://github.com/johndpope/MegaPortrait-hack/blob/main/model.py#L286

why do you need 2048 channels ?

when I concatenate the values - I get these dimensions - but then it's supposed to magically become 2048 - I don't understand.

        vs, es = self.Eapp(xs) # Appearance encoder source
        Rs, ts, zs = self.Emtn(xs) # motion encoder source
        Rd, td, zd = self.Emtn(xd) # motion encoder driving

        # es shape: torch.Size([1, 512])
        # Rs shape: torch.Size([1, 3])
        # ts shape: torch.Size([1, 3])
        # zs shape: torch.Size([1, 50])

I've got 512 + 3 + 3 + 50?

is 50 dimensions for expression accurate?

UPDATE - with some help - this is working https://github.com/johndpope/MegaPortrait-hack/issues/36

neeek2303 / MegaPortraits

About 3D warping operation #2