2021 (EG3D) Efficient Geometry-aware 3D Generative Adversarial Networks

[pomelyu]

Introduction

This paper proposed an efficient way for high quality(512x512) multi-view-consistent GAN trained on 2D data that can be ran on realtime.

Method

• This method combined tri-plane neural rendering and StyleGAN structure
• tri-plane neural represent
  • use 2D convolution to create three orthogonal feature plane (DxDxC)
  • sample the feature depends on the xyz coordinates of the points, and sum up these feature (N, C)
  • use two layer MLP to get color(32) and density(1)
  • perform neural rendering to get the image feature map(H, W, 32)
• super resolution module(StyleGAN)
  • conditioned on camera pose
  • decoder the feature map to final RGB image
• Dual discriminator
  • concatenate the super resolution output and the first three feature map of neural rendering(low-resolution) as 6-channel input
  • conditioned on camera pose

Highlight

• realtime
• tri-plane is a very efficient 3D representation and can utilize the merits of 2D convolution
• dual discriminator focus on the outputs of both neural rendering and decoder help the multi-view consistency
• robust for the imprecise or even noisy camera estimation^1
• using camera pose as condition is also the key to multi-view consistency
• pose condition can handle the unbalanced head-expression problems
• good extrapolation to steep camera angles
• can use PTI[^2] for image inversion

Limitation

• pose condition can not fully disentangle head pose and expression
• eye direction is not consistent. It will wrongly follow the camera position

Comments

[^2]: Pivotal Tuning for Latent-based editing of Real Images