search

szymanowiczs / viewset-diffusion

Implementation of `Viewset Diffusion: (0-)Image-Conditioned 3D Generative Models from 2D Data' (ICCV 2023)
99 stars 7 forks source link

Bug in LayerNorm? #1

Open feixh opened 1 year ago

feixh commented 1 year ago

Hi there,

I feel like this line is not correct: https://github.com/szymanowiczs/viewset-diffusion/blame/c25c94d310268b98ba4c07f6307b593cc644e568/model/unet_parts.py#L139

which, in my opinion, should be return (x - mean) / (var + eps).rsqrt() * self.g.

Or, did I miss something?

szymanowiczs commented 1 year ago

Yes, you're right - thanks for pointing out.

It's the same couple of lines down in LayerNorm2D. I inherited this bit from https://github.com/lucidrains/denoising-diffusion-pytorch/blob/05bbae8ce12ab85d1729ff7af11b0d20b0abad3f/denoising_diffusion_pytorch/denoising_diffusion_pytorch.py#L120C55-L120C55

and didn't notice.

That being said, the models train fine and my existing checkpoints use the formulation I have now, so I don't want to make the change just yet. I'll investigate if it affects perf if I get some time.