search

ziyc / drivestudio

A 3DGS framework for omni urban scene reconstruction and simulation.
https://ziyc.github.io/omnire/
MIT License
599 stars 51 forks source link

About affine transformation #12

Closed Korace0v0 closed 2 months ago

Korace0v0 commented 2 months ago

Hi, thanks for your great work.

I am confused about the `Affine' model, which is used to transfrom 2D rgb image. I could not understant it's effect. Why do we really need this transformation?

It seems affine transformation is only applied to rgb_sky_blend, while depth is not transformed. Depth is then supervised by Lidar. As I can understand, if Affine model is used to transfrom 2D image, does depth also need to be transformed to align with rgb?

Thanks for your help!

ziyc commented 2 months ago

Hi @Korace0v0, This is a very good question.

Affine transformation was used in Urban Radiance Field (section 4.1.1) to handle inconsistent exposure of images.

A similar exposure situation occurs in driving data. For images within the same sequence, the exposure, color balance might differ across cameras and frames. These inconsistencies could hurt the reconstruction quality. The affine transformation is used to encode camera-specific features related to color balance, exposure etc. This way, the inconsistent exposures won't be fitted by the scene representation.

Korace0v0 commented 2 months ago

I understand it. Thank you very much!

pierremerriaux-leddartech commented 1 month ago

Hi @ziyc , i think there is a issue with Affine module gradient propagation, due to 0 mlp init. I proposed a PR here https://github.com/ziyc/drivestudio/pull/38 thanks Pierre