Intrinsics used for Cityscapes

lhoyer / improving_segmentation_with_selfsupervised_depth

[CVPR21] Implementation of our work "Three Ways to Improve Semantic Segmentation with Self-Supervised Depth Estimation"

247 stars 30 forks source link

Intrinsics used for Cityscapes #8

Closed nbansal90 closed 3 years ago

nbansal90 commented 3 years ago

Hey @lhoyer ,

I think there might be a mistake here while using the intrinsics. Please let me know if my understanding it wrong here. In the CityscapesLoader class we have the intrinsics as the average of all intrinsics for cityscapes dataset ( I presume) which by default is set to: self.fx = 2262.52 self.fy = 2265.3017905988554 self.u0 = 1096.98 self.v0 = 513.137 and through out the code, for different scales we change the intrinsics accordingly.

But as preprocessing we also resized the image from (2048, 1024) to (1024, 512). So shouldn't the default intrinsics be half of what is defined in loader/cityscapes_loader.py

Regards, Nitin Bansal

lhoyer commented 3 years ago

Hi Nitin,

Yes, you are right about the missing scale factor for the focal length. We are aware of this error but kept it as the network can deal with it and in order to have the source code consistent with the provided pretrained depth models and the CVPR results. Talking about bugs in the self-supervised depth estimation, you also might want to consider removing the following line: https://github.com/lhoyer/improving_segmentation_with_selfsupervised_depth/blob/1a6f4a88863602807d4192072ce5527c97a5b213/loader/sequence_segmentation_loader.py#L337 as there is only a horizontal flip.

We will soon publish an extension of this project, which will include the mentioned bug fixes.

Best regards, Lukas

nbansal90 commented 3 years ago

Hi @lhoyer ,

Thank you for clarifying and also (an imminent )publishing an extension of this project!

I was just looking to run an experiment from my end, and I was wondering what changes shall I make from my end. I have the following list of things which needs to be changed. Please let me know if I am missing something, or If I am doing something wrong.

Change the default intrinsics from self.fx = 2262.52 self.fy = 2265.3017905988554 self.u0 = 1096.98 self.v0 = 513.137 to self.fx = 1131.26 self.fy = 1132.6508953 self.u0 = 548.49 self.v0 = 256.5685, which is half the intrinsics provided.
Remove v0 = self.full_res_shape[1] - v0.
Should self.full_res_shape[1] we replace it by self.full_res_shape[1]/2
If I missing any other changes please let me know.

Thank You!

Regards, Nitin Bansal

lhoyer commented 3 years ago

Hi Nitin,

Yes, that looks good.

Best, Lukas