How to get correct data for vanishing points

[ThePedroboy]

Hi,

I have been trying to make this very intersting SIHE algorithm work but now I feel like I am stuck a bit. To be more specific, I can't really find a way to generate correct data for vanishing points.

In your ReadMe you say that for getting data of vanishinig points, the NeurVPS algorithm should be used. Additionaly, in the ReadMe, you say that the newly trained model should be used - based on this I assume that for getting data of the vanishing points, the method SU3 should be used in the NeurVPS algorithm - because the exactly same model is also used by default in the NeurVPS, SU3 method with Conic_4X settings (correct me please if I am wrong with this assumption).

And now we are getting to my issue. NeurVPS runs fine on my computer (all 3 methods work). After the examination of the images data that is being used for the 3 methods (SU3, ScanNet, TMM17) - every method needs data in a different way. And since I assumed that the SU3 method should be used I looked more closely on data for that one. For every image that is being processed by this method there is a config file of camera settings (_camera.json). I take it that this file was generated along with the image (since all the images used for SU3 are synthetic). This config file contains information such as camera location, positions,... etc. And I don't know how to get this config file for my images that I take from street view maps.

I also tried using the TMM17 method in the NeurVPS algorithm but the output data seems to be insufficient. I get to the point that when I generate the .npz file from NeurVPS and try to plug it into the SIHE, the algorithm expects 3 points with x and y coordinates in the vpts_re variable - but there is only 1 point.

Could you please correct my approach to integrating the SIHE algorithm or point me in the right direction? Or could you maybe elaborate a bit on how did you get this _camera.json file for your example images please? Any help would be greatly appreciated.

[yannick-reich]

Hi @ThePedroboy , perhaps this is not a satisfying answer, but I use vanishing points computed from the pitch. If you get your own svi, you can demand a specific pitch. The pitch is hardcoded somewhere in the code. You'd just need to set it to your value.

[ThePedroboy]

Hi @yannick-reich , thanks for your answer. I will definitely try that. Did it work okay for you? Like was the height estimation accurate?

Btw. where were you taking the street view images from? I was quite struggling to find a good source (with the pitch infomation for example) because I want some really easy to use source where the user would just get the desired image within seconds.

[djoshi1000]

@ThePedroboy Did you find the answer to your question?

[yannick-reich]

@ThePedroboy when you get them through the Street View API you can set a pitch parameter. If you don't, a "suitable" pitch will be chosen for you. I haven't seen results yet, but I'm not sure there's even any error in computing the vertical vanishing point from the pitch.

[yzre]

Hi，

For the problem, I have uploaded two new files in the ‘./misc/vps_models/‘ folder for your reference. One is the config file (config.yaml) and it is similar to the SU3 method with Conic_4X settings (modified from its config file). However, the correponding dataset loading codes need to be modified for loading custom data.

Another file contains the methods to transform the 3D coordinates of vpts (vanishing points) to 2D image coordinates. I think this may be the key of the problem. The direct output from neurvps are 3D coordinates of the vpts, but in the later height calculation their 2D form in image coordiates are used. I hope the uploaded “vpt_transform.py” can be of help.

In addition, the above answer is right. The vpts computed from the pitch can also be directly used in the height calculation, and the pitch parameter can be set and known using the street view api. You can try this by setting the ‘pitch’ value in the ‘heightCalc() ‘function and set ‘use_detected_vpt_only’ to 1.

[ThePedroboy]

Thanks a lot you guys.

I will definitely check the config file, thank you for uploading. This also helps because of the fact that at first I wasn't 100% sure that you used the SU3 method, will look into that. The conversion from 3D coordinates to 2D image coordinates wasn't an issue for me - I think that in neurvps itself is a method that does the job. And yeah, I will try to get it from street view api with the pitch information and try to use that as well.

I got to a different project recently but when I get back to this, I will let you know how it works.