search

cdcseacave / openMVS

open Multi-View Stereo reconstruction library
http://cdcseacave.github.io
GNU Affero General Public License v3.0
3.38k stars 911 forks source link

Camera Model with Skew Parameter #539

Open SBCV opened 4 years ago

SBCV commented 4 years ago

Hi cdcseacave,

I'm currently working on texturing 3D models computed from satellite images. I've used this colmap fork to compute the reconstruction. Since satellite sensors are typically represented by (complex) pushbroom cameras, the authors of the fork approximate this with a camera model containing the following parameters fx, fy, cx, cy and s (s is the skew of the pinhole camera). Now, I've seen that InterfaceColmap only supports fx, fy, cx and cy (which makes sense, since the original colmap version does not support camera models with skew). Since the code in MVS/SceneTexture.cpp is quite complex, I was wondering

Bests, Sebastian

cdcseacave commented 4 years ago

Hi Sebastian, Cool project you are working on. I'll check the paper to find out more. As for the skew param support, it is indeed ignored in some parts of the code like: https://github.com/cdcseacave/openMVS/blob/develop/libs/MVS/Camera.h#L148 but used in others, like: https://github.com/cdcseacave/openMVS/blob/develop/libs/MVS/Camera.h#L181

The needed changes should not be big, normally making sure https://github.com/cdcseacave/openMVS/blob/develop/libs/MVS/Camera.h class is considering the skew param should do it.

I do not have time to work on this right now, but I can guide you if needed. For instance I suggest to start by putting together a small test case that with some important skewness, complete the Camera model and see if the results are as expected.

SBCV commented 4 years ago

Hi, thanks for the quick reply. I'll check if I can create some (simple) artificial test case. (The satellite reconstructions are quite big and not suitable for debugging / testing) I'll come back to you and report the progress - thank you!

cdcseacave commented 4 years ago

Hi @SBCV, How is it going with the satellite reconstructions? Did you have time to work on this? I found some time to read the paper today. They propose also a way to undistort the images accounting for skewness as well, so in the end there might be no need for skew support in MVS.

SBCV commented 4 years ago

Hi, sorry for the late reply.

Since the satellite imagery does not provide a good playground, I was building a test scenario with VTK to create some synthetic images with different skew-values.

Thanks for pointing out that part in the paper. I've seen this as well. In the first place I wanted to avoid transforming the images, since this could potentially introduce some transformation artifacts. (For example: The areas that are part of the skew-free images, which have not been part of the original images). I was not sure how (or if) this would affect the performance of openMVS. But you have your point - probably it will be easier to apply the transformation directly to the images. I'll have a look and come back to you.

And thanks for your interest.

Bests Sebastian

SBCV commented 4 years ago

I got some decent results (see image below) by transforming the images (i.e. removing the skew) and adjusting the principal points.

openmvs_delauney_textured01

For comparison purposes, the following image shows the (poisson) reconstruction computed with the Colmap fork. Colmap associates each vertex in the poisson reconstruction with additional color information.

colmap_poisson00

SBCV commented 4 years ago

I further compared the mesh shapes of the colmap-poisson-reconstruction and the openmvs-delaunay-reconstruction. Both results use the dense point cloud computed by the colmap fork. This is necessary, since the depth reparametrization of Zhang et al. substantially improves the reconstruction results .

OpenMVS Mesh: openmvs jpg00

Colmap Mesh: colmap jpg01

Overlay (Colmap blue, OpenMVS green) colmap_vs_openmvs jpg02

SBCV commented 4 years ago

As one can see in the images above, the OpenMVS mesh is more smooth than the Colmap result. Is there a way to import a pre-computed mesh into OpenMVS? If not, could you give me an advice, where to add such functionality? (For example, do you think that ReconstructMesh would be an appropriate place or rather something like InterfacePLY?)

Bests, Sebastian

cdcseacave commented 4 years ago

the smoothness is not intrinsic to the meshing algorithm, actually the contrary is true; you have a param to disable the smoothing which is done after the mesh reconstruction and yes, there is a way to import an external mesh as long as both are in the same coordinate system, however instead you should fix the meshing in OpenMVS

SBCV commented 4 years ago

I adjusted the smooth parameter and it improved the results. However, the characteristics of both meshes are still quite different (which is not really surprising, since different methods have been applied).

For comparison purposes, I would like to use the same mesh geometry. How can I load the mesh computed by colmap (which is defined in the same coordinate system) into openMVS?

cdcseacave commented 4 years ago

use --mesh-file <mesh.ply> param

SBCV commented 4 years ago

Thank you for the hint, I was not aware of the hidden parameters! (I checked the options using the command line ... ). I'm not sure, if hiding these options is the most user friendly way. Maybe it would be better to show them as advanced options or something similar?

Btw: The decimate parameter is pure gold - it substantially reduces the mesh size while maintaining the mesh characteristics!

Thank you for your support!