visualization for 3d reconstruction pipeline

[NikolausDemmel]

There are a couple of visualization tools, e.g. for inspecting matches, or visualizing the final pointcloud and cameras with a C++ and a web-based viewer.

I am working on some alternatives/variations for the bundle adjustment optimization. To develop and experiment, it would be quite helpful to have an interative GUI that shows images with residuals (e.g. reprojection errors) as well as the current 3d model with points and camera poses. It should allow to interactively go through the different stages of incremental reconstruction. E.g. by adding new images to the current map one at a time, or running BA with specified parameters at the press of a button with the 2d / 3d visualization updating after every iteration to see the effect not just on the energy, but also on the state.

Is there something like this? Or something where this could be based on?

If not, I plan to create something for my own purposes. Likely I'll use https://github.com/stevenlovegrove/Pangolin as a GUI library since it is cross platform and geared towards visualizing exactly 2d image data as well as 3d scenes. Ideally after I have something work, I would love to contribute it back to openMVG. So to minimize future friction, do you think Pangolin would be an acceptable dependency? If you have reservations, can you suggest alternatives?

[pmoulon]

Hi @NikolausDemmel, Thx for your message.

I also think that now OpenMVG would benefit from GUI(s) and visualizer(s). As you said we already have some interfaces:

• matches and track export to SVG.
• OpenGL sfm_data viewer.
• QT matches viewer (adjacency and matches).

I agree that the project would benefit with new tools that allow more user interactions.

@rperrot is working on a GUI to launch the various tools of the SfM pipeline https://github.com/rperrot/openMVG/tree/OpenMVG-GUI

@rhiestan started Regard3D https://github.com/rhiestan/Regard3D

I think the following would be an answer to your need and would be greatly beneficial for the project: a sfm_data viewer. This way you could call this viewer at any step in a SfM pipeline.

I think it would be awesome to have a sfm_data viewer. Basically the tool would just give a view of the camera location and the point cloud. On top of that we could be some interaction and tools:

• interact with the cameras (picking) -> Feature visualization and reprojection vector or statistics
• interact with the 3d point (picking) -> Know from which view a 3d point comes from.
• add a representation of the view graph -> add edges between cameras if they share some observation of common 3d points (The edge color can be encoded depending on the number of points).

Note: Note that OpenMVG have different SfM pipeline (Sequential, Global, ... StructureFromKnownPoses) and so the stop and go that some other software are doing can only apply to sequential pipeline.

Regarding the used framework, I would prefer to stick to Qt and why not stick with Qt 3D. Using Qt allows mixing widget and so gives the ability to build more flexible interface. We can think about mixing many widgets together in the future (SfM_Data viewer, Matches viewer, ..., Intrinsic_data Editor...).

Let me know what you think about this.

[rperrot]

Hi,

I'm interested in all that can be made to improve user experience.

For now, the OpenMVG-GUI loads ply model for visualization purpose. Maybe we can think to work directly from the sfm_data file. If it works, you will need to export the sfm_data each time you make a modification then reload the file into the interface. That may be a "simple" approach to do it.

Regarding the framework, I agree with @pmoulon, I use Qt (and OpenGL) for OpenMVG-GUI, there is no need to use another framework (and we also want to limit the number of dependencies).

Regarding the various viewers, I need to understand what are exactly the features needed. @pmoulon @NikolausDemmel could we make an exhaustive list of all requested features and corresponding window behavior ?

[NikolausDemmel]

Thanks for your prompt and comprehensive responses. Seems like there is a lot of interest and work already going on. Very cool :smiley:.

Just to make it clear, what I am looking for in the short term is something I can use for 3d-reconstruction algorithm development. This shares many elements with a GUI geared towards the end user that wants to use the pipeline to generate 3D models from images, but the focus and audience are different. For example, I'm quite happy to, actually I prefer to run everything from command line, once I know the algorithms are working and what parameters I need.

@rperrot is working on a GUI to launch the various tools of the SfM pipeline https://github.com/rperrot/openMVG/tree/OpenMVG-GUI

Cool. Sounds like something I could maybe build-upon / contribute-to. Even though I'm not sure what the exact aim is with that yet and where development is at.

@rhiestan started Regard3D https://github.com/rhiestan/Regard3D

This seem aimed at the the end user, so maybe not directly usable by me. Even though, it might be helpful to get started e.g. with dense reconstruction, which I know little about yet.

I think the following would be an answer to your need and would be greatly beneficial for the project: a sfm_data viewer. This way you could call this viewer at any step in a SfM pipeline.

I think it would be awesome to have a sfm_data viewer. Basically the tool would just give a view of the camera location and the point cloud. On top of that we could be some interaction and tools:

• interact with the cameras (picking) -> Feature visualization and reprojection vector or statistics
• interact with the 3d point (picking) -> Know from which view a 3d point comes from.
• add a representation of the view graph -> add edges between cameras if they share some observation of common 3d points (The edge color can be encoded depending on the number of points).

Yes, this definitely sounds like the kind of thing I need, since the workflow I envision is maybe running some stages of the pipeline from command line, then opening stm_data with the GUI, performing additional steps, rerunning others, saving back to file to continue after code changes, etc...

All your mentioned tools / interactions sound useful. More on that below.

Note: Note that OpenMVG have different SfM pipeline (Sequential, Global, ... StructureFromKnownPoses) and so the stop and go that some other software are doing can only apply to sequential pipeline.

Yes I realize. Since I want to use it as a dev tool it is fine to focus on the things I need / want to work on. Of course ultimately it would be nice to cover all aspects of the framework and have different variants for different use cases.

I'm interested in all that can be made to improve user experience.

Very cool. Though to make sure we talk about the same thing, I see myself not really as a user but as a developer in this case.

For now, the OpenMVG-GUI loads ply model for visualization purpose. Maybe we can think to work directly from the sfm_data file. If it works, you will need to export the sfm_data each time you make a modification then reload the file into the interface. That may be a "simple" approach to do it.

For me the workflow of exporting to ply after every change does not sound like a good fit, since I want to interactively run individual steps of algorithms (like a single iteration of BA, or adding one image in incremental SfM) and see the effect right away. And I would like to save the current state (e.g. to sfm_data) and continue later.

Regarding the various viewers, I need to understand what are exactly the features needed. @pmoulon @NikolausDemmel could we make an exhaustive list of all requested features and corresponding window behavior ?

Yes, sounds a like a good idea to start this. Maybe somewhere on a wiki? I suggest to detail different use-cases with their aims, audience, etc, and then see what features they share and what is different.

Regarding the framework, I agree with @pmoulon, I use Qt (and OpenGL) for OpenMVG-GUI, there is no need to use another framework (and we also want to limit the number of dependencies).

I agree that Qt is very nice and I understand your sentiment to stick with, e.g. for modularity and keeping dependencies minimal. I haven't worked with Qt 3D myself. What is nice about Pangolin is that it has many of the things I need already built in conveniently, e.g. (3D accelerated) 2d image rendering with zoom/pan/rescaling/histogram-normalization, 2d time-series plotting with navigation (zoom/pan/refit/sliders), camera animation, etc...

See here from the Pangolin Readme, stating their intent to reduce boilerplate, yet remain simple, lightweight, and performant:

Pangolin is a lightweight portable rapid development library for managing OpenGL display / interaction and abstracting video input. At its heart is a simple OpenGl viewport manager which can help to modularise 3D visualisation without adding to its complexity, and offers an advanced but intuitive 3D navigation handler. Pangolin also provides a mechanism for manipulating program variables through config files and ui integration, and has a flexible real-time plotter for visualising graphical data.

The ethos of Pangolin is to reduce the boilerplate code that normally gets written to visualise and interact with (typically image and 3D based) systems, without compromising performance. It also enables write-once code for a number of platforms, currently including Windows, Linux, OSX, Android and IOS.

If I work on this (which at least initially would be a dev-tool for myself) I'd have to see if similar things are supported with Qt 3D, but last time I wrote a computer vision dev tool with Qt, just getting a 2D image widget with the features I wanted took quite some time (there was no 3D at that point). Still, with respect to regular GUI elements (buttons, text-edits, tabs, property browser, progress bars, ...) Qt is obviously vastly superior.

[NikolausDemmel]

Maybe a bit more detail on the kinds of visualization / interactions I am looking for:

3D viewer:

• visualize points, camera frustrums, connections between cameras (e.g. covisibility graph, or constraints from odometry)
• free 3d viewport navigation in the typical pan / rotate style (e.g. like rviz in ROS, if you happen to know that).
• nice to have: visualize ground truth poses/trajectories (for sequential data) and possibly ground truth structure
• nice to have: interaction with points and cameras, e.g. displaying details, highlighting related elements (visible points for cameras, or observant cameras for points), triggering actions (e.g. removing points, perturbing current state position / pose, ...)
• nice to have: camera animation for videos

2D viewer:

• sliding through image set / sequence
• showing 2d feature points, residuals (e.g. reprojection errors)
• nice to have: projecting the current 3D scene into the actual image as an overlay

Time series viewer (for sequential data):

• visualize x,y,z position and roll/pitch/yaw orientation of camera vs time in a time series plot
• compare to ground truth loaded from file
• nice to have: visualize other data like velocities, IMU measurements

GUI:

• select algorithms to run (examples: add new image, run BA, perturbe poses/positions, find new points/associations, remove outliers, dense reconstruction)
• run algorithms, see progress
• control viewer options
• edit parameters
• load/save/export

PS: I am not just interested in the classical SfM scenario with a set of images, but also the backend / mapping part for SLAM with a clear sequential / incremental aspect, things like loop closures and pose graph optimization, etc... But one step at a time!

[rperrot]

Very cool. Though to make sure we talk about the same thing, I see myself not really as a user but as a developer in this case.

I was meaning user in a large term ;-) I mean end-users (as it was the goal for OpenMVG-GUI), advanced user (as it was also the goal for OpenMVG-GUI), and developers (not the first audience but some of your need may cover the advanced-user audience).

The main goal of OpenMVG-GUI is to bring a user friendly experience to OpenMVG (that is one of our lack for now). But if some tools develop in context of OpenMVG-GUI could be helpful for another purpose, we may think about some kind of "modularization" to provide separate things.

For me the workflow of exporting to ply after every change does not sound like a good fit, since I want to interactively run individual steps of algorithms (like a single iteration of BA, or adding one image in incremental SfM) and see the effect right away. And I would like to save the current state (e.g. to sfm_data) and continue later.

Yes, I understand, for now the OpenMVG-GUI pipeline is lacking of a simple sfm_data visualizer. Some kind of an analysis toolbox for advanced users. You may need a very low-level access to the API.

I haven't worked with Qt 3D myself. What is nice about Pangolin is that it has many of the things I need already built in conveniently, e.g. (3D accelerated) 2d image rendering with zoom/pan/rescaling/histogram-normalization, 2d time-series plotting with navigation (zoom/pan/refit/sliders), camera animation, etc...

You have access to various things, some are hand coded for OpenMVG-GUI, for ex 3d navigation using OpenGL, for simple 2d image manipulation, you have QGraphicsView system that is highly optimized and finally for plotting you have Charts (https://doc.qt.io/qt-5/qtcharts-overview.html)

The goal of Pangolin and Qt are not the same, and I think, in official OpenMVG tools we may have some difficulties to use two frameworks together. Maybe we have to list the needs then we try to see if it's possible to do it with Qt, or by hand.

Yes, sounds a like a good idea to start this. Maybe somewhere on a wiki? I suggest to detail different use-cases with their aims, audience, etc, and then see what features they share and what is different.

Yes, maybe in a first step we could use this thread as a starting point. We'll see later if we need a more formal way (wiki, gdoc, ...) to express the needs.

[NikolausDemmel]

The main goal of OpenMVG-GUI is to bring a user friendly experience to OpenMVG (that is one of our lack for now). But if some tools develop in context of OpenMVG-GUI could be helpful for another purpose, we may think about some kind of "modularization" to provide separate things.

So yes, the immediate goal is different from mine, but as you say, maybe we can share some synergies e.g. in the 3D visualization code.

You have access to various things, some are hand coded for OpenMVG-GUI, for ex 3d navigation using OpenGL, for simple 2d image manipulation, you have QGraphicsView system that is highly optimized and finally for plotting you have Charts (https://doc.qt.io/qt-5/qtcharts-overview.html)

Thanks for the pointers. I will give Qt 3D, QGraphicsView and Charts a good look.

Yes, maybe in a first step we could use this thread as a starting point. We'll see later if we need a more formal way (wiki, gdoc, ...) to express the needs.

Sounds good. I started above with my a quick outline of my requirements.

Just fyi: I plan to spend serious time on implementation not before beginning of March because of other projects, but I wanted to get into discussion early.

[NikolausDemmel]

So it has been a while. I was hold up with other projects, but I finally have time to look at this again and start implementing. I just wanted to check in to ask if there are any new developments, in particular with @rperrot's work towards a 3D viewer.

[rperrot]

Hi @NikolausDemmel,

there were only small changes since our talk (I'm quite busy).

GUI things are develop in my repo for now : https://github.com/rperrot/openMVG/commits/OpenMVG-GUI

The most significant thing that may interest you is that now the 3d view highlights the links between the views (with varying color depending on the number of links between the cameras).

[NikolausDemmel]

Thanks for the update. I tried the GUI and it works nicely, thanks! (I needed a small fix for compilation, and I also merged recent develop branch, see https://github.com/NikolausDemmel/openMVG/tree/OpenMVG-GUI)

As mentioned above, I need something more interactive, i.e. a viewer that doesn't just load the result after running the algo, but visualizes the current data structure from memory and allows to update the viewer as parts change.

I will see what makes sense for us and will come back either for more questions or also in case I have something worth sharing.

[pmoulon]

@NikolausDemmel It does not change the main idea we expressed earlier:

• We can build a viewer that can be used for interactive viewing of any sfm_data file and use the same viewer as a widget inside the OpenMVG-GUI.

[NikolausDemmel]

Agreed :-).

[rperrot]

@pmoulon was faster than me but I also agree. It could be in the other direction too : some of the widgets used in the GUI can be isolated to your need.