Core Solver - Add Ceres Solver

[david-cattermole]

Feature

Ceres Solver is a state-of-the-art non-linear least squares solver and would likely show performance improvements over the currently supported CMinpack and LevMar solvers.

Ceres solver will be installed as part of #144, and so we should make sure mmSolver can use it as a solver type if available.

This will involve writing a new wrapper around our internal "solverFunc" for Ceres, and providing the maya.cmds.mmSolver Python command with the ability to use Ceres solver.

Software Versions

• mmSolver version: v0.4.x
• Maya version: All supported
• Operating System (OS): All supported

[david-cattermole]

Ceres Solver has been added as a dependency in v0.4.0, however Ceres is not yet supported as a solver back-end.

Progress will continue to add support as a Solver back-end.

[bhack]

What kind of features listed in the bullet points at https://github.com/david-cattermole/mayaMatchMoveSolver/issues/144#issue-598665505

are already covered by OpenMVG?

[david-cattermole]

Hello @bhack,

OpenMVG provides Fundamental Matrix solving, Essential Matrix Solving and Unconstrained Bundle Adjustment (via Ceres Solver). These have been used in the Camera solver in mmSolver, although I'm not very happy with my implementation as I don't find it very robust, and can provide vastly different results each run of the solver.

OpenMVG does not provide 2D pattern tracking, only libmv can provide that. OpenMVG only provides photogrammetry-style matching using SIFT-like image feature detection. 2D pattern tracking is a non-goal of MM Solver (at least in the immediate future), so not having that feature is a non-issue.

David

[bhack]

These have been used in the Camera solver in mmSolver, although I'm not very happy with my implementation as I don't find it very robust, and can provide vastly different results each run of the solver.

Do you have any specific plan to improve on this?

I see that you also import tracks from 3Dequalizer. Have you compared with its own solver?

[david-cattermole]

I do not have a specific plan, however I have a general idea of what needs to be improved.

Generally, the main issue with quality is caused because the bundle adjustment step is only performed every 5 camera poses that are approximated. Changing this value to 1 improves the quality overall but results in a much slower camera solve. I have experimented with changing the camera solver algorithm to find a good compromise.

The bundle adjustment performance can probably be improved by using the Bundle Adjustment function inside OpenMVG (which uses Ceres). This is not yet implemented, and the Camera solver is currently using the mmSceneGraph and the mmSolver core for bundle adjustment.

Yes, importing 2D Marker data is supported from 3DEqualizer and Blender (which is using libmv for 2D tracking and solving) and any other software that can export the .uv file format (which is pretty simple).

I test against 3DEqualizer solved camera output, or any other software, as long as the markers and bundles match with a low error/deviation and is physically plausible, I consider that correct. Generally my goal is to solve an image sequence of say 250 frames with 20 to 40 markers/bundles in under 1 second.

You can see the current camera solver implementation here: https://github.com/david-cattermole/mayaMatchMoveSolver/blob/be8813fcbb991eebe69fc10dca9df44d5729371b/python/mmSolver/_api/solvercamera.py I have an uncommitted camera solver which improves the quality, with a big performance drop. Until I can speed up the camera solver (perhaps by using Ceres solver for bundle adjustment), I won't be releasing my changes.

Thanks, David

[bhack]

Thanks for the details.. do you have any empirical estimation (I don't know if you have small internal benchmark) about the 3d equalizer solver performances against what we currently have in mmsolve?

[david-cattermole]

mmSolver has a test suite to test for correctness, but there isn't any performance benchmark tests (other than me being annoyed that tests take too long to finish). https://github.com/david-cattermole/mayaMatchMoveSolver/blob/master/tests/README.md

Anecdotally, 3DEqualizer's camera solver is the fastest and most robust solver for SfM tasks that I've used.

I am not intending mmSolver to compete directly with 3DEqualizer's camera solver, instead mmSolver provides a different type of solver and focuses on getting results that 3DEqualizer cannot solve; for example the core of mmSolver allows arbitrary transform hierarchies with connections between objects, utilizing the Maya scene graph and the ability to solve arbitrary attributes. The mmSolver camera solver is intended to provide an even lower number of minimum points (only 5 points needed per-frame, as opposed to 3DEqualizer's 6 points per-frame), and is also intended to be used to solve complex camera moves that need to be stitched together, and combined with the mmSolver core solver with arbitrary attributes and transform hierarchies.

[bhack]

Ok thanks for the details. I subscribe to this ticket to check if we have some updates for the OpenMVG/Ceres bundle adjustment.