Support for translation

[DiTo97]

Hi @lukasalexanderweber,

The OpenCV bundle adjuster does not support translation between frames, but only in-place rotation, in the global optimization step.

During my thesis work two years ago I made something similar to your package, wrapping some of OpenCV's stitching module, but not really quite as polished. As I was dealing with frames that at times suffered from quite a severe translation between them, I found OpenCV's results to be somewhat underwhelming. The bundle adjuster works just fine when the translation is small, but gradually starts to fail the larger it gets.

I started looking into other packages that implemented sparse bundle adjustment, as they had support for translation, but did not have the time to implement the Python bindings and ended up using directly in C++.

Have you ever looked into this? Recently, a package with bindings of Lourakis' sparse bundle adjustment C library has been updated after many years for Python 3 compatibility. Would you be interested in trying to add this functionality to your package?

[lukasalexanderweber]

Hey @DiTo97,

thanks for reaching out! :)

Would you be interested in trying to add this functionality to your package?

I am generally open to new features if they are developed, tested and documented by the person they are of interest to.

First of all a sample dataset would be nice that shows the issue and that we can use for testing.

Have you ever looked into this?

no, I'm not familiar with the concept of bundle adjustment, till now I've just used what's there.

[DiTo97]

Hi @lukasalexanderweber,

I will ask if I can share the dataset, as I have not touched it in two years, but I already isolated the problematic frames at the time.

I will let you know next week!

[DiTo97]

Hi again @lukasalexanderweber,

I asked for permission, but it was denied, and I cannot share the dataset.

In retrospect, I will close the issue because reworking OpenCV stitching module to enable a translation model would have to dig much deeper than just the bundle adjustment stage, as has already been pointed out in the answer of this post.

Regardless, the problem arises only when the (absolute or relative) 6-dimensional camera poses are unknown and the stitching pipeline has to perform auto-registration to estimate them from the acquired frames.

I would probably consider adding a cameras arg to the stitch function of your Stitcher wrapper class. In that way, the auto-registration stage could be made optional in the stitching pipeline. What do you think about that?

Appendix

For future reference, if anyone had to auto-register 6-dimensional camera poses with translation, I would use the Pixsfm package, which uses COLMAP and Ceres Solver for solving bundle adjustment with support for deep features and GPU acceleration.

Once the auto-registration is complete, the composition code from this package will then have to be used.

If you want, @lukasalexanderweber, I can provide an example of the integration of the two.