Missing: Rotation estimation, Bundle adjustment, Levenberg Marquardt

[JoanCharmant]

Summary of your issue

The classes and functions related to Rotation Estimation and Bundle adjustment seems to be missing. These are part of the core Stitching module but can normally be used directly.

The high level stitching pipeline makes a number of assumptions that are not great for other applications and it's sometimes necessary to write custom logic for some steps and then call the lower level helpers directly.

It looks like the classes of this page are missing from the wrapper: https://docs.opencv.org/4.9.0/d7/d74/group__stitching__rotation.html

The opencv implementation is in motion_estimators.cpp https://github.com/opencv/opencv/blob/4.x/modules/stitching/src/motion_estimators.cpp

The core loop of bundle adjustment makes use of a Levenberg Marquardt solver (CvLevMarq), which also appears to be missing, so it's also not easy to port the bundle adjustment code to C# manually.

Environment

It looks like some of these classes were present in the old opencvsharp_2410, but aren't present in the current version.

Example code:

The idea is essentially to be able to write a custom version of Stitcher::estimateCameraParams() Something around these lines:

BundleAdjusterReproj bundleAdjuster = new BundleAdjusterReproj();

ImageFeatures[] features; 
MatchesInfo[] pairwiseMatches; 
CameraParams[] cameras;
// Fill the above with custom logic. 
// `cameras` is in/out and contains initial guesses that are being refined by bundle adjustment.

bundleAdjuster.estimate(features, pairwiseMatches, cameras);

Output:

paste your output

[JoanCharmant]

I'm looking into contributing this but the type conversion is a steep learning curve.

While I'm studying how we go from IEnumerable<ImageFeatures> to std::vector<cv::detail::ImageFeatures>&, via WImageFeatures[] and detail_ImageFeatures*, I noticed something odd in the second Apply overload of FeaturesMatcher. It seems we never really convert the KeyPoint variable from the ImageFeatures items.

Is it possible that on line 109 the call to construct VectorOfKeyPoint misses the actual data. Should look like this instead:

keypointVecs[i] = new VectorOfKeyPoint(featuresArray[i].Keypoints);

Haven't tested this, just trying to make sense of it all.