kalwalt / webarkit-jsfeat-cpp

c++ jsfeat version
GNU Lesser General Public License v2.1
6 stars 3 forks source link

[WIP] - Feature train orb #31

Open kalwalt opened 2 years ago

kalwalt commented 2 years ago

In this Pull request i will try do develop some utilities to train a Orb pattern. Partially following the jsfeat code but also OpenCV implementation as it was the original one.

kalwalt commented 2 years ago

Speaking about sorting the keypoints, OpenCV use nor qsort or sort but anohter sorting algorithm, probably because more efficient ? KeyPoint OpenCV class and Keypoint_t from jsfeat are almost similar, and so probably we can adapt the code from OpenCV to our needs. If we look the Orb code in ComputeKeypoints method keypoints are removed those close to the border and then filtered:

// Remove keypoints very close to the border
KeyPointsFilter::runByImageBorder(keypoints, img.size(), edgeThreshold);

// Keep more points than necessary as FAST does not give amazing corners
KeyPointsFilter::retainBest(keypoints, scoreType == ORB_Impl::HARRIS_SCORE ? 2 * featuresNum : featuresNum);

https://github.com/opencv/opencv/blob/64aad34cb4abfb6a2603f3f4ecae7f4f0ba1414d/modules/features2d/src/orb.cpp#L853-L857

this is the retainBest method:

// takes keypoints and culls them by the response
void KeyPointsFilter::retainBest(std::vector<KeyPoint>& keypoints, int n_points)
{
    //this is only necessary if the keypoints size is greater than the number of desired points.
    if( n_points >= 0 && keypoints.size() > (size_t)n_points )
    {
        if (n_points==0)
        {
            keypoints.clear();
            return;
        }
        //first use nth element to partition the keypoints into the best and worst.
        std::nth_element(keypoints.begin(), keypoints.begin() + n_points - 1, keypoints.end(), KeypointResponseGreater());
        //this is the boundary response, and in the case of FAST may be ambiguous
        float ambiguous_response = keypoints[n_points - 1].response;
        //use std::partition to grab all of the keypoints with the boundary response.
        std::vector<KeyPoint>::const_iterator new_end =
        std::partition(keypoints.begin() + n_points, keypoints.end(),
                       KeypointResponseGreaterThanOrEqualToThreshold(ambiguous_response));
        //resize the keypoints, given this new end point. nth_element and partition reordered the points inplace
        keypoints.resize(new_end - keypoints.begin());
    }
}

https://github.com/opencv/opencv/blob/64aad34cb4abfb6a2603f3f4ecae7f4f0ba1414d/modules/features2d/src/keypoint.cpp#L68-L90 May we implement this inside webarkit-jsfeat-cpp? we have to try!

kalwalt commented 1 year ago

while testing with docker and emsdk:3.1.25 i got this error and of course fails to build:

/home/kalwalt/kalwalt-github/webarkit-jsfeat-cpp/emscripten/WebARKitLib/lib/SRC/KPM/FreakMatcher/framework/image.h:89:37: error: ISO C++17 does not allow dynamic exception specifications [-Wdynamic-exception-spec]
                   size_t channels) throw(Exception);
                                    ^~~~~~~~~~~~~~~~
/home/kalwalt/kalwalt-github/webarkit-jsfeat-cpp/emscripten/WebARKitLib/lib/SRC/KPM/FreakMatcher/framework/image.h:89:37: note: use 'noexcept(false)' instead
                   size_t channels) throw(Exception);
                                    ^~~~~~~~~~~~~~~~
                                    noexcept(false)
1 error generated.
emcc: error: '/home/kalwalt/emsdk/upstream/bin/clang++ -target wasm32-unknown-emscripten -fignore-exceptions -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -DEMSCRIPTEN -Werror=implicit-function-declaration -I/home/kalwalt/emsdk/upstream/emscripten/cache/sysroot/include/SDL --sysroot=/home/kalwalt/emsdk/upstream/emscripten/cache/sysroot -Xclang -iwithsysroot/include/compat -I/home/kalwalt/kalwalt-github/webarkit-jsfeat-cpp/emscripten/WebARKitLib/include -I/home/kalwalt/kalwalt-github/webarkit-jsfeat-cpp/build/ -I/home/kalwalt/kalwalt-github/webarkit-jsfeat-cpp/emscripten/ -I/home/kalwalt/kalwalt-github/webarkit-jsfeat-cpp/emscripten/WebARKitLib/lib/SRC/KPM/FreakMatcher -Oz -D HAVE_NFT /home/kalwalt/kalwalt-github/webarkit-jsfeat-cpp/emscripten/WebARKitLib/lib/SRC/KPM/FreakMatcher/detectors/DoG_scale_invariant_detector.cpp -c -o /tmp/emscripten_temp_qwriigpt/DoG_scale_invariant_detector_94.o' failed (returned 1)

exec error: 1
Built at Tue Nov 22 19:10:49 CET 2022
Done!

Need to open a issue on webarkitlib in the meanwhile i will continue to use emsdk 3.1.20

kalwalt commented 1 year ago

Somewhere the code is broken, infact in the Yape06 detectfunction the compute_laplacian function print anything (just put a cout to print the dst). It means that or tsome of the args are empty or there is a memory leak. Probably can be heplful to add some specific flags for debugging.

kalwalt commented 1 year ago

I discovered that i missed an important step: the pattern image imported by ar2ReadJpegImage it should be converted to grayscale (this is the missed step) but i can't apply because all the methods are specific for emscripten but not to be used internally. Before all i have to make the imgproc methods more flexible and rearrange the code a bit. I will do this in another specific PR.

kalwalt commented 1 year ago

Ok with the latest commit https://github.com/kalwalt/webarkit-jsfeat-cpp/pull/31/commits/6fe0b12be30cfd0ae214a38981e187de3cc78264 we recieve a count number from detect_keypoints. I will make some comments in the code to use as future reference. The main problem was to adapt javascript code to C++. See the while cicle and you can understand what i mean.

kalwalt commented 1 year ago

Things to do:

kalwalt commented 1 year ago

At this point of the code we got this from te output log:

jsfeatES6cpp_debug.js:1667 JSLOG [info] Level 0 with 3352576 keypoints.
put_char @ jsfeatES6cpp_debug.js:1667
jsfeatES6cpp_debug.js:1667 JSLOG [info] Level 1 with 1676288 keypoints.
put_char @ jsfeatES6cpp_debug.js:1667
jsfeatES6cpp_debug.js:1667 JSLOG [info] Level 2 with 838144 keypoints.
put_char @ jsfeatES6cpp_debug.js:1667
jsfeatES6cpp_debug.js:1667 JSLOG [info] Level 3 with 419072 keypoints.
put_char @ jsfeatES6cpp_debug.js:1667
jsfeatES6cpp_debug.js:1655 Size of first lev_corners: 3352576
jsfeatES6cpp_debug.js:1667 JSLOG [info] After Gaussian blur
put_char @ jsfeatES6cpp_debug.js:1667
jsfeatES6cpp_debug.js:1667 JSLOG [debug] deleting data_t
put_char @ jsfeatES6cpp_debug.js:1667
jsfeatES6cpp_debug.js:1655 Count inside Yape06 detect_internal: 64192
jsfeatES6cpp_debug.js:1667 JSLOG [debug] deleting data_t
put_char @ jsfeatES6cpp_debug.js:1667
jsfeatES6cpp_debug.js:1655 Count inside detect_keypoints: 64192
jsfeatES6cpp_debug.js:1667 JSLOG [info] train 1637 x 2048 points: 300

instead from jsfeatNext:

train 512x384 points: 300
sample_orb.html:195 train 362x271 points: 300
sample_orb.html:195 train 255x191 points: 261
sample_orb.html:195 train 181x135 points: 139

what is wrong with the c++ code?

kalwalt commented 1 year ago

JSLOG [info] train 1637 x 2048 points: 300 display only one line because we only print for the first corners level. i will fix this.

kalwalt commented 1 year ago

with commit https://github.com/kalwalt/webarkit-jsfeat-cpp/pull/31/commits/4a0cf972479da4792f106c1acbe71ca5d145bd64 i added the training for the other levels, but it doesn't correctly calculate. I think there is an issue with the resample method, not sure if it ever worked correctly.