DSPSIFT detects less features that SIFT

[originlake]

Hi,

The implementation of dspsift use first octave 0 and I believe OpenCV's first octave is -1. This makes the difference that the medium quality of SIFT is kinda on par with high quality in dspsift (due to the upscaling of -1 first octave). I can observe the difference during processing that the SIFT usually detects more features than dspsift, and when I set the first octave of dspsift to -1, I can then get similar amount of features. Would it be better to make them comparable at the same quality settings?

(Opencv's default firstOctave) https://github.com/opencv/opencv/blob/c6ace77e21d9db3fcee5953ef36d5ba7ec289995/modules/features2d/src/sift.dispatch.cpp#L508

(DSPSIFT implementation) https://github.com/OpenDroneMap/OpenSfM/blob/792b38714e42588b6f0e9354527660c7e54f18b9/opensfm/src/features/src/dspsift.cc#L76

[pierotofy]

Interesting; would certainly be worth investigating.

[originlake]

Anything I can help on this, I can provide some comparisons for the number of features extracted with different settings. But how that affect the final reconstruction is hard to measure, when a dataset is well surveyed, 4000 or 8000 features might not have big difference with respect to accuracy.

[pierotofy]

Comparison numbers would be interesting. Attaching the PDF reports would also provide some metrics on the feature quality between comparisons.

[originlake]

The tests are performed on Brighton beach with default settings and set min-num-features to 50000 so it can exhaustively find features at peak threshold 0.0001. I'll use dspsift(0) to indicate first octave 0 and dspsift(-1) for first octave -1.

feature-quality	feature-type	features at threshold 0.0001(avg)	report
high	sift	35735	report
high	dspsift(0)	8381	report
high	dspsift(-1)	29740	report
medium	sift	9566	report
medium	dspsift(-1)	8094	report
medium	dspsift(0)	2165	report

The table can show that the dspsift(0) has basically one quality level less features comparing to sift and dspsift(-1). But dspsift(-1) will definitely cost more processing time.

Some logs for processing time below SIFT(feature-quality:high, min-num-features:50000)

2024-07-16 18:14:26,012 DEBUG: Found 31197 points in 14.80418872833252s
2024-07-16 18:14:26,340 DEBUG: Found 34377 points in 15.046931505203247s
2024-07-16 18:14:26,340 DEBUG: Found 35512 points in 14.795958757400513s
2024-07-16 18:14:26,432 DEBUG: Found 40664 points in 15.10500955581665s
2024-07-16 18:14:26,918 DEBUG: Found 33920 points in 15.632619857788086s
2024-07-16 18:14:26,989 DEBUG: Found 40333 points in 15.692596673965454s
2024-07-16 18:14:27,160 DEBUG: Found 35750 points in 15.955885887145996s
2024-07-16 18:14:27,199 DEBUG: Found 33632 points in 16.08137845993042s
2024-07-16 18:14:27,345 DEBUG: Found 38991 points in 16.219543933868408s
2024-07-16 18:14:27,534 DEBUG: Found 35360 points in 16.11851716041565s
2024-07-16 18:14:27,599 DEBUG: Found 36015 points in 16.198935508728027s
2024-07-16 18:14:27,770 DEBUG: Found 40620 points in 16.575142860412598s
2024-07-16 18:14:27,857 DEBUG: Found 35062 points in 16.45927929878235s
2024-07-16 18:14:27,874 DEBUG: Found 28695 points in 16.662339210510254s
2024-07-16 18:14:27,949 DEBUG: Found 36087 points in 16.840041399002075s
2024-07-16 18:14:28,182 DEBUG: Found 35721 points in 16.65530753135681s
2024-07-16 18:14:28,190 DEBUG: Found 39788 points in 16.68756914138794s
2024-07-16 18:14:28,251 DEBUG: Found 40216 points in 17.128756284713745s

DSPSIFT(first-octave:-1, feature-quality:high, min-num-features:50000)

2024-07-16 19:24:50,881 DEBUG: Found 23222 points in 22.074816703796387s
2024-07-16 19:24:53,382 DEBUG: Found 27789 points in 24.67154335975647s
2024-07-16 19:24:56,609 DEBUG: Found 28961 points in 27.637070417404175s
2024-07-16 19:24:57,748 DEBUG: Found 29774 points in 28.672625064849854s
2024-07-16 19:24:57,853 DEBUG: Found 32422 points in 28.75583267211914s
2024-07-16 19:24:59,373 DEBUG: Found 29267 points in 30.383256673812866s
2024-07-16 19:25:00,903 DEBUG: Found 30021 points in 31.931330680847168s
2024-07-16 19:25:02,400 DEBUG: Found 32841 points in 33.51357078552246s
2024-07-16 19:25:02,781 DEBUG: Found 25545 points in 33.956865072250366s
2024-07-16 19:25:02,814 DEBUG: Found 31992 points in 34.09671449661255s
2024-07-16 19:25:02,999 DEBUG: Found 30344 points in 34.29115104675293s
2024-07-16 19:25:03,043 DEBUG: Found 32864 points in 34.316842794418335s
2024-07-16 19:25:03,416 DEBUG: Found 28907 points in 34.307552337646484s
2024-07-16 19:25:05,180 DEBUG: Found 28056 points in 36.20088768005371s
2024-07-16 19:25:05,227 DEBUG: Found 28514 points in 36.34261107444763s
2024-07-16 19:25:07,103 DEBUG: Found 32694 points in 38.2079119682312s
2024-07-16 19:25:07,854 DEBUG: Found 29706 points in 39.04868268966675s
2024-07-16 19:25:11,988 DEBUG: Found 33049 points in 43.197829484939575s

DSPSIFT(first-octave:0, feature-quality:high, min-num-features:50000) report

2024-07-16 19:32:26,096 DEBUG: Found 6539 points in 3.704296112060547s
2024-07-16 19:32:26,629 DEBUG: Found 8000 points in 4.111623287200928s
2024-07-16 19:32:26,803 DEBUG: Found 8080 points in 4.047041654586792s
2024-07-16 19:32:26,999 DEBUG: Found 7257 points in 4.630603790283203s
2024-07-16 19:32:27,075 DEBUG: Found 9111 points in 4.787600517272949s
2024-07-16 19:32:27,371 DEBUG: Found 8540 points in 4.743211984634399s
2024-07-16 19:32:27,773 DEBUG: Found 8245 points in 5.179584264755249s
2024-07-16 19:32:27,785 DEBUG: Found 7997 points in 5.249058961868286s
2024-07-16 19:32:27,817 DEBUG: Found 7682 points in 5.535998344421387s
2024-07-16 19:32:27,839 DEBUG: Found 9202 points in 5.387580871582031s
2024-07-16 19:32:27,879 DEBUG: Found 8653 points in 5.6039416790008545s
2024-07-16 19:32:27,962 DEBUG: Found 7721 points in 5.522029876708984s
2024-07-16 19:32:28,102 DEBUG: Found 8946 points in 5.487308979034424s
2024-07-16 19:32:28,284 DEBUG: Found 8412 points in 5.917739152908325s
2024-07-16 19:32:28,580 DEBUG: Found 9270 points in 6.293765068054199s
2024-07-16 19:32:28,632 DEBUG: Found 8843 points in 6.1773505210876465s
2024-07-16 19:32:28,907 DEBUG: Found 8351 points in 6.197965145111084s
2024-07-16 19:32:28,951 DEBUG: Found 8979 points in 6.59200382232666s

SIFT(feature-quality:high, min-num-features:10000) report

2024-07-16 19:51:28,311 DEBUG: Found 12939 points in 0.9924793243408203s
2024-07-16 19:51:28,752 DEBUG: Found 15000 points in 1.4248030185699463s
2024-07-16 19:51:28,816 DEBUG: Found 15001 points in 1.482863426208496s
2024-07-16 19:51:28,989 DEBUG: Found 10591 points in 1.571342945098877s
2024-07-16 19:51:28,989 DEBUG: Found 10671 points in 1.393477439880371s
2024-07-16 19:51:28,990 DEBUG: Found 12021 points in 1.6655843257904053s
2024-07-16 19:51:28,990 DEBUG: Found 11594 points in 1.3948383331298828s
2024-07-16 19:51:29,006 DEBUG: Found 14335 points in 1.5941359996795654s
2024-07-16 19:51:29,021 DEBUG: Found 15001 points in 1.2983579635620117s
2024-07-16 19:51:29,021 DEBUG: Found 11673 points in 1.3964016437530518s
2024-07-16 19:51:29,037 DEBUG: Found 15000 points in 1.5360908508300781s
2024-07-16 19:51:29,038 DEBUG: Found 13959 points in 1.5021522045135498s
2024-07-16 19:51:29,053 DEBUG: Found 15000 points in 1.3494606018066406s
2024-07-16 19:51:29,084 DEBUG: Found 15000 points in 1.58085298538208s
2024-07-16 19:51:29,187 DEBUG: Found 15001 points in 1.7817540168762207s
2024-07-16 19:51:29,333 DEBUG: Found 15000 points in 1.573132038116455s
2024-07-16 19:51:29,373 DEBUG: Found 15000 points in 1.8778228759765625s
2024-07-16 19:51:29,723 DEBUG: Found 13555 points in 2.304274320602417s

DSPSIFT(first-octave:-1, feature-quality:high, min-num-features:10000) report

2024-07-16 19:46:07,604 DEBUG: Found 10000 points in 5.424947023391724s
2024-07-16 19:46:08,167 DEBUG: Found 10000 points in 5.828838109970093s
2024-07-16 19:46:08,819 DEBUG: Found 10000 points in 6.291464805603027s
2024-07-16 19:46:08,923 DEBUG: Found 10000 points in 6.5672361850738525s
2024-07-16 19:46:09,229 DEBUG: Found 10000 points in 7.052605628967285s
2024-07-16 19:46:09,311 DEBUG: Found 10000 points in 6.8645920753479s
2024-07-16 19:46:09,542 DEBUG: Found 10000 points in 7.2778708934783936s
2024-07-16 19:46:09,807 DEBUG: Found 10000 points in 7.432805299758911s
2024-07-16 19:46:09,918 DEBUG: Found 10000 points in 7.655879020690918s
2024-07-16 19:46:10,044 DEBUG: Found 10000 points in 7.598248481750488s
2024-07-16 19:46:10,178 DEBUG: Found 10000 points in 7.612064361572266s
2024-07-16 19:46:10,652 DEBUG: Found 10000 points in 8.47565484046936s
2024-07-16 19:46:10,804 DEBUG: Found 10000 points in 8.627728462219238s
2024-07-16 19:46:10,929 DEBUG: Found 10000 points in 8.478309869766235s
2024-07-16 19:46:11,194 DEBUG: Found 10000 points in 8.939449310302734s
2024-07-16 19:46:11,338 DEBUG: Found 10000 points in 8.781983852386475s
2024-07-16 19:46:11,822 DEBUG: Found 10000 points in 9.558658599853516s
2024-07-16 19:46:12,122 DEBUG: Found 10000 points in 9.763870239257812s

DSPSIFT(first-octave:0, feature-quality:high, min-num-features:10000) Same as 50000 as it couldn't find more than 10000 features

[pierotofy]

Nice, thanks for the detailed data reports!

Almost all metrics in the report for dspsift(-1) indicate an improvement in matching. An increase in runtime is expected, but seems reasonable especially when compared to sift.

I think this can be patched in our OpenSfM fork and brought into ODM. :+1: If you can open a PR there with the proposed change targeting the 352 branch in OpenSfM I can make the appropriate changes in ODM to bring the changes in.

[originlake]

OK, I'll create a PR for this. I also noticed that the max number of features is 1.5 times of min-num-features in sift, do we need to have the same behavior in dspsift?

[pierotofy]

I don't think that's required.

[pierotofy]

1780 closes this. Thanks for the research and fix @originlake !