Comparative Evaluation of Hand-Crafted and Learned Local Features

This repository contains the instructions and the code for evaluating feature descriptors on our image-based reconstruction benchmark. The details of our local feature benchmark can be found in our paper:

"Comparative Evaluation of Hand-Crafted and Learned Local Features".
J.L. Schönberger, H. Hardmeier, T. Sattler and M. Pollefeys. CVPR 2017.

Paper, Supplementary, Bibtex

You might also be interested in the HPatches benchmark by Balntas and Lenc et al. presented at CVPR 2017.

Benchmark Results

This table lists the latest benchmark results. Note that the results differ from the original paper, since they were updated with the latest COLMAP version. If you want to submit your own results, please open a new issue or pull request for this repository. Note that the below table extends to the right and alternatively can be viewed in a code or text editor.

Metrics:

Dataset	Method	# Images	# Reg. Images	# Sparse Points	# Observations	Track Length	Obs. Per Image	Reproj. Error [px]	# Dense Points	# Inlier Pairs	# Inlier Matches
Fountain	SIFT	11	11	14722	70631	4.79765	6421.00	0.392893	292609	55	127734
	SIFT-PCA		11	14281	67776	4.74588	6161.45	0.379411	295870	55	117257
	DSP-SIFT		11	14867	71153	4.78596	6468.45	0.414944	293789	55	130820
	ConvOpt		11	14717	70614	4.79812	6419.45	0.393435	296522	55	127540
	TFeat		11	14273	67584	4.73509	6144.00	0.372782	298433	55	113928
	LIFT		11	6003	28296	4.71364	2572.36	0.580594	304258	55	52293

Herzjesu	SIFT	8	8	7502	31670	4.22154	3958.75	0.431632	241347	28	48965
	SIFT-PCA		8	7161	29735	4.15235	3716.87	0.409061	245291	28	44443
	DSP-SIFT		8	7769	32809	4.22306	4101.12	0.459535	238122	28	51893
	ConvOpt		8	4957	20227	4.08049	2528.37	0.387640	242262	26	27830
	TFeat		8	7061	29232	4.13992	3654.00	0.404879	247065	28	43297
	LIFT		8	3742	14890	3.97915	1861.25	0.620034	241173	28	22683

South-Building	SIFT	128	128	108124	653975	6.04838	5109.18	0.545747	2141964	3822	2036024
	SIFT-PCA		128	105612	632145	5.98554	4938.63	0.531500	2090915	3979	1927873
	DSP-SIFT		128	112719	666808	5.91566	5209.43	0.580537	2141873	3958	2076833
	ConvOpt		128	62306	397579	6.38107	3106.08	0.487924	2117221	1901	984762
	TFeat		128	102143	604357	5.91677	4721.53	0.510260	2089004	4342	1751327
	LIFT		128	42601	233110	5.47193	1821.17	0.730874	2154755	2830	711142

Madrid Metropolis	SIFT	1344	500	116088	733745	6.32053	1467.49	0.605330	1822434	227092	6969437
	SIFT-PCA		469	111090	645437	5.81003	1376.19	0.586054	1571584	644573	13970478
	DSP-SIFT		467	99514	649704	6.52877	1391.22	0.660135	1643614	135215	4586807
	ConvOpt		348	40749	213176	5.23144	612.57	0.534638	1251705	665669	12531539
	TFeat		435	102775	574980	5.59455	1321.79	0.566243	1536760	712501	15207011
	LIFT		416	44056	303055	6.87885	728.497	0.768777	1577304	82562	2531640

Gendarmenmarkt	SIFT	1463	1035	338972	1872308	5.52348	1809.00	0.699118	4225031	321854	12625310
	SIFT-PCA		975	349217	1690464	4.84072	1733.80	0.701904	3649260	822997	20321433
	DSP-SIFT		979	293209	1577921	5.38155	1611.76	0.749714	2600189	265575	9315075
	ConvOpt		772	178859	694211	3.88133	899.23	0.723822	2955105	811724	15583270
	TFeat		902	280233	1324931	4.72796	1468.88	0.695517	3384513	655181	15040928
	LIFT		959	142982	819940	5.73456	854.99	0.841945	3939957	125084	5012767

Tower of London	SIFT	1576	804	239951	1863301	7.76534	2317.53	0.615406	3050252	165097	11249925
	SIFT-PCA		693	220381	1491686	6.76866	2152.50	0.602057	2518677	558173	14605601
	DSP-SIFT		799	267906	1940752	7.24415	2428.97	0.655440	2946702	260963	12750104
	ConvOpt		537	143397	788855	5.50119	1469.00	0.580207	2448215	742322	14648025
	TFeat		675	255666	1605322	6.27898	2378.25	0.580068	2583560	926517	21742783
	LIFT		713	96848	739340	7.63402	1036.94	0.728200	2879455	60841	3628677

Alamo	SIFT	2915	963	198433	2437084	12.28164	2530.72	0.647271	3737516	64068	21263831
	SIFT-PCA		921	197723	2279339	11.52791	2474.85	0.626812	3256364	143747	20145150
	DSP-SIFT		961	223192	2564659	11.49082	2668.73	0.712005	3815012	79973	23375984
	ConvOpt		684	110261	1167754	10.59081	1707.24	0.537849	2546861	168383	8065721
	TFeat		865	180730	2040775	11.29184	2359.27	0.609598	2973035	192115	16518550
	LIFT		796	78892	1011117	12.816471	1270.24	0.768177	2900266	40219	8151208

Roman Forum	SIFT	2364	1679	433152	3603662	8.31962	2146.31	0.708420	9630170	76547	16424472
	SIFT-PCA		1663	434317	3267075	7.52232	1964.56	0.674920	9379870	151694	15134227
	DSP-SIFT		1644	464792	3653745	7.86103	2222.47	0.749306	9429283	100827	16469792
	ConvOpt		1282	182922	1263324	6.90635	985.43	0.627904	7404163	158940	6151296
	TFeat		1603	401965	2897537	7.20843	1807.57	0.647753	9096825	180301	12869235
	LIFT		1503	174430	1420800	8.14538	945.30	0.814467	8584480	49413	5775222

Cornell	SIFT	6514	6073	1847141	12865681	6.96518	2118.50	0.660522	35232209	227478	61428156
	SIFT-PCA		6010	1856258	12307131	6.63007	2047.77	0.643796	35263104	417668	59874790
	DSP-SIFT		6069	2071407	13671952	6.60032	2252.75	0.708143	35449395	283503	64364585
	ConvOpt		5009	938316	6082683	6.48255	1214.35	0.570824	30619302	353461	25017605
	TFeat		5779	1730263	11292717	6.52659	1954.09	0.622775	33917778	489447	55385797
	LIFT		5518	739059	4602081	6.22694	834.01	0.730208	33372173	143408	19144270

Runtime:

Method	Runtime	Hardware
SIFT	9.3s	(Intel E5-2697 2.60GHz CPU - single-threaded)
SIFT-PCA	10.5s	(Intel E5-2697 2.60GHz CPU - single-threaded)
DSP-SIFT	23.7s	(Intel E5-2697 2.60GHz CPU - single-threaded)
ConvOpt	49.9s	(Intel E5-2697 2.60GHz CPU, NVIDIA Titan X GPU)
TFeat	11.8s	(Intel E5-2697 2.60GHz CPU, NVIDIA Titan X GPU)
LIFT	212.3s	(Intel E5-2697 2.60GHz CPU, NVIDIA Titan X GPU)

References:

SIFT: D.G. Lowe: Object Recognition from Local Scale-Invariant Features. ICCV, 1999. R. Arandjelovic and A. Zisserman. Three things everyone should know to improve object retrieval. CVPR, 2012.
SIFT-PCA: A. Bursuc, G. Tolias, and H. Jegou. Kernel local descriptors with implicit rotation matching. ACM Multimedia, 2015.
DSP-SIFT: J.Dong and S.Soatto. Domain-size pooling in local descriptors: DSP-SIFT. CVPR, 2015.
ConvOpt: K. Simonyan, A. Vedaldi, and A. Zisserman. Learning local feature descriptors using convex optimisation. PAMI, 2014.
TFeat: V.Balntas, E.Riba, D.Ponsa, and K.Mikolajczyk. Learning local feature descriptors with triplets and shallow convolutional neural networks. BMVC, 2016.
LIFT: M. Kwang, E. Trulls, V. Lepetit, and P. Fua. LIFT: Learned Invariant Feature Transform. ECCV, 2016.

ahojnnes / local-feature-evaluation

readme

Comparative Evaluation of Hand-Crafted and Learned Local Features

Benchmark Results