mbanani / probe3d

[CVPR 2024] Probing the 3D Awareness of Visual Foundation Models
MIT License
237 stars 11 forks source link

Add RADIO backbone #4

Closed gheinrich closed 4 months ago

gheinrich commented 4 months ago

This PR adds a backbone definition and associated config file for the RADIO foundation model from:

[CVPR2024] AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into One https://arxiv.org/abs/2312.06709

We measured metrics on the NAVI dataset, and re-ran the DINOv2 baselines for a fair comparison. Below are the corresponding logs files:

Depth estimation:

17042024-0541, dinov2_vitb14, 14, 2-5-8-11  , dense-cls , bindepth_dpt_k3,  10, 1.50, 0.0005, 0.0, 8, NAVI_multiview_all_reldepth, NAVI_wild_all_reldepth, 0.4187, 0.6963, 0.8490, 0.1280, 0.5987, 0.8322, 0.9159, 0.0948 
17042024-0612, dinov2_vitl14, 14, 5-11-17-23, dense-cls , bindepth_dpt_k3,  10, 1.50, 0.0005, 0.0, 8, NAVI_multiview_all_reldepth, NAVI_wild_all_reldepth, 0.4537, 0.7297, 0.8654, 0.1213, 0.6302, 0.8499, 0.9242, 0.0887 
17042024-0717, radio_v2     , 16, 7-15-23-31, dense     , bindepth_dpt_k3,  10, 1.50, 0.0005, 0.0, 8, NAVI_multiview_all_reldepth, NAVI_wild_all_reldepth, 0.4894, 0.7631, 0.8894, 0.1117, 0.6391, 0.8580, 0.9309, 0.0859 

Surface normals:

17042024-1253, radio_v2       , 16, 7-15-23-31, dense     , snorm_dpt_k3_UA,  10, 1.50, 5.00e-04, 0.00e+00,    8, NAVI_multiview_all, NAVI_wild_all, 0.3457, 0.6482, 0.7627, 28.6711 
17042024-1314, dinov2_vitl14  , 14, 5-11-17-23, dense-cls , snorm_dpt_k3_UA,  10, 1.50, 5.00e-04, 0.00e+00,    8, NAVI_multiview_all, NAVI_wild_all, 0.3586, 0.6529, 0.7644, 28.6790 

Multi-view correspondance:

18042024-0455, dinov2_vitb14, 14, 11   , dense     , 1000, 0.25, NAVI_wild_all, 35.90, 54.25, 82.17,  5.24, 32.32, 49.00, 90.00, 68.82, 51.69, 29.22 
18042024-0503, dinov2_vitl14, 14, 23   , dense     , 1000, 0.25, NAVI_wild_all, 39.28, 57.21, 83.50,  6.05, 35.63, 52.28, 91.60, 72.59, 54.65, 32.15 
18042024-0507, radio_v2     , 16, 31   , dense     , 1000, 0.25, NAVI_wild_all, 40.06, 60.08, 86.12,  4.83, 35.18, 54.65, 91.97, 75.39, 58.26, 35.54 
mbanani commented 4 months ago

That's awesome! Thanks @gheinrich for contributing to the code base, RADIO looks pretty interesting.