Open dgcnz opened 1 month ago
Officially, dinov2 is used as segmentation backbone with the help of the get_intermediate_layers
function which default norm=True
, cls=False
.
dinov2_vitb14.get_intermediate_layers(x, n=1, norm=True, reshape=True)
Which returns a single-element tuple with a tensor entry of shape [1, 768, 37, 37]
Both in the case of ViT and EVA-02 for detrex/detectron2, the cls token and last layernorm are not accounted for, so the first thing to try is to also not forward it to the layernorm.
dinov2 = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitb14').eval()
dinov2_timm = timm.create_model('vit_base_patch14_dinov2.lvd142m', pretrained=True, features_only=True, out_indices=(-1,)).eval()
z1 = dinov2.get_intermediate_layers(x, norm=True, reshape=True, return_class_token=False)[0]
z2 = dinov2.forward_features(x)["x_prenorm"]["x_prenorm"][:, 1:, :].unflatten(1, (37, 37)).permute(0, 3, 1, 2)
z3 = dinov2_timm.forward(x)[0]
Also equivalent to detrex.TimmBackbone:
backbone = TimmBackbone(
model_name="vit_base_patch14_dinov2.lvd142m", # name in timm
features_only=True,
pretrained=True,
in_channels=3,
out_indices=(-1, ),
)
backbone(x)["p-1"]
More silly issues:
We use a patch size of 16 for all ViT backbones. As ViT-H in [14] by default has a patch size of 14, after pre-training we interpolate the patch embedding filters from 14×14×3 to 16×16×3.
Found a couple of options:
Making a small comparison, there doesn't seem to be much of a difference between both approaches:
The load_pretrained
method with interpolation can also be implicitly called like this:
dinov2 = timm.create_model('vit_base_patch14_dinov2.lvd142m', pretrained=True, patch_size=16, features_only=True, out_indices=(-1, )).eval()
Relevant DinoV2 snippet
Notes:
x_prenorm
is of shape[1, 1370, 768]
or[B, P*P + 1, D]
Relevant detectron2 snippet