How to save new zero shot weight?

[joshmyersdean]

Hello!

What weight are you saving for the zero shot weights?

[joshmyersdean]

Ah found the line here! https://github.com/facebookresearch/VLPart/blob/main/demo/predictor.py#L23

[joshmyersdean]

Hi @PeizeSun,

For the zero-shot weights, do you just run each part of the vocabulary with CLIP and then save the resulting matrix?

[joshmyersdean]

Update: When I do the above I get different weights than what are saved.

form = lambda x: f"A photo of a {' '.join(x.split(':'))}"

embeds = torch.zeros(77,1024)
with torch.no_grad():
    for idx,i in enumerate(PASCAL_PART_BASE_CATEGORIES):
        embeds[idx,:] = text_encoder(text_encoder.tokenize(form(i['name'])))

saved_embeds = np.load("datasets/metadata/pascal_part_base_clip_RN50_a+cname.npy")

print(embeds)
tensor([[-0.0448,  0.0072, -0.1996,  ...,  0.0097, -0.2787,  0.3235],
        [-0.0216,  0.2441, -0.2372,  ..., -0.1914, -0.1295,  0.1115],
        [ 0.0405, -0.1673, -0.1807,  ..., -0.2903, -0.2093,  0.0923],
        ...,
        [ 0.0482, -0.1842, -0.0162,  ..., -0.3602,  0.1764,  0.0118],
        [ 0.0588, -0.1146, -0.0537,  ..., -0.1661, -0.0903,  0.0582],
        [ 0.4419,  0.0859,  0.1314,  ..., -0.0912, -0.3575, -0.1130]])

print(saved_embeds)
array([[-0.2603  ,  0.1097  , -0.2805  , ..., -0.03482 , -0.1514  ,
         0.3855  ],
       [-0.1748  ,  0.328   , -0.3494  , ..., -0.3804  , -0.0924  ,
         0.141   ],
       [-0.09174 , -0.04358 , -0.2373  , ..., -0.3523  , -0.1426  ,
         0.1539  ],
       ...,
       [-0.01486 , -0.1587  , -0.11475 , ..., -0.4219  ,  0.1277  ,
         0.1537  ],
       [-0.0338  , -0.004128, -0.2137  , ..., -0.3127  , -0.04684 ,
         0.1978  ],
       [ 0.2861  ,  0.1469  , -0.0263  , ..., -0.2102  , -0.3286  ,
         0.03644 ]], dtype=float16)