MAE Spatiotemporal Learners (Garbage Results)

[innat]

Background

I was following the interesting work done by facebookresearch on Masked Autoencoders As Spatiotemporal Learners which later added to SlowFast/mae_st. With the provided pretrained weight, I tried to call inference on series of video frame (a smple from kinetric400), however, the results with the pretrained model gives totally nonsene prediction.

Reproducible Code

Please find the gist here. COLAB

Info

• I have ported the model definition from mae_st repo as it was compratively cleaner and to the point.
• The pre-trained weight links of both mae_st and slowfast_mae_st repos are tried, but gives nonsene prediction.
• The above colab is ready-to-go; just run all cell.

[innat]

cc. @haooooooqi @feichtenhofer

[innat]

Hi @r-barnes I've just noticed some update on mae_st repo. By any chance, could you please inform if the provided weights are in good health?

[alpargun]

Looking at the collab, you have a long list of missing keys, i.e., not restored layers, when you load the checkpoint and state dictionary:

So, you are running inference with random weights.

The vision models in SlowFast, timm, MAE_ST, torchhub, etc., do not necessarily (very rarely) have perfectly matching implementations. Mostly the layer names, i.e., state dictionary keys, will differ in name. So, I would suggest you to run inference using directly the SlowFast framework, including the corresponding config file, and class implementations.

If you have troubles restoring the model, I can provide more information or code snippets.

[innat]

Hello @alpargun , I stopped working on it long ago and forgot to close this ticket. I emailed to one of the authors but no response so far. However, if you are able to fix the colab issue, and fix cost is minimal, please go ahead. I highly appreciate your help. Thank you.