abhinanda-punnakkal / BABEL

Code accompanying the BABEL dataset (CVPR 2021).
156 stars 9 forks source link

Null annotations in test split #15

Open Radu1999 opened 10 months ago

Radu1999 commented 10 months ago

Hi, all entries in the test split seem to have null for annotations. Is this intended? If so, why :) Thanks! Example:

 "3357": {
    "babel_sid": 3357,
    "url": "https://babel-renders.s3.eu-central-1.amazonaws.com/003357.mp4",
    "feat_p": "CMU/CMU/64/64_11_poses.npz",
    "dur": 4.97,
    "seq_ann": {
      "babel_lid": "63f83640-22ae-4412-bfbe-5cc0ce467aa4",
      "anntr_id": "b1825e14-d35a-48c1-b389-2cb6bef6a2af",
      "mul_act": false,
      "labels": [
        {
          "raw_label": null,
          "proc_label": null,
          "seg_id": "2970072b-07e4-4b18-bcad-ce165ee0b659",
          "act_cat": null
        }
      ]
    },
    "frame_ann": null
  },
DominikVincent commented 9 months ago

I suppose it is intended (test will be evaluated by the evaluation server) but I wonder how to create the test set for the model anyway. One should run the test set and upload its results to the evaluation server but the code only provides the information on how to create the train and validation split.

Especially since we do not know the labels of the test split it seems impossible to create the 60/120 test split