search

ekazakos / temporal-binding-network

Implementation of "EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition, ICCV, 2019" in PyTorch
Other
107 stars 24 forks source link

return process_data, record.label, record.metadata #21

Open fengfan028 opened 3 years ago

fengfan028 commented 3 years ago

What is the record.metadata here? in dataset.py line 184 return process_data, record.label, record.metadata

fengfan028 commented 3 years ago

i use ucf101, not epic-kitchens

ekazakos commented 3 years ago

Hi, record.metadata is the narration_id for EPIC-KITCHENS-100 data, and is used for model evaluation / submission in the action recognition challenge. It is defined here: https://github.com/ekazakos/temporal-binding-network/blob/9e4e108f85124e5bbbae7292d05d59a15dc763e3/video_records/epickitchens100_record.py#L40-L42

For UCF101, first you need to create a ucf101_record.py in video_records/, see video_records/epickitchens100_record.py as an example. Also have a look in the readme under Train/evaluate with other datasets for more info.

Then, if there are any useful metadata in UCF101, that you would like to store along with the predictions of the model for evaluation purposes, you can also store them in the UCF101 video record as shown above.

fengfan028 commented 3 years ago

Hi, record.metadata is the narration_id for EPIC-KITCHENS-100 data, and is used for model evaluation / submission in the action recognition challenge. It is defined here: https://github.com/ekazakos/temporal-binding-network/blob/9e4e108f85124e5bbbae7292d05d59a15dc763e3/video_records/epickitchens100_record.py#L40-L42

For UCF101, first you need to create a ucf101_record.py in video_records/, see video_records/epickitchens100_record.py as an example. Also have a look in the readme under Train/evaluate with other datasets for more info.

Then, if there are any useful metadata in UCF101, that you would like to store along with the predictions of the model for evaluation purposes, you can also store them in the UCF101 video record as shown above.

Thanks very much for your reply! I have a another question: I use pytorch version 1.8.1, when running train.py, it has the following error: image it seems def forward(self, input_tensor): and def backward(self, grad_output): in class SegmentConsensus(torch.autograd.Function): should use @staticmethod, but how to modify them?

ekazakos commented 3 years ago

Ah thanks for spotting that! There are 2 valid options:

  1. You can just use static methods without any further modifications as shown here: https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function
  2. You can just delete backward, PyTorch is doing the backprop automatically so it is not needed.

I will update the code soon to be compatible with PyTorch 1.8.1 by implementing option 2.

fengfan028 commented 3 years ago

Ah thanks for spotting that! There are 2 valid options:

  1. You can just use static methods without any further modifications as shown here: https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function
  2. You can just delete backward, PyTorch is doing the backprop automatically so it is not needed.

I will update the code soon to be compatible with PyTorch 1.8.1 by implementing option 2.

thans a lot! bug has been fixed.