About EgoVLP feature for Ego4D.

[EdenGabriel]

Hi, thanks for great work!

Excuse me, does the egovlp feature need to convert .pt files to .npy files, and if so, is there any conversion interface provided?

Looking forward to hearing from you. Thanks.

[EdenGabriel]

Besides, i downloaded the omnivore features according to your prompt, and use "convert_ego4d_trainval.py" you provided to generate "ego4d.json" and "omnivore_features". When i use only omnivore features to complete the ego4d task, the program throw exception "AssertionError: GT action labels does not exist".
command line: python ./train.py config ./configs/ego4d_omnivore.yaml --output reproduce Can you help me? Looking forward to hearing from you. Thanks.

[fmu2]

I've updated the conversion script and it now filters out video clips without annotation. The EgoVLP features should be stored in .npy format. Simply convert the features in .pt format to numpy arrays and use the same annotation file for training.

[EdenGabriel]

Thanks for your reply. According to your reply, when i use only omnivore features or egovlp features, the program is worked, but when i use both omnivore features and egovlp features, the program throws exception: 'ValueError: all the input array dimensions for the concatenation axis must match exactly, but along dimension 0, the array at index 0 has size 899 and the array at index 1 has size 897'. In Line158, firstly concatenate all features, so whether need to resize different features to the same dimension firstly?

Besides, in the tech report for your submission to the Ego4D Moment Queries (MQ) Challenge, you said that ''SlowFast, Omnivore and EgoVLP features are independently projected to 386, 386 and 256 dimensions and then concatenated together as the input to ActionFormer.''

Looking forward to hearing from you. Thanks. Have a nice day.

[fmu2]

• Yes, interpolate shorter features (often EgoVLP) to match the size of other features (SlowFast or Omnivore).
• In our latest experiments we found separate projection is not necessary. The config files in the repo yield better results.

[EdenGabriel]

Thanks for your reply. Thus, I need to expand the shorter feature dimension to the other feature dimension, can i interpolate 0 directly or use linear interpolation？

[tzzcl]

I think you need to use linear interpolation, an example can be found in https://github.com/happyharrycn/actionformer_release/blob/main/libs/datasets/anet.py#L193-L198. you can also use scipy.interpolate.interp1d to do this job.

[EdenGabriel]

Thank you very much for your reply. I will try to do it.

Have a nice day.