search

facebookresearch / HierVL

[CVPR 2023] HierVL Learning Hierarchical Video-Language Embeddings
Other
43 stars 3 forks source link

ShuffleMCQ benchmark #11

Closed chaitanya100100 closed 3 months ago

chaitanya100100 commented 4 months ago

Hello, Where can I find ShuffleMCQ benchmark data? It would be great if you can provide it to compare with HierVL. Thanks.

thechargedneutron commented 4 months ago

Hi @chaitanya100100 , thanks for the interest in our work. Since this is a pretraining evaluation, we did not create a separate benchmark for ShuffleMCQ. We simply randomize the clips for every input for SummaryMCQ. Here is the relevant line of code: https://github.com/facebookresearch/HierVL/blob/998a8527ed6a3306e031ee73ed81978db6e99861/trainer/trainer_egoaggregate.py#L429

The incorrect options are just a random permutation of clips in the correct frame. If you really want to get the exact same seed, you can probably run this code and save the outputs and then use it in your evaluation OR simply use the same logic in your evaluation and report.

chaitanya100100 commented 4 months ago

Thanks for the quick response! I will probably use the same logic in my code and make sure that the evaluation is robust by trying out multiple random seeds.

thechargedneutron commented 4 months ago

Yeah that is perfect, thanks!

Let me know if you have other questions.