search

Agentic-Learning-AI-Lab / lifelong-memory

Code for LifelongMemory: Leveraging LLMs for Answering Queries in Long-form Egocentric Videos
https://lifelongmemory.github.io/
MIT License
13 stars 0 forks source link

Reproduction of Egoschema results #1

Open Ziyang412 opened 5 months ago

Ziyang412 commented 5 months ago

Hi, thanks for the great work! I am wondering whether you could share the caption.csv file for egoschema dataset for better reproducibility?

Any help would be highly appreciated!

Ziyang412 commented 2 days ago

Hi, hope to get reply from the authors.

I tried to reproduce the code using the caption I extracted using the paper's description, but I got two issue:

  1. The success rate of EgoSchema QA is only around 80%, nearly 20% sample cannot process to the output csv.
  2. The accuracy in the successed example is low: on ES subset, only 64.1 Acc using 90 captions, which is lower than the reported numbers.

My reproduction on egoschema fullset is only 54.3 accuracy, which is 13% lower than your kaggle submission. It would be awesome if the authors could give some help on the reproduction.

Thanks a lot!