Closed ZijiaLewisLu closed 2 weeks ago
It also seems ego4d_roberta_token_extractor.py
and data/ego4d_naq/video_features/features/egovlp/
used in DATASET.md are missing. Can you provide these files as well? Thanks!
Thank you for your interest.
I have added this script train_ego4d.sh for training ego4d without NaQ.
I did not use Roberta Features. However you can use ego4d_roberta_token_extractor.py
from CONE paper (Link)
The downloaded NaQ dataset features should be placed in this directory data/ego4d_naq/video_features/features/egovlp/
Hi,
Thanks for your reply!
I tested finetuning from NaQ and training from scratch using your scripts. Finetuning gives the reported numbers yet training from scratch yields very low results. I pasted one result below. Could there be any error in the train_ego4d.sh
?
+Window Pre-filtering Epoch 149-------+---------+---------+
| Rank@1 | Rank@5 | Rank@10 | Rank@20 | Rank@30 | Rank@50 |
+--------+--------+---------+---------+---------+---------+
| 13.46 | 54.07 | 80.73 | 97.31 | 98.87 | 100.00 |
+--------+--------+---------+---------+---------+---------+
Evaluated: 0 / 3875 instances
+Fusion Epoch 149-----+----------+----------+----------+----------+----------+----------+----------+----------+------+
| Rank@1 | Rank@1 | Rank@5 | Rank@5 | Rank@10 | Rank@10 | Rank@50 | Rank@50 | Rank@100 | Rank@100 | mIoU |
| mIoU@0.3 | mIoU@0.5 | mIoU@0.3 | mIoU@0.5 | mIoU@0.3 | mIoU@0.5 | mIoU@0.3 | mIoU@0.5 | mIoU@0.3 | mIoU@0.5 | |
+----------+----------+----------+----------+----------+----------+----------+----------+----------+----------+------+
| 8.44 | 4.48 | 23.15 | 12.64 | 23.15 | 12.64 | 23.15 | 12.64 | 23.15 | 12.64 | 5.76 |
+----------+----------+----------+----------+----------+----------+----------+----------+----------+----------+------+
Evaluated: 0 / 3875 instances
+Proposal Epoch 149---+----------+----------+----------+----------+----------+----------+----------+----------+------+
| Rank@1 | Rank@1 | Rank@5 | Rank@5 | Rank@10 | Rank@10 | Rank@50 | Rank@50 | Rank@100 | Rank@100 | mIoU |
| mIoU@0.3 | mIoU@0.5 | mIoU@0.3 | mIoU@0.5 | mIoU@0.3 | mIoU@0.5 | mIoU@0.3 | mIoU@0.5 | mIoU@0.3 | mIoU@0.5 | |
+----------+----------+----------+----------+----------+----------+----------+----------+----------+----------+------+
| 4.51 | 2.38 | 15.47 | 8.08 | 15.47 | 8.08 | 15.47 | 8.08 | 15.47 | 8.08 | 3.09 |
+----------+----------+----------+----------+----------+----------+----------+----------+----------+----------+------+
Evaluated: 0 / 3875 instances
+Matching Epoch 149---+----------+----------+----------+----------+----------+----------+----------+----------+------+
| Rank@1 | Rank@1 | Rank@5 | Rank@5 | Rank@10 | Rank@10 | Rank@50 | Rank@50 | Rank@100 | Rank@100 | mIoU |
| mIoU@0.3 | mIoU@0.5 | mIoU@0.3 | mIoU@0.5 | mIoU@0.3 | mIoU@0.5 | mIoU@0.3 | mIoU@0.5 | mIoU@0.3 | mIoU@0.5 | |
+----------+----------+----------+----------+----------+----------+----------+----------+----------+----------+------+
| 7.76 | 3.85 | 20.01 | 11.22 | 20.01 | 11.22 | 20.01 | 11.22 | 20.01 | 11.22 | 5.76 |
+----------+----------+----------+----------+----------+----------+----------+----------+----------+----------+------+
stop_score 6.4607537546047045
Thank you for pointing out the problem. I have updated the script train_ego4d.sh.
Thanks. The new script works.
Thanks for your great work and releasing the code.
I want to train your model for ego4d NLQ without using Naq pretraining. Could you also release the script for this?
Thank you!