script to training on Ego4d NLQ without Naq

[ZijiaLewisLu]

Thanks for your great work and releasing the code.

I want to train your model for ego4d NLQ without using Naq pretraining. Could you also release the script for this?

Thank you!

[ZijiaLewisLu]

It also seems ego4d_roberta_token_extractor.py and data/ego4d_naq/video_features/features/egovlp/ used in DATASET.md are missing. Can you provide these files as well? Thanks!

[Tanveer81]

Thank you for your interest.

I have added this script train_ego4d.sh for training ego4d without NaQ.

[Tanveer81]

I did not use Roberta Features. However you can use ego4d_roberta_token_extractor.py from CONE paper (Link)

The downloaded NaQ dataset features should be placed in this directory data/ego4d_naq/video_features/features/egovlp/

[ZijiaLewisLu]

Hi,

Thanks for your reply!

I tested finetuning from NaQ and training from scratch using your scripts. Finetuning gives the reported numbers yet training from scratch yields very low results. I pasted one result below. Could there be any error in the train_ego4d.sh?

+Window Pre-filtering Epoch 149-------+---------+---------+
| Rank@1 | Rank@5 | Rank@10 | Rank@20 | Rank@30 | Rank@50 |
+--------+--------+---------+---------+---------+---------+
| 13.46  | 54.07  |  80.73  |  97.31  |  98.87  |  100.00 |
+--------+--------+---------+---------+---------+---------+
Evaluated: 0 / 3875 instances
+Fusion Epoch 149-----+----------+----------+----------+----------+----------+----------+----------+----------+------+
|  Rank@1  |  Rank@1  |  Rank@5  |  Rank@5  | Rank@10  | Rank@10  | Rank@50  | Rank@50  | Rank@100 | Rank@100 | mIoU |
| mIoU@0.3 | mIoU@0.5 | mIoU@0.3 | mIoU@0.5 | mIoU@0.3 | mIoU@0.5 | mIoU@0.3 | mIoU@0.5 | mIoU@0.3 | mIoU@0.5 |      |
+----------+----------+----------+----------+----------+----------+----------+----------+----------+----------+------+
|   8.44   |   4.48   |  23.15   |  12.64   |  23.15   |  12.64   |  23.15   |  12.64   |  23.15   |  12.64   | 5.76 |
+----------+----------+----------+----------+----------+----------+----------+----------+----------+----------+------+
Evaluated: 0 / 3875 instances
+Proposal Epoch 149---+----------+----------+----------+----------+----------+----------+----------+----------+------+
|  Rank@1  |  Rank@1  |  Rank@5  |  Rank@5  | Rank@10  | Rank@10  | Rank@50  | Rank@50  | Rank@100 | Rank@100 | mIoU |
| mIoU@0.3 | mIoU@0.5 | mIoU@0.3 | mIoU@0.5 | mIoU@0.3 | mIoU@0.5 | mIoU@0.3 | mIoU@0.5 | mIoU@0.3 | mIoU@0.5 |      |
+----------+----------+----------+----------+----------+----------+----------+----------+----------+----------+------+
|   4.51   |   2.38   |  15.47   |   8.08   |  15.47   |   8.08   |  15.47   |   8.08   |  15.47   |   8.08   | 3.09 |
+----------+----------+----------+----------+----------+----------+----------+----------+----------+----------+------+
Evaluated: 0 / 3875 instances
+Matching Epoch 149---+----------+----------+----------+----------+----------+----------+----------+----------+------+
|  Rank@1  |  Rank@1  |  Rank@5  |  Rank@5  | Rank@10  | Rank@10  | Rank@50  | Rank@50  | Rank@100 | Rank@100 | mIoU |
| mIoU@0.3 | mIoU@0.5 | mIoU@0.3 | mIoU@0.5 | mIoU@0.3 | mIoU@0.5 | mIoU@0.3 | mIoU@0.5 | mIoU@0.3 | mIoU@0.5 |      |
+----------+----------+----------+----------+----------+----------+----------+----------+----------+----------+------+
|   7.76   |   3.85   |  20.01   |  11.22   |  20.01   |  11.22   |  20.01   |  11.22   |  20.01   |  11.22   | 5.76 |
+----------+----------+----------+----------+----------+----------+----------+----------+----------+----------+------+
stop_score  6.4607537546047045

[Tanveer81]

Thank you for pointing out the problem. I have updated the script train_ego4d.sh.

[ZijiaLewisLu]

Thanks. The new script works.