search

facebookresearch / EgoTV

EgoTV Egocentric Task Verification from Natural Language Task Descriptions
Other
27 stars 4 forks source link

About bad test performance. #8

Open huangzy-55 opened 3 months ago

huangzy-55 commented 3 months ago

Hi, I tried to run and test the code, but here are some issues, can you help me? Cause my GPU is not enough, I tried to save the representations after coca and then load them to train the model. But after doing it, the test performance is very bad of novel tasks and novel steps, but is good as the article saying of novel scenes and abstraction. For what reason this may happen?

RishiHazra commented 3 months ago

Hi @huangzy-55, to get some clarity, can you tell me exactly which baseline you're trying to reproduce or modify? From what I understand, you're using the CoCa representations for training the NSG model? Is that the case?

huangzy-55 commented 3 months ago

Yes. I am using vis_feat='coca',text_feat='coca',context_encoder='mha' this config to train NSG. And coca is not finetuned by me. I just download it from openclip of the given link.

RishiHazra commented 3 months ago

@huangzy-55
Can you tell me how many epochs you trained it for? Based on the training details in the paper (Sec 10.1), it should be --epochs 50.

If you've already tried that, can you try training with frames per segment = 12 instead of 20 (default value) like so --fp_seg 12? If the problem persists, I can try replicating it on my end.

huangzy-55 commented 3 months ago

I have trained 80 epochs. But I didn't try to change fp_seg, will it make any difference? For what reason?

RishiHazra commented 3 months ago

fp_seg = 12 should idelally increase your Novel Tasks performance, and likely decrease your Novel Steps performance. I'm just trying to narrow down the error. If it doesn't change as expected, there could be an issue with the loading of features.

huangzy-55 commented 3 months ago

OK. I will train this way to see the test performance.

huangzy-55 commented 4 days ago

Hi, I am still working on this, could you please upload the tsv file for CTV t5 pre-training? I really need it.

huangzy-55 commented 1 day ago

And, what is the groundtruth of train and test files in CTV you uploaded, and how to use the json files in meta folder ?

RishiHazra commented 1 day ago

@bc2754 Hi Brian, could you take a look at this? Thanks