search

facebookresearch / AVT

Code release for ICCV 2021 paper "Anticipative Video Transformer"
Apache License 2.0
152 stars 28 forks source link

Unable to reproduce val results #8

Closed haofengac closed 2 years ago

haofengac commented 2 years ago

Hi @rohitgirdhar, I'm trying to test the irCSN-152 (IG65M) model for EK-55. I used the model https://dl.fbaipublicfiles.com/avt/checkpoints/expts/10_ek55_avt_ig65m.txt/0/checkpoint.pth and the config expts/10_ek55_avt_ig65m.txt, and added these lines to the config:

test_only=true
train.init_from_model=[[${cwd}/DATA/models/10_ek55_avt_ig65m.pth]]

However, I'm getting

[2021-10-05 12:37:04,999][root][INFO] - Reading from resfiles
[2021-10-05 12:37:11,072][func.train][INFO] - []
[2021-10-05 12:37:11,073][root][INFO] - iter_time: 0.294328
[2021-10-05 12:37:11,073][root][INFO] - data_time: 0.135377
[2021-10-05 12:37:11,074][root][INFO] - loss: 6.164686
[2021-10-05 12:37:11,074][root][INFO] - acc1/action: 7.351763
[2021-10-05 12:37:11,074][root][INFO] - acc5/action: 19.931891
[2021-10-05 12:37:11,074][root][INFO] - cls_action: 6.134162
[2021-10-05 12:37:11,074][root][INFO] - feat: 0.030524

which is far from the 14.4 and 31.7 Top 1/5 performance. Do you know what might be wrong here?

rohitgirdhar commented 2 years ago

Hi @haofengac, this is strange. I just ran it again after downloading the checkpoint and got the following:

[2021-10-05 13:30:21,590][root][INFO] - iter_time: 0.226066
[2021-10-05 13:30:21,590][root][INFO] - data_time: 0.024945
[2021-10-05 13:30:21,590][root][INFO] - loss: 5.459376
[2021-10-05 13:30:21,591][root][INFO] - acc1/action: 14.374428
[2021-10-05 13:30:21,591][root][INFO] - acc5/action: 31.583868
[2021-10-05 13:30:21,591][root][INFO] - cls_action: 5.399561
[2021-10-05 13:30:21,591][root][INFO] - feat: 0.059814

Can you share the full log file? Also, can you try another model, say based on TSN features, to see if you're only seeing this issue with IG features or with TSN features too?

haofengac commented 2 years ago

Hi @rohitgirdhar, I re-downloaded the weights and called python launch.py -c expts/10_ek55_avt_ig65m.txt -g with the changes mentioned above and the full log is as follows:

[2021-10-05 14:45:38,069][py.warnings][WARNING] - /scr2/haofeng/anaconda3/envs/avt/lib/python3.7/site-packages/torchvision/__init__.py:78: UserWarning: video_reader video backend is not available. Please compile torchvision from source and try again
  warnings.warn(message)

[2021-10-05 14:45:41,181][func.train][INFO] - Dist info:
[2021-10-05 14:45:41,181][func.train][INFO] - torch version: 1.7.0
[2021-10-05 14:45:41,181][func.train][INFO] - torchvision version: 0.8.1
[2021-10-05 14:45:41,181][func.train][INFO] - hydra version: 1.1.0dev4
[2021-10-05 14:45:42,204][func.train][INFO] - Loading data
[2021-10-05 14:45:42,204][func.train][INFO] -    Loading datasets
[2021-10-05 14:45:42,210][func.train][INFO] - Creating the dataset object...
[2021-10-05 14:45:42,210][py.warnings][WARNING] - /scr2/haofeng/anaconda3/envs/avt/lib/python3.7/site-packages/hydra/_internal/instantiate/_instantiate2.py:105: UserWarning: `OmegaConf.is_none()` is deprecated, see https://github.com/omry/omegaconf/issues/547
  if config is None or OmegaConf.is_none(config):

[2021-10-05 14:45:42,215][py.warnings][WARNING] - /scr2/haofeng/anaconda3/envs/avt/lib/python3.7/site-packages/hydra/_internal/instantiate/_instantiate2.py:163: UserWarning: `OmegaConf.is_none()` is deprecated, see https://github.com/omry/omegaconf/issues/547
  if config is None or OmegaConf.is_none(config):

[2021-10-05 14:45:42,215][py.warnings][WARNING] - /scr2/haofeng/anaconda3/envs/avt/lib/python3.7/site-packages/hydra/_internal/instantiate/_instantiate2.py:27: UserWarning: `OmegaConf.is_none()` is deprecated, see https://github.com/omry/omegaconf/issues/547
  if OmegaConf.is_dict(x) and not OmegaConf.is_none(x):

[2021-10-05 14:45:42,231][root][INFO] - Loading original EPIC pkl annotations /sailhome/haofeng/code/ActionPartonomy/external/epic-kitchens-55-annotations/EPIC_train_action_labels.pkl
[2021-10-05 14:45:48,043][root][INFO] - Discarded 63 elements in anticipate conversion
[2021-10-05 14:45:48,213][root][INFO] - Created EPIC 0.1 dataset with 23430 samples
[2021-10-05 14:45:48,216][func.train][INFO] - Computing clips...
[2021-10-05 14:45:48,216][func.train][WARNING] - No video_clips present
[2021-10-05 14:45:48,216][func.train][INFO] - Created dataset with 23430 elts
[2021-10-05 14:45:48,222][func.train][INFO] - Creating the dataset object...
[2021-10-05 14:45:48,376][root][INFO] - Loading original EPIC pkl annotations /sailhome/haofeng/code/ActionPartonomy/external/epic-kitchens-55-annotations/EPIC_train_action_labels.pkl
[2021-10-05 14:45:53,696][root][INFO] - Discarded 10 elements in anticipate conversion
[2021-10-05 14:45:53,788][root][INFO] - Created EPIC 0.1 dataset with 4969 samples
[2021-10-05 14:45:53,789][func.train][INFO] - Computing clips...
[2021-10-05 14:45:53,789][func.train][WARNING] - No video_clips present
[2021-10-05 14:45:53,789][func.train][INFO] - Created dataset with 4969 elts
[2021-10-05 14:45:53,789][func.train][INFO] - Took 11
[2021-10-05 14:45:53,790][func.train][INFO] - Creating data loaders
[2021-10-05 14:45:53,791][func.train][INFO] - Creating model with {'action': 2513} classes
[2021-10-05 14:46:11,987][func.train][WARNING] - Could not init from /sailhome/haofeng/code/ActionPartonomy/DATA/models/10_ek55_avt_ig65m.pth: []
[2021-10-05 14:46:11,988][func.train][WARNING] - Unused keys in /sailhome/haofeng/code/ActionPartonomy/DATA/models/10_ek55_avt_ig65m.pth: []
[2021-10-05 14:46:12,738][func.train][INFO] - Using LR 0.000005 WD 0.000100 for parameters dict_keys(['__all__.future_predictor.encoder.weight', '__all__.future_predictor.decoder.weight', '__all__.future_predictor.gpt_model.wpe.weight', '__all__.future_predictor.gpt_model.h.0.ln_1.weight', '__all__.future_predictor.gpt_model.h.0.attn.c_attn.weight', '__all__.future_predictor.gpt_model.h.0.attn.c_proj.weight', '__all__.future_predictor.gpt_model.h.0.ln_2.weight', '__all__.future_predictor.gpt_model.h.0.mlp.c_fc.weight', '__all__.future_predictor.gpt_model.h.0.mlp.c_proj.weight', '__all__.future_predictor.gpt_model.h.1.ln_1.weight', '__all__.future_predictor.gpt_model.h.1.attn.c_attn.weight', '__all__.future_predictor.gpt_model.h.1.attn.c_proj.weight', '__all__.future_predictor.gpt_model.h.1.ln_2.weight', '__all__.future_predictor.gpt_model.h.1.mlp.c_fc.weight', '__all__.future_predictor.gpt_model.h.1.mlp.c_proj.weight', '__all__.future_predictor.gpt_model.h.2.ln_1.weight', '__all__.future_predictor.gpt_model.h.2.attn.c_attn.weight', '__all__.future_predictor.gpt_model.h.2.attn.c_proj.weight', '__all__.future_predictor.gpt_model.h.2.ln_2.weight', '__all__.future_predictor.gpt_model.h.2.mlp.c_fc.weight', '__all__.future_predictor.gpt_model.h.2.mlp.c_proj.weight', '__all__.future_predictor.gpt_model.h.3.ln_1.weight', '__all__.future_predictor.gpt_model.h.3.attn.c_attn.weight', '__all__.future_predictor.gpt_model.h.3.attn.c_proj.weight', '__all__.future_predictor.gpt_model.h.3.ln_2.weight', '__all__.future_predictor.gpt_model.h.3.mlp.c_fc.weight', '__all__.future_predictor.gpt_model.h.3.mlp.c_proj.weight', '__all__.future_predictor.gpt_model.h.4.ln_1.weight', '__all__.future_predictor.gpt_model.h.4.attn.c_attn.weight', '__all__.future_predictor.gpt_model.h.4.attn.c_proj.weight', '__all__.future_predictor.gpt_model.h.4.ln_2.weight', '__all__.future_predictor.gpt_model.h.4.mlp.c_fc.weight', '__all__.future_predictor.gpt_model.h.4.mlp.c_proj.weight', '__all__.future_predictor.gpt_model.h.5.ln_1.weight', '__all__.future_predictor.gpt_model.h.5.attn.c_attn.weight', '__all__.future_predictor.gpt_model.h.5.attn.c_proj.weight', '__all__.future_predictor.gpt_model.h.5.ln_2.weight', '__all__.future_predictor.gpt_model.h.5.mlp.c_fc.weight', '__all__.future_predictor.gpt_model.h.5.mlp.c_proj.weight', '__all__.future_predictor.gpt_model.h.6.ln_1.weight', '__all__.future_predictor.gpt_model.h.6.attn.c_attn.weight', '__all__.future_predictor.gpt_model.h.6.attn.c_proj.weight', '__all__.future_predictor.gpt_model.h.6.ln_2.weight', '__all__.future_predictor.gpt_model.h.6.mlp.c_fc.weight', '__all__.future_predictor.gpt_model.h.6.mlp.c_proj.weight', '__all__.future_predictor.gpt_model.h.7.ln_1.weight', '__all__.future_predictor.gpt_model.h.7.attn.c_attn.weight', '__all__.future_predictor.gpt_model.h.7.attn.c_proj.weight', '__all__.future_predictor.gpt_model.h.7.ln_2.weight', '__all__.future_predictor.gpt_model.h.7.mlp.c_fc.weight', '__all__.future_predictor.gpt_model.h.7.mlp.c_proj.weight', '__all__.future_predictor.gpt_model.h.8.ln_1.weight', '__all__.future_predictor.gpt_model.h.8.attn.c_attn.weight', '__all__.future_predictor.gpt_model.h.8.attn.c_proj.weight', '__all__.future_predictor.gpt_model.h.8.ln_2.weight', '__all__.future_predictor.gpt_model.h.8.mlp.c_fc.weight', '__all__.future_predictor.gpt_model.h.8.mlp.c_proj.weight', '__all__.future_predictor.gpt_model.h.9.ln_1.weight', '__all__.future_predictor.gpt_model.h.9.attn.c_attn.weight', '__all__.future_predictor.gpt_model.h.9.attn.c_proj.weight', '__all__.future_predictor.gpt_model.h.9.ln_2.weight', '__all__.future_predictor.gpt_model.h.9.mlp.c_fc.weight', '__all__.future_predictor.gpt_model.h.9.mlp.c_proj.weight', '__all__.future_predictor.gpt_model.h.10.ln_1.weight', '__all__.future_predictor.gpt_model.h.10.attn.c_attn.weight', '__all__.future_predictor.gpt_model.h.10.attn.c_proj.weight', '__all__.future_predictor.gpt_model.h.10.ln_2.weight', '__all__.future_predictor.gpt_model.h.10.mlp.c_fc.weight', '__all__.future_predictor.gpt_model.h.10.mlp.c_proj.weight', '__all__.future_predictor.gpt_model.h.11.ln_1.weight', '__all__.future_predictor.gpt_model.h.11.attn.c_attn.weight', '__all__.future_predictor.gpt_model.h.11.attn.c_proj.weight', '__all__.future_predictor.gpt_model.h.11.ln_2.weight', '__all__.future_predictor.gpt_model.h.11.mlp.c_fc.weight', '__all__.future_predictor.gpt_model.h.11.mlp.c_proj.weight', '__all__.future_predictor.gpt_model.ln_f.weight', '__all__.classifiers.action.weight'])
[2021-10-05 14:46:12,739][func.train][INFO] - Using LR 0.000005 WD 0.000100 for parameters dict_keys(['__all__.future_predictor.gpt_model.h.0.ln_1.bias', '__all__.future_predictor.gpt_model.h.0.attn.c_attn.bias', '__all__.future_predictor.gpt_model.h.0.attn.c_proj.bias', '__all__.future_predictor.gpt_model.h.0.ln_2.bias', '__all__.future_predictor.gpt_model.h.0.mlp.c_fc.bias', '__all__.future_predictor.gpt_model.h.0.mlp.c_proj.bias', '__all__.future_predictor.gpt_model.h.1.ln_1.bias', '__all__.future_predictor.gpt_model.h.1.attn.c_attn.bias', '__all__.future_predictor.gpt_model.h.1.attn.c_proj.bias', '__all__.future_predictor.gpt_model.h.1.ln_2.bias', '__all__.future_predictor.gpt_model.h.1.mlp.c_fc.bias', '__all__.future_predictor.gpt_model.h.1.mlp.c_proj.bias', '__all__.future_predictor.gpt_model.h.2.ln_1.bias', '__all__.future_predictor.gpt_model.h.2.attn.c_attn.bias', '__all__.future_predictor.gpt_model.h.2.attn.c_proj.bias', '__all__.future_predictor.gpt_model.h.2.ln_2.bias', '__all__.future_predictor.gpt_model.h.2.mlp.c_fc.bias', '__all__.future_predictor.gpt_model.h.2.mlp.c_proj.bias', '__all__.future_predictor.gpt_model.h.3.ln_1.bias', '__all__.future_predictor.gpt_model.h.3.attn.c_attn.bias', '__all__.future_predictor.gpt_model.h.3.attn.c_proj.bias', '__all__.future_predictor.gpt_model.h.3.ln_2.bias', '__all__.future_predictor.gpt_model.h.3.mlp.c_fc.bias', '__all__.future_predictor.gpt_model.h.3.mlp.c_proj.bias', '__all__.future_predictor.gpt_model.h.4.ln_1.bias', '__all__.future_predictor.gpt_model.h.4.attn.c_attn.bias', '__all__.future_predictor.gpt_model.h.4.attn.c_proj.bias', '__all__.future_predictor.gpt_model.h.4.ln_2.bias', '__all__.future_predictor.gpt_model.h.4.mlp.c_fc.bias', '__all__.future_predictor.gpt_model.h.4.mlp.c_proj.bias', '__all__.future_predictor.gpt_model.h.5.ln_1.bias', '__all__.future_predictor.gpt_model.h.5.attn.c_attn.bias', '__all__.future_predictor.gpt_model.h.5.attn.c_proj.bias', '__all__.future_predictor.gpt_model.h.5.ln_2.bias', '__all__.future_predictor.gpt_model.h.5.mlp.c_fc.bias', '__all__.future_predictor.gpt_model.h.5.mlp.c_proj.bias', '__all__.future_predictor.gpt_model.h.6.ln_1.bias', '__all__.future_predictor.gpt_model.h.6.attn.c_attn.bias', '__all__.future_predictor.gpt_model.h.6.attn.c_proj.bias', '__all__.future_predictor.gpt_model.h.6.ln_2.bias', '__all__.future_predictor.gpt_model.h.6.mlp.c_fc.bias', '__all__.future_predictor.gpt_model.h.6.mlp.c_proj.bias', '__all__.future_predictor.gpt_model.h.7.ln_1.bias', '__all__.future_predictor.gpt_model.h.7.attn.c_attn.bias', '__all__.future_predictor.gpt_model.h.7.attn.c_proj.bias', '__all__.future_predictor.gpt_model.h.7.ln_2.bias', '__all__.future_predictor.gpt_model.h.7.mlp.c_fc.bias', '__all__.future_predictor.gpt_model.h.7.mlp.c_proj.bias', '__all__.future_predictor.gpt_model.h.8.ln_1.bias', '__all__.future_predictor.gpt_model.h.8.attn.c_attn.bias', '__all__.future_predictor.gpt_model.h.8.attn.c_proj.bias', '__all__.future_predictor.gpt_model.h.8.ln_2.bias', '__all__.future_predictor.gpt_model.h.8.mlp.c_fc.bias', '__all__.future_predictor.gpt_model.h.8.mlp.c_proj.bias', '__all__.future_predictor.gpt_model.h.9.ln_1.bias', '__all__.future_predictor.gpt_model.h.9.attn.c_attn.bias', '__all__.future_predictor.gpt_model.h.9.attn.c_proj.bias', '__all__.future_predictor.gpt_model.h.9.ln_2.bias', '__all__.future_predictor.gpt_model.h.9.mlp.c_fc.bias', '__all__.future_predictor.gpt_model.h.9.mlp.c_proj.bias', '__all__.future_predictor.gpt_model.h.10.ln_1.bias', '__all__.future_predictor.gpt_model.h.10.attn.c_attn.bias', '__all__.future_predictor.gpt_model.h.10.attn.c_proj.bias', '__all__.future_predictor.gpt_model.h.10.ln_2.bias', '__all__.future_predictor.gpt_model.h.10.mlp.c_fc.bias', '__all__.future_predictor.gpt_model.h.10.mlp.c_proj.bias', '__all__.future_predictor.gpt_model.h.11.ln_1.bias', '__all__.future_predictor.gpt_model.h.11.attn.c_attn.bias', '__all__.future_predictor.gpt_model.h.11.attn.c_proj.bias', '__all__.future_predictor.gpt_model.h.11.ln_2.bias', '__all__.future_predictor.gpt_model.h.11.mlp.c_fc.bias', '__all__.future_predictor.gpt_model.h.11.mlp.c_proj.bias', '__all__.future_predictor.gpt_model.ln_f.bias', '__all__.classifiers.action.bias'])
[2021-10-05 14:46:12,748][func.train][INFO] - Wrapping model into DDP
[2021-10-05 14:46:12,792][func.train][INFO] - Starting test_only
[2021-10-05 14:46:12,793][func.train][INFO] - Running evaluation for dataset_eval
[2021-10-05 14:46:12,793][func.train][INFO] - Clearing /scr2/haofeng/outputs/expts/10_ek55_avt_ig65m.txt/local/results//*
[2021-10-05 14:46:13,521][func.train][INFO] - [] Test:  [  0/156]  eta: 0:01:38  iter_time: 0.6304  data_time: 0.1588  loss: 6.0605 (6.0605)  acc1/action: 12.5000 (12.5000)  acc5/action: 31.2500 (31.2500)  cls_action: 6.0296 (6.0296)  feat: 0.0309 (0.0309)  time: 0.6304  data: 0.1588  max mem: 4865
[2021-10-05 14:46:14,165][func.train][INFO] - [] Test:  [  2/156]  eta: 0:01:05  iter_time: 0.4236  data_time: 0.1534  loss: 6.2812 (6.2370)  acc1/action: 6.2500 (6.2500)  acc5/action: 12.5000 (18.7500)  cls_action: 6.2433 (6.2013)  feat: 0.0380 (0.0357)  time: 0.4236  data: 0.1534  max mem: 4881
[2021-10-05 14:46:14,755][func.train][INFO] - [] Test:  [  4/156]  eta: 0:00:56  iter_time: 0.3717  data_time: 0.1431  loss: 6.2812 (6.2511)  acc1/action: 3.1250 (4.3750)  acc5/action: 12.5000 (18.1250)  cls_action: 6.2433 (6.2159)  feat: 0.0369 (0.0352)  time: 0.3717  data: 0.1431  max mem: 4881
[2021-10-05 14:46:15,365][func.train][INFO] - [] Test:  [  6/156]  eta: 0:00:52  iter_time: 0.3522  data_time: 0.1421  loss: 6.3691 (6.3716)  acc1/action: 3.1250 (4.0179)  acc5/action: 12.5000 (16.9643)  cls_action: 6.3309 (6.3369)  feat: 0.0360 (0.0348)  time: 0.3522  data: 0.1421  max mem: 4882
[2021-10-05 14:46:15,942][func.train][INFO] - [] Test:  [  8/156]  eta: 0:00:49  iter_time: 0.3378  data_time: 0.1382  loss: 6.2812 (6.2040)  acc1/action: 6.2500 (4.8611)  acc5/action: 15.6250 (19.0972)  cls_action: 6.2433 (6.1697)  feat: 0.0333 (0.0343)  time: 0.3378  data: 0.1382  max mem: 4882
[2021-10-05 14:46:16,510][func.train][INFO] - [] Test:  [ 10/156]  eta: 0:00:47  iter_time: 0.3277  data_time: 0.1358  loss: 6.0605 (6.1654)  acc1/action: 6.2500 (5.6818)  acc5/action: 21.8750 (20.4545)  cls_action: 6.0296 (6.1305)  feat: 0.0333 (0.0350)  time: 0.3277  data: 0.1358  max mem: 4882
[2021-10-05 14:46:17,061][func.train][INFO] - [] Test:  [ 12/156]  eta: 0:00:46  iter_time: 0.3195  data_time: 0.1324  loss: 6.0605 (6.1814)  acc1/action: 6.2500 (5.7692)  acc5/action: 21.8750 (20.9135)  cls_action: 6.0296 (6.1475)  feat: 0.0327 (0.0339)  time: 0.3195  data: 0.1324  max mem: 4882
[2021-10-05 14:46:17,622][func.train][INFO] - [] Test:  [ 14/156]  eta: 0:00:44  iter_time: 0.3141  data_time: 0.1311  loss: 6.0605 (6.1371)  acc1/action: 6.2500 (5.6250)  acc5/action: 21.8750 (21.0417)  cls_action: 6.0296 (6.1037)  feat: 0.0320 (0.0334)  time: 0.3141  data: 0.1311  max mem: 4882
[2021-10-05 14:46:18,270][func.train][INFO] - [] Test:  [ 16/156]  eta: 0:00:44  iter_time: 0.3152  data_time: 0.1337  loss: 6.0664 (6.2001)  acc1/action: 6.2500 (6.0662)  acc5/action: 21.8750 (21.1397)  cls_action: 6.0364 (6.1673)  feat: 0.0320 (0.0328)  time: 0.3152  data: 0.1337  max mem: 4882
[2021-10-05 14:46:18,861][func.train][INFO] - [] Test:  [ 18/156]  eta: 0:00:43  iter_time: 0.3129  data_time: 0.1331  loss: 6.2812 (6.3021)  acc1/action: 6.2500 (6.2500)  acc5/action: 18.7500 (20.5592)  cls_action: 6.2433 (6.2697)  feat: 0.0318 (0.0324)  time: 0.3129  data: 0.1331  max mem: 4882
[2021-10-05 14:46:19,464][func.train][INFO] - [] Test:  [ 20/156]  eta: 0:00:42  iter_time: 0.2958  data_time: 0.1321  loss: 6.3691 (6.4441)  acc1/action: 6.2500 (5.9524)  acc5/action: 15.6250 (19.6429)  cls_action: 6.3309 (6.4123)  feat: 0.0314 (0.0318)  time: 0.2958  data: 0.1321  max mem: 4882
[2021-10-05 14:46:20,065][func.train][INFO] - [] Test:  [ 22/156]  eta: 0:00:41  iter_time: 0.2937  data_time: 0.1300  loss: 6.3824 (6.3881)  acc1/action: 6.2500 (6.2500)  acc5/action: 18.7500 (19.9728)  cls_action: 6.3545 (6.3567)  feat: 0.0297 (0.0314)  time: 0.2937  data: 0.1300  max mem: 4882
[2021-10-05 14:46:20,692][func.train][INFO] - [] Test:  [ 24/156]  eta: 0:00:41  iter_time: 0.2956  data_time: 0.1329  loss: 6.3824 (6.4327)  acc1/action: 6.2500 (6.0000)  acc5/action: 15.6250 (19.1250)  cls_action: 6.3545 (6.4015)  feat: 0.0290 (0.0312)  time: 0.2956  data: 0.1329  max mem: 4882
[2021-10-05 14:46:21,262][func.train][INFO] - [] Test:  [ 26/156]  eta: 0:00:40  iter_time: 0.2936  data_time: 0.1314  loss: 6.2858 (6.4390)  acc1/action: 6.2500 (6.1343)  acc5/action: 15.6250 (18.7500)  cls_action: 6.2429 (6.4072)  feat: 0.0290 (0.0318)  time: 0.2936  data: 0.1314  max mem: 4882
[2021-10-05 14:46:21,817][func.train][INFO] - [] Test:  [ 28/156]  eta: 0:00:39  iter_time: 0.2925  data_time: 0.1305  loss: 6.4946 (6.4939)  acc1/action: 6.2500 (5.9267)  acc5/action: 15.6250 (18.4267)  cls_action: 6.4666 (6.4624)  feat: 0.0281 (0.0314)  time: 0.2925  data: 0.1305  max mem: 4882
[2021-10-05 14:46:22,420][func.train][INFO] - [] Test:  [ 30/156]  eta: 0:00:38  iter_time: 0.2943  data_time: 0.1317  loss: 6.7386 (6.5098)  acc1/action: 6.2500 (5.6452)  acc5/action: 15.6250 (18.4476)  cls_action: 6.7110 (6.4787)  feat: 0.0279 (0.0311)  time: 0.2943  data: 0.1317  max mem: 4882
[2021-10-05 14:46:23,031][func.train][INFO] - [] Test:  [ 32/156]  eta: 0:00:37  iter_time: 0.2972  data_time: 0.1345  loss: 6.4946 (6.4473)  acc1/action: 6.2500 (5.8712)  acc5/action: 15.6250 (19.3182)  cls_action: 6.4666 (6.4162)  feat: 0.0280 (0.0311)  time: 0.2972  data: 0.1345  max mem: 4882
[2021-10-05 14:46:23,591][func.train][INFO] - [] Test:  [ 34/156]  eta: 0:00:37  iter_time: 0.2972  data_time: 0.1343  loss: 6.4946 (6.4337)  acc1/action: 6.2500 (5.7143)  acc5/action: 15.6250 (19.0179)  cls_action: 6.4666 (6.4028)  feat: 0.0276 (0.0309)  time: 0.2972  data: 0.1343  max mem: 4882
[2021-10-05 14:46:24,131][func.train][INFO] - [] Test:  [ 36/156]  eta: 0:00:36  iter_time: 0.2918  data_time: 0.1304  loss: 6.4946 (6.3355)  acc1/action: 6.2500 (7.1791)  acc5/action: 15.6250 (20.0169)  cls_action: 6.4666 (6.3041)  feat: 0.0280 (0.0314)  time: 0.2918  data: 0.1304  max mem: 4882
[2021-10-05 14:46:24,767][func.train][INFO] - [] Test:  [ 38/156]  eta: 0:00:35  iter_time: 0.2941  data_time: 0.1337  loss: 6.7386 (6.3814)  acc1/action: 3.1250 (6.9712)  acc5/action: 12.5000 (19.3109)  cls_action: 6.7110 (6.3503)  feat: 0.0270 (0.0311)  time: 0.2941  data: 0.1337  max mem: 4882
[2021-10-05 14:46:25,389][func.train][INFO] - [] Test:  [ 40/156]  eta: 0:00:35  iter_time: 0.2950  data_time: 0.1342  loss: 6.2858 (6.3642)  acc1/action: 3.1250 (7.5457)  acc5/action: 15.6250 (20.0457)  cls_action: 6.2429 (6.3325)  feat: 0.0281 (0.0317)  time: 0.2950  data: 0.1342  max mem: 4882
[2021-10-05 14:46:25,966][func.train][INFO] - [] Test:  [ 42/156]  eta: 0:00:34  iter_time: 0.2938  data_time: 0.1341  loss: 6.2858 (6.3881)  acc1/action: 3.1250 (7.7762)  acc5/action: 15.6250 (19.9128)  cls_action: 6.2429 (6.3562)  feat: 0.0282 (0.0319)  time: 0.2938  data: 0.1341  max mem: 4882
[2021-10-05 14:46:26,541][func.train][INFO] - [] Test:  [ 44/156]  eta: 0:00:33  iter_time: 0.2912  data_time: 0.1309  loss: 6.2858 (6.3854)  acc1/action: 3.1250 (7.5694)  acc5/action: 15.6250 (19.4444)  cls_action: 6.2429 (6.3536)  feat: 0.0282 (0.0318)  time: 0.2912  data: 0.1309  max mem: 4882
[2021-10-05 14:46:27,152][func.train][INFO] - [] Test:  [ 46/156]  eta: 0:00:33  iter_time: 0.2932  data_time: 0.1325  loss: 6.2896 (6.3817)  acc1/action: 3.1250 (7.5133)  acc5/action: 15.6250 (19.4149)  cls_action: 6.2562 (6.3500)  feat: 0.0281 (0.0317)  time: 0.2932  data: 0.1325  max mem: 4882
[2021-10-05 14:46:27,728][func.train][INFO] - [] Test:  [ 48/156]  eta: 0:00:32  iter_time: 0.2943  data_time: 0.1329  loss: 6.2896 (6.3937)  acc1/action: 3.1250 (7.2066)  acc5/action: 15.6250 (19.1327)  cls_action: 6.2562 (6.3620)  feat: 0.0281 (0.0316)  time: 0.2943  data: 0.1329  max mem: 4882
[2021-10-05 14:46:28,322][func.train][INFO] - [] Test:  [ 50/156]  eta: 0:00:31  iter_time: 0.2939  data_time: 0.1326  loss: 6.3074 (6.4067)  acc1/action: 3.1250 (7.1078)  acc5/action: 15.6250 (19.0564)  cls_action: 6.2797 (6.3750)  feat: 0.0290 (0.0316)  time: 0.2939  data: 0.1326  max mem: 4882
[2021-10-05 14:46:28,894][func.train][INFO] - [] Test:  [ 52/156]  eta: 0:00:31  iter_time: 0.2919  data_time: 0.1313  loss: 6.4471 (6.4260)  acc1/action: 3.1250 (6.9575)  acc5/action: 12.5000 (18.6321)  cls_action: 6.4159 (6.3945)  feat: 0.0281 (0.0315)  time: 0.2919  data: 0.1313  max mem: 4882
[2021-10-05 14:46:29,482][func.train][INFO] - [] Test:  [ 54/156]  eta: 0:00:30  iter_time: 0.2933  data_time: 0.1331  loss: 6.6849 (6.4511)  acc1/action: 3.1250 (6.8182)  acc5/action: 12.5000 (18.2386)  cls_action: 6.6568 (6.4197)  feat: 0.0282 (0.0314)  time: 0.2933  data: 0.1331  max mem: 4882
[2021-10-05 14:46:30,030][func.train][INFO] - [] Test:  [ 56/156]  eta: 0:00:29  iter_time: 0.2937  data_time: 0.1334  loss: 6.6849 (6.4714)  acc1/action: 3.1250 (6.8531)  acc5/action: 12.5000 (18.0921)  cls_action: 6.6568 (6.4400)  feat: 0.0282 (0.0313)  time: 0.2937  data: 0.1334  max mem: 4882
[2021-10-05 14:46:30,641][func.train][INFO] - [] Test:  [ 58/156]  eta: 0:00:29  iter_time: 0.2924  data_time: 0.1320  loss: 6.6290 (6.4804)  acc1/action: 3.1250 (6.8326)  acc5/action: 12.5000 (18.0085)  cls_action: 6.5988 (6.4491)  feat: 0.0290 (0.0313)  time: 0.2924  data: 0.1320  max mem: 4882
[2021-10-05 14:46:31,183][func.train][INFO] - [] Test:  [ 60/156]  eta: 0:00:28  iter_time: 0.2885  data_time: 0.1295  loss: 6.6849 (6.4961)  acc1/action: 3.1250 (6.8135)  acc5/action: 12.5000 (17.7254)  cls_action: 6.6568 (6.4649)  feat: 0.0282 (0.0312)  time: 0.2885  data: 0.1295  max mem: 4882
[2021-10-05 14:46:31,730][func.train][INFO] - [] Test:  [ 62/156]  eta: 0:00:27  iter_time: 0.2869  data_time: 0.1283  loss: 6.7116 (6.5150)  acc1/action: 3.1250 (6.6468)  acc5/action: 12.5000 (17.4107)  cls_action: 6.6818 (6.4839)  feat: 0.0282 (0.0311)  time: 0.2869  data: 0.1283  max mem: 4882
[2021-10-05 14:46:32,355][func.train][INFO] - [] Test:  [ 64/156]  eta: 0:00:27  iter_time: 0.2894  data_time: 0.1300  loss: 6.8305 (6.5349)  acc1/action: 3.1250 (6.5385)  acc5/action: 9.3750 (17.1635)  cls_action: 6.8037 (6.5040)  feat: 0.0280 (0.0309)  time: 0.2894  data: 0.1300  max mem: 4882
[2021-10-05 14:46:32,935][func.train][INFO] - [] Test:  [ 66/156]  eta: 0:00:26  iter_time: 0.2879  data_time: 0.1290  loss: 6.8305 (6.5190)  acc1/action: 3.1250 (6.5299)  acc5/action: 9.3750 (17.1642)  cls_action: 6.8037 (6.4883)  feat: 0.0280 (0.0307)  time: 0.2879  data: 0.1290  max mem: 4882
[2021-10-05 14:46:33,575][func.train][INFO] - [] Test:  [ 68/156]  eta: 0:00:26  iter_time: 0.2910  data_time: 0.1322  loss: 6.7987 (6.5090)  acc1/action: 3.1250 (6.4764)  acc5/action: 9.3750 (17.1196)  cls_action: 6.7725 (6.4784)  feat: 0.0279 (0.0306)  time: 0.2910  data: 0.1322  max mem: 4882
[2021-10-05 14:46:34,341][func.train][INFO] - [] Test:  [ 70/156]  eta: 0:00:25  iter_time: 0.2996  data_time: 0.1408  loss: 6.7987 (6.5128)  acc1/action: 3.1250 (6.4701)  acc5/action: 9.3750 (17.0335)  cls_action: 6.7725 (6.4823)  feat: 0.0275 (0.0305)  time: 0.2996  data: 0.1408  max mem: 4882
[2021-10-05 14:46:35,034][func.train][INFO] - [] Test:  [ 72/156]  eta: 0:00:25  iter_time: 0.3057  data_time: 0.1454  loss: 6.7280 (6.4898)  acc1/action: 3.1250 (6.3784)  acc5/action: 12.5000 (17.1233)  cls_action: 6.7000 (6.4593)  feat: 0.0279 (0.0305)  time: 0.3057  data: 0.1454  max mem: 4882
[2021-10-05 14:46:35,715][func.train][INFO] - [] Test:  [ 74/156]  eta: 0:00:24  iter_time: 0.3103  data_time: 0.1489  loss: 6.4905 (6.4813)  acc1/action: 3.1250 (6.3750)  acc5/action: 12.5000 (17.0833)  cls_action: 6.4648 (6.4509)  feat: 0.0275 (0.0304)  time: 0.3103  data: 0.1489  max mem: 4882
[2021-10-05 14:46:36,298][func.train][INFO] - [] Test:  [ 76/156]  eta: 0:00:24  iter_time: 0.3120  data_time: 0.1505  loss: 6.4905 (6.4837)  acc1/action: 3.1250 (6.4123)  acc5/action: 12.5000 (17.1672)  cls_action: 6.4648 (6.4534)  feat: 0.0272 (0.0303)  time: 0.3120  data: 0.1505  max mem: 4882
[2021-10-05 14:46:37,051][func.train][INFO] - [] Test:  [ 78/156]  eta: 0:00:23  iter_time: 0.3191  data_time: 0.1563  loss: 6.3486 (6.4857)  acc1/action: 3.1250 (6.4873)  acc5/action: 12.5000 (17.2468)  cls_action: 6.3223 (6.4554)  feat: 0.0268 (0.0303)  time: 0.3191  data: 0.1563  max mem: 4882
[2021-10-05 14:46:37,677][func.train][INFO] - [] Test:  [ 80/156]  eta: 0:00:23  iter_time: 0.3234  data_time: 0.1602  loss: 6.1281 (6.4551)  acc1/action: 3.1250 (6.9059)  acc5/action: 15.6250 (17.7469)  cls_action: 6.0993 (6.4246)  feat: 0.0268 (0.0305)  time: 0.3234  data: 0.1602  max mem: 4882
[2021-10-05 14:46:38,255][func.train][INFO] - [] Test:  [ 82/156]  eta: 0:00:22  iter_time: 0.3249  data_time: 0.1615  loss: 6.1281 (6.4739)  acc1/action: 6.2500 (6.8524)  acc5/action: 15.6250 (17.6205)  cls_action: 6.0993 (6.4435)  feat: 0.0265 (0.0304)  time: 0.3249  data: 0.1615  max mem: 4882
[2021-10-05 14:46:38,870][func.train][INFO] - [] Test:  [ 84/156]  eta: 0:00:21  iter_time: 0.3244  data_time: 0.1615  loss: 5.9965 (6.4588)  acc1/action: 6.2500 (6.9118)  acc5/action: 15.6250 (17.5735)  cls_action: 5.9642 (6.4285)  feat: 0.0265 (0.0303)  time: 0.3244  data: 0.1615  max mem: 4882
[2021-10-05 14:46:39,443][func.train][INFO] - [] Test:  [ 86/156]  eta: 0:00:21  iter_time: 0.3241  data_time: 0.1605  loss: 5.9295 (6.4393)  acc1/action: 9.3750 (7.0043)  acc5/action: 18.7500 (17.8520)  cls_action: 5.9014 (6.4091)  feat: 0.0272 (0.0303)  time: 0.3241  data: 0.1605  max mem: 4882
[2021-10-05 14:46:40,107][func.train][INFO] - [] Test:  [ 88/156]  eta: 0:00:20  iter_time: 0.3254  data_time: 0.1626  loss: 5.7870 (6.4075)  acc1/action: 9.3750 (7.1278)  acc5/action: 18.7500 (18.1531)  cls_action: 5.7561 (6.3773)  feat: 0.0275 (0.0302)  time: 0.3254  data: 0.1626  max mem: 4882
[2021-10-05 14:46:40,705][func.train][INFO] - [] Test:  [ 90/156]  eta: 0:00:20  iter_time: 0.3170  data_time: 0.1543  loss: 5.7870 (6.4014)  acc1/action: 9.3750 (7.1772)  acc5/action: 21.8750 (18.1662)  cls_action: 5.7561 (6.3712)  feat: 0.0281 (0.0302)  time: 0.3170  data: 0.1543  max mem: 4882
[2021-10-05 14:46:41,353][func.train][INFO] - [] Test:  [ 92/156]  eta: 0:00:19  iter_time: 0.3147  data_time: 0.1537  loss: 5.6795 (6.3736)  acc1/action: 12.5000 (7.4261)  acc5/action: 21.8750 (18.5148)  cls_action: 5.6523 (6.3432)  feat: 0.0281 (0.0305)  time: 0.3147  data: 0.1537  max mem: 4882
[2021-10-05 14:46:42,013][func.train][INFO] - [] Test:  [ 94/156]  eta: 0:00:18  iter_time: 0.3137  data_time: 0.1522  loss: 5.6237 (6.3695)  acc1/action: 12.5000 (7.4342)  acc5/action: 25.0000 (18.6184)  cls_action: 5.5911 (6.3391)  feat: 0.0281 (0.0304)  time: 0.3137  data: 0.1522  max mem: 4882
[2021-10-05 14:46:42,663][func.train][INFO] - [] Test:  [ 96/156]  eta: 0:00:18  iter_time: 0.3171  data_time: 0.1556  loss: 5.6237 (6.3777)  acc1/action: 9.3750 (7.3776)  acc5/action: 21.8750 (18.5245)  cls_action: 5.5911 (6.3473)  feat: 0.0284 (0.0304)  time: 0.3171  data: 0.1556  max mem: 4882
[2021-10-05 14:46:43,254][func.train][INFO] - [] Test:  [ 98/156]  eta: 0:00:17  iter_time: 0.3089  data_time: 0.1486  loss: 5.4015 (6.3559)  acc1/action: 9.3750 (7.4179)  acc5/action: 25.0000 (18.6869)  cls_action: 5.3720 (6.3255)  feat: 0.0290 (0.0304)  time: 0.3089  data: 0.1486  max mem: 4882
[2021-10-05 14:46:43,855][func.train][INFO] - [] Test:  [100/156]  eta: 0:00:17  iter_time: 0.3076  data_time: 0.1475  loss: 5.4015 (6.3189)  acc1/action: 9.3750 (7.6114)  acc5/action: 25.0000 (19.2141)  cls_action: 5.3720 (6.2885)  feat: 0.0290 (0.0304)  time: 0.3076  data: 0.1475  max mem: 4882
[2021-10-05 14:46:44,448][func.train][INFO] - [] Test:  [102/156]  eta: 0:00:16  iter_time: 0.3084  data_time: 0.1482  loss: 5.3630 (6.2933)  acc1/action: 9.3750 (7.7973)  acc5/action: 28.1250 (19.4782)  cls_action: 5.3370 (6.2629)  feat: 0.0295 (0.0304)  time: 0.3084  data: 0.1482  max mem: 4882
[2021-10-05 14:46:45,056][func.train][INFO] - [] Test:  [104/156]  eta: 0:00:15  iter_time: 0.3080  data_time: 0.1486  loss: 5.3630 (6.2856)  acc1/action: 9.3750 (7.8571)  acc5/action: 28.1250 (19.5536)  cls_action: 5.3370 (6.2553)  feat: 0.0295 (0.0303)  time: 0.3080  data: 0.1486  max mem: 4882
[2021-10-05 14:46:45,667][func.train][INFO] - [] Test:  [106/156]  eta: 0:00:15  iter_time: 0.3099  data_time: 0.1504  loss: 5.3630 (6.2728)  acc1/action: 9.3750 (7.8563)  acc5/action: 28.1250 (19.7722)  cls_action: 5.3370 (6.2426)  feat: 0.0295 (0.0302)  time: 0.3099  data: 0.1504  max mem: 4882
[2021-10-05 14:46:46,230][func.train][INFO] - [] Test:  [108/156]  eta: 0:00:14  iter_time: 0.3048  data_time: 0.1456  loss: 5.6237 (6.2719)  acc1/action: 9.3750 (7.7982)  acc5/action: 28.1250 (19.7821)  cls_action: 5.5911 (6.2418)  feat: 0.0294 (0.0302)  time: 0.3048  data: 0.1456  max mem: 4882
[2021-10-05 14:46:46,828][func.train][INFO] - [] Test:  [110/156]  eta: 0:00:14  iter_time: 0.3048  data_time: 0.1452  loss: 5.4015 (6.2554)  acc1/action: 9.3750 (7.7703)  acc5/action: 28.1250 (19.9324)  cls_action: 5.3720 (6.2253)  feat: 0.0294 (0.0301)  time: 0.3048  data: 0.1452  max mem: 4882
[2021-10-05 14:46:47,385][func.train][INFO] - [] Test:  [112/156]  eta: 0:00:13  iter_time: 0.3003  data_time: 0.1403  loss: 5.7040 (6.2595)  acc1/action: 9.3750 (7.6604)  acc5/action: 25.0000 (19.7179)  cls_action: 5.6745 (6.2293)  feat: 0.0294 (0.0301)  time: 0.3003  data: 0.1403  max mem: 4882
[2021-10-05 14:46:47,982][func.train][INFO] - [] Test:  [114/156]  eta: 0:00:12  iter_time: 0.2971  data_time: 0.1386  loss: 5.7938 (6.2664)  acc1/action: 6.2500 (7.5815)  acc5/action: 25.0000 (19.6196)  cls_action: 5.7692 (6.2363)  feat: 0.0291 (0.0301)  time: 0.2971  data: 0.1386  max mem: 4882
[2021-10-05 14:46:48,564][func.train][INFO] - [] Test:  [116/156]  eta: 0:00:12  iter_time: 0.2937  data_time: 0.1346  loss: 5.7040 (6.2568)  acc1/action: 6.2500 (7.5053)  acc5/action: 25.0000 (19.5780)  cls_action: 5.6745 (6.2268)  feat: 0.0284 (0.0301)  time: 0.2937  data: 0.1346  max mem: 4882
[2021-10-05 14:46:49,189][func.train][INFO] - [] Test:  [118/156]  eta: 0:00:11  iter_time: 0.2955  data_time: 0.1361  loss: 5.7040 (6.2415)  acc1/action: 6.2500 (7.5893)  acc5/action: 25.0000 (19.7742)  cls_action: 5.6745 (6.2112)  feat: 0.0291 (0.0303)  time: 0.2955  data: 0.1361  max mem: 4882
[2021-10-05 14:46:49,801][func.train][INFO] - [] Test:  [120/156]  eta: 0:00:10  iter_time: 0.2961  data_time: 0.1354  loss: 5.7040 (6.1998)  acc1/action: 6.2500 (7.7738)  acc5/action: 25.0000 (20.4029)  cls_action: 5.6745 (6.1694)  feat: 0.0291 (0.0304)  time: 0.2961  data: 0.1354  max mem: 4882
[2021-10-05 14:46:50,423][func.train][INFO] - [] Test:  [122/156]  eta: 0:00:10  iter_time: 0.2975  data_time: 0.1361  loss: 5.7040 (6.1884)  acc1/action: 6.2500 (7.8252)  acc5/action: 25.0000 (20.5539)  cls_action: 5.6745 (6.1580)  feat: 0.0291 (0.0304)  time: 0.2975  data: 0.1361  max mem: 4882
[2021-10-05 14:46:50,948][func.train][INFO] - [] Test:  [124/156]  eta: 0:00:09  iter_time: 0.2933  data_time: 0.1320  loss: 5.5172 (6.1809)  acc1/action: 6.2500 (7.8000)  acc5/action: 25.0000 (20.4250)  cls_action: 5.4841 (6.1506)  feat: 0.0292 (0.0304)  time: 0.2933  data: 0.1320  max mem: 4882
[2021-10-05 14:46:51,526][func.train][INFO] - [] Test:  [126/156]  eta: 0:00:09  iter_time: 0.2918  data_time: 0.1308  loss: 5.5172 (6.1758)  acc1/action: 6.2500 (7.7018)  acc5/action: 21.8750 (20.4232)  cls_action: 5.4841 (6.1454)  feat: 0.0300 (0.0304)  time: 0.2918  data: 0.1308  max mem: 4882
[2021-10-05 14:46:52,124][func.train][INFO] - [] Test:  [128/156]  eta: 0:00:08  iter_time: 0.2935  data_time: 0.1320  loss: 5.5172 (6.1707)  acc1/action: 6.2500 (7.7519)  acc5/action: 25.0000 (20.4942)  cls_action: 5.4841 (6.1402)  feat: 0.0301 (0.0304)  time: 0.2935  data: 0.1320  max mem: 4882
[2021-10-05 14:46:52,766][func.train][INFO] - [] Test:  [130/156]  eta: 0:00:07  iter_time: 0.2957  data_time: 0.1340  loss: 5.6508 (6.1673)  acc1/action: 6.2500 (7.6574)  acc5/action: 18.7500 (20.4198)  cls_action: 5.6175 (6.1369)  feat: 0.0301 (0.0304)  time: 0.2957  data: 0.1340  max mem: 4882
[2021-10-05 14:46:53,387][func.train][INFO] - [] Test:  [132/156]  eta: 0:00:07  iter_time: 0.2989  data_time: 0.1366  loss: 5.5172 (6.1624)  acc1/action: 6.2500 (7.6128)  acc5/action: 18.7500 (20.3477)  cls_action: 5.4841 (6.1320)  feat: 0.0301 (0.0304)  time: 0.2989  data: 0.1366  max mem: 4882
[2021-10-05 14:46:53,932][func.train][INFO] - [] Test:  [134/156]  eta: 0:00:06  iter_time: 0.2962  data_time: 0.1342  loss: 5.4801 (6.1491)  acc1/action: 6.2500 (7.5463)  acc5/action: 25.0000 (20.4861)  cls_action: 5.4486 (6.1187)  feat: 0.0302 (0.0304)  time: 0.2962  data: 0.1342  max mem: 4882
[2021-10-05 14:46:54,503][func.train][INFO] - [] Test:  [136/156]  eta: 0:00:06  iter_time: 0.2957  data_time: 0.1343  loss: 5.4801 (6.1438)  acc1/action: 6.2500 (7.5274)  acc5/action: 25.0000 (20.4608)  cls_action: 5.4486 (6.1135)  feat: 0.0302 (0.0303)  time: 0.2957  data: 0.1343  max mem: 4882
[2021-10-05 14:46:55,097][func.train][INFO] - [] Test:  [138/156]  eta: 0:00:05  iter_time: 0.2941  data_time: 0.1323  loss: 5.5172 (6.1414)  acc1/action: 6.2500 (7.5315)  acc5/action: 21.8750 (20.5036)  cls_action: 5.4841 (6.1111)  feat: 0.0302 (0.0303)  time: 0.2941  data: 0.1323  max mem: 4882
[2021-10-05 14:46:55,715][func.train][INFO] - [] Test:  [140/156]  eta: 0:00:04  iter_time: 0.2944  data_time: 0.1332  loss: 5.6508 (6.1392)  acc1/action: 6.2500 (7.4468)  acc5/action: 18.7500 (20.4566)  cls_action: 5.6175 (6.1090)  feat: 0.0301 (0.0303)  time: 0.2944  data: 0.1332  max mem: 4882
[2021-10-05 14:46:56,277][func.train][INFO] - [] Test:  [142/156]  eta: 0:00:04  iter_time: 0.2914  data_time: 0.1310  loss: 5.9320 (6.1377)  acc1/action: 6.2500 (7.4519)  acc5/action: 18.7500 (20.3453)  cls_action: 5.9048 (6.1074)  feat: 0.0297 (0.0303)  time: 0.2914  data: 0.1310  max mem: 4882
[2021-10-05 14:46:56,873][func.train][INFO] - [] Test:  [144/156]  eta: 0:00:03  iter_time: 0.2949  data_time: 0.1332  loss: 5.9320 (6.1318)  acc1/action: 6.2500 (7.4784)  acc5/action: 18.7500 (20.3448)  cls_action: 5.9048 (6.1014)  feat: 0.0301 (0.0304)  time: 0.2949  data: 0.1332  max mem: 4882
[2021-10-05 14:46:57,470][func.train][INFO] - [] Test:  [146/156]  eta: 0:00:03  iter_time: 0.2959  data_time: 0.1339  loss: 5.8343 (6.1253)  acc1/action: 6.2500 (7.5255)  acc5/action: 18.7500 (20.3444)  cls_action: 5.8029 (6.0949)  feat: 0.0293 (0.0304)  time: 0.2959  data: 0.1339  max mem: 4882
[2021-10-05 14:46:58,036][func.train][INFO] - [] Test:  [148/156]  eta: 0:00:02  iter_time: 0.2943  data_time: 0.1323  loss: 5.9320 (6.1494)  acc1/action: 3.1250 (7.4245)  acc5/action: 12.5000 (20.0923)  cls_action: 5.9048 (6.1190)  feat: 0.0291 (0.0304)  time: 0.2943  data: 0.1323  max mem: 4882
[2021-10-05 14:46:58,698][func.train][INFO] - [] Test:  [150/156]  eta: 0:00:01  iter_time: 0.2953  data_time: 0.1339  loss: 5.8343 (6.1457)  acc1/action: 6.2500 (7.4296)  acc5/action: 18.7500 (20.1780)  cls_action: 5.8029 (6.1151)  feat: 0.0301 (0.0307)  time: 0.2953  data: 0.1339  max mem: 4882
[2021-10-05 14:46:59,358][func.train][INFO] - [] Test:  [152/156]  eta: 0:00:01  iter_time: 0.2972  data_time: 0.1357  loss: 5.9499 (6.1489)  acc1/action: 6.2500 (7.3938)  acc5/action: 18.7500 (20.1797)  cls_action: 5.9113 (6.1183)  feat: 0.0293 (0.0306)  time: 0.2972  data: 0.1357  max mem: 4882
[2021-10-05 14:46:59,950][func.train][INFO] - [] Test:  [154/156]  eta: 0:00:00  iter_time: 0.2995  data_time: 0.1363  loss: 6.1148 (6.1597)  acc1/action: 6.2500 (7.3992)  acc5/action: 12.5000 (20.0605)  cls_action: 6.0856 (6.1291)  feat: 0.0293 (0.0306)  time: 0.2995  data: 0.1363  max mem: 4882
[2021-10-05 14:47:00,052][func.train][INFO] - [] Test: Total time: 0:00:47
[2021-10-05 14:47:03,192][root][INFO] - Reading from resfiles
[2021-10-05 14:47:09,567][func.train][INFO] - []
[2021-10-05 14:47:09,568][root][INFO] - iter_time: 0.301034
[2021-10-05 14:47:09,568][root][INFO] - data_time: 0.138168
[2021-10-05 14:47:09,568][root][INFO] - loss: 6.164686
[2021-10-05 14:47:09,568][root][INFO] - acc1/action: 7.351763
[2021-10-05 14:47:09,568][root][INFO] - acc5/action: 19.931891
[2021-10-05 14:47:09,568][root][INFO] - cls_action: 6.134162
[2021-10-05 14:47:09,569][root][INFO] - feat: 0.030524

I will try to test some other models now.

rohitgirdhar commented 2 years ago

To check in case the features are corrupted, I tried downloading the LMDB features again and running with that, and still get similar numbers

[2021-10-05 16:56:33,023][root][INFO] - acc1/action: 14.342949
[2021-10-05 16:56:33,023][root][INFO] - acc5/action: 31.612803

Would be great if you could try the TSN features, might help figure the issue. Also, can you try deleting all the logs/checkpoints from the output directory (likely OUTPUTS/expts/10_ek55_avt_ig65m.txt/) and try again?

haofengac commented 2 years ago

Hi @rohitgirdhar, I tested on expts/08_ek55_avt_tsn.txt and I believe I'm getting expected results:

[2021-10-06 00:25:26,612][root][INFO] - iter_time: 0.245223
[2021-10-06 00:25:26,613][root][INFO] - data_time: 0.129204
[2021-10-06 00:25:26,613][root][INFO] - loss: 7.132918
[2021-10-06 00:25:26,613][root][INFO] - acc1/action: 13.103187
[2021-10-06 00:25:26,613][root][INFO] - acc5/action: 28.098291
[2021-10-06 00:25:26,613][root][INFO] - cls_action: 5.715809
[2021-10-06 00:25:26,613][root][INFO] - feat: 1.417109

However, neither 10_ek55_avt_ig65m.txt nor 10_ek55_avt_ig65m_forAR.txt works. I've deleted the output directory, re-downloaded the weights, and added the below lines to the config txts:

test_only=true
train.init_from_model=[[${cwd}/DATA/models/10_ek55_avt_ig65m.pth]]

Results for 10_ek55_avt_ig65m.txt:

[2021-10-06 00:34:52,824][root][INFO] - iter_time: 0.344397
[2021-10-06 00:34:52,824][root][INFO] - data_time: 0.180417
[2021-10-06 00:34:52,825][root][INFO] - loss: 6.164686
[2021-10-06 00:34:52,825][root][INFO] - acc1/action: 7.351763
[2021-10-06 00:34:52,825][root][INFO] - acc5/action: 19.931891
[2021-10-06 00:34:52,825][root][INFO] - cls_action: 6.134162
[2021-10-06 00:34:52,825][root][INFO] - feat: 0.030524

Results for 10_ek55_avt_ig65m_forAR.txt:

[2021-10-06 00:39:18,820][root][INFO] - iter_time: 0.292118
[2021-10-06 00:39:18,820][root][INFO] - data_time: 0.134512
[2021-10-06 00:39:18,820][root][INFO] - loss: 9.247215
[2021-10-06 00:39:18,820][root][INFO] - acc1/action: 3.325321
[2021-10-06 00:39:18,820][root][INFO] - acc5/action: 8.653846
[2021-10-06 00:39:18,820][root][INFO] - cls_action: 9.217207
[2021-10-06 00:39:18,821][root][INFO] - feat: 0.030008
rohitgirdhar commented 2 years ago

Hmm that is strange. It seems then the problem might be with the IG65M features. Can you try re-downloading the LMDB file? I have already tried it with a fresh download of the LMDB file and it seems to work. And could you also try with the Epic Kitchens-100 IG65M LMDB file and try that experiment?

Btw for the AR numbers, I actually don't print them in the logs at the end, however they should be in the tensorboard files. So you can just run tensorboard on the output directory and see the AR5 numbers.

haofengac commented 2 years ago

Hi @rohitgirdhar, thank you for the suggestion! I've re-downloaded the EK-55 LMDB file and it works now. Also, EK-100 runs OK.

A side question: what is the GPU memory used to train 10_ek55_avt_ig65m.txt? I get CUDA out of memory with -l or -g during training, even when batch_size is set to 1. I noticed that in this config, the future predictor is larger than those in the other experiments:

+model.future_predictor.n_head=8
+model.future_predictor.n_layer=12

Is there a reason why the predictor is larger than the one used in EK-100? Would the performance be different if I use the same future_predictor as 04_ek100_avt_ig65m.txt, with which there's no CUDA out of memory problem?

rohitgirdhar commented 2 years ago

Great! The configs should run with a 16GB GPU. From my initial experiments I found that more heads/layers for EK55 did help in getting better performance. You can try with fewer though the performance might be a bit lower. Closing this task, but feel free to open another task if you face any other issues.