search

alibaba-mmai-research / HyRSM

Code for our CVPR 2022 Paper "Hybrid Relation Guided Set Matching for Few-shot Action Recognition".
Apache License 2.0
26 stars 5 forks source link

the NUM_TRAIN_TASKS and USE_CLASSIFICATION #7

Open yczhan opened 1 year ago

yczhan commented 1 year ago

Hi authors, I noticed that the NUM_TRAIN_TASKS under UCF setting is only 2500, is it enough to train the model? cuz I tried to reproduce your method, but the output seems strange (val_epoch "min_top1_err": 33.023999 , "top1_err": 66.112001) Besides, I also noticed that you implement supervised loss (USE_CLASSIFICATION=True) and set different weights for ucf and ssv2 (0.15 and 0.6, respectively), the data augmentations are also complex (including random erasing). I wonder how much of an impact these extra settings will have on the final result?

wangxiang1230 commented 1 year ago

Hi authors, I noticed that the NUM_TRAIN_TASKS under UCF setting is only 2500, is it enough to train the model? cuz I tried to reproduce your method, but the output seems strange (val_epoch "min_top1_err": 33.023999 , "top1_err": 66.112001) Besides, I also noticed that you implement supervised loss (USE_CLASSIFICATION=True) and set different weights for ucf and ssv2 (0.15 and 0.6, respectively), the data augmentations are also complex (including random erasing). I wonder how much of an impact these extra settings will have on the final result?

Hi, thanks for your attention. Since the ImageNet pre-trained ResNet-50 is relatively friendly for the UCF101 dataset, i.e. the UCF101 dataset is more appearance-based, 2500 iters are enough. For your reproduced results, could it be that the ImageNet pre-trained model was not loaded? Because the results are too low and heavily overfitted. Regarding supervised loss and data augmentation, we performed HyRSM on top of reproducing ITANet, and we found that we could not reproduce ITANet without supervised loss and data augmentation, and the results were about 2% worse for your reference.

yczhan commented 1 year ago

I downloaded the pretrained Resnet-50 and loaded it before training. I tried 1-shot UCF and I set the query_num = 1 due to limited CUDA memory, the results: (val_epoch "min_top1_err": 33.02 , "top1_err": 66.11). I found that when I set the query_num = 3, the results becomes (val epoch "min_top1_err": 26.56, "top1_err": 49.84). Does the number of query make such a big difference？ Besides, I read the log and found that the top1_err is best at first, then becomes worse during training. This is shown in the results above (26.56% at first and 49.84% at final). Also, the NUM_TRAIN_TASKS is only 2500 and the lr is still warming up at the end of training. Are these phenomena normal? or maybe you could share your training logs with me?

wangxiang1230 commented 1 year ago

Hi, for reproduce our results, you may need to set the same settings as ours. The log for 5-way 1-shot UCF101 (evaluation per 300 iters) is here:

[09/09 23:34:41][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "1/10", "eta": "111 days, 8:29:35", "iter": "50/250000", "loss": 7.711459, "lr": 0.000013, "time_diff": 3.848467, "top1_err": 28.571426, "top5_err": 0.000000} [09/09 23:37:05][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "1/10", "eta": "35 days, 8:52:15", "iter": "100/250000", "loss": 5.949836, "lr": 0.000016, "time_diff": 1.222423, "top1_err": 22.857141, "top5_err": 0.000000} [09/09 23:39:36][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "1/10", "eta": "252 days, 6:03:22", "iter": "150/250000", "loss": 4.992104, "lr": 0.000019, "time_diff": 8.718364, "top1_err": 24.285713, "top5_err": 0.000000} [09/09 23:41:52][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "1/10", "eta": "35 days, 13:41:24", "iter": "200/250000", "loss": 3.955957, "lr": 0.000021, "time_diff": 1.229412, "top1_err": 17.142857, "top5_err": 0.000000} [09/09 23:44:14][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "1/10", "eta": "216 days, 7:33:26", "iter": "250/250000", "loss": 3.184105, "lr": 0.000024, "time_diff": 7.476590, "top1_err": 14.285713, "top5_err": 0.000000}

[09/09 23:46:36][INFO] utils.bucket: 60: putting 'output/test/checkpoints/checkpoint_epoch_00001.pyth' to 'model/few-shot/Few_shot_Relation-Attention_final/UCF/v1/checkpoint_epoch_00001.pyth' [09/09 23:47:09][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:14:40", "gpu_mem": "27.09 GB", "iter": "50/2500", "time_diff": 0.359567, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:47:28][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:04:22", "gpu_mem": "27.09 GB", "iter": "100/2500", "time_diff": 0.109315, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:47:46][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:04:28", "gpu_mem": "27.09 GB", "iter": "150/2500", "time_diff": 0.114141, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:48:05][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:04:16", "gpu_mem": "27.09 GB", "iter": "200/2500", "time_diff": 0.111340, "top1_err": 0.000000, "top5_err": 0.000000} [09/09 23:48:21][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:04:08", "gpu_mem": "27.09 GB", "iter": "250/2500", "time_diff": 0.110474, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:48:35][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:04:02", "gpu_mem": "27.09 GB", "iter": "300/2500", "time_diff": 0.110003, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:48:49][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:04:09", "gpu_mem": "27.09 GB", "iter": "350/2500", "time_diff": 0.116002, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:49:03][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:31:46", "gpu_mem": "27.09 GB", "iter": "400/2500", "time_diff": 0.908024, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:49:17][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:03:47", "gpu_mem": "27.09 GB", "iter": "450/2500", "time_diff": 0.110890, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:49:30][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:17:46", "gpu_mem": "27.09 GB", "iter": "500/2500", "time_diff": 0.533087, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:49:43][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:03:38", "gpu_mem": "27.09 GB", "iter": "550/2500", "time_diff": 0.112168, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:49:54][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:11:41", "gpu_mem": "27.09 GB", "iter": "600/2500", "time_diff": 0.369428, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:50:06][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:14:04", "gpu_mem": "27.09 GB", "iter": "650/2500", "time_diff": 0.456663, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:50:19][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:03:27", "gpu_mem": "27.09 GB", "iter": "700/2500", "time_diff": 0.115359, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:50:31][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:15:54", "gpu_mem": "27.09 GB", "iter": "750/2500", "time_diff": 0.545533, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:50:43][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:03:09", "gpu_mem": "27.09 GB", "iter": "800/2500", "time_diff": 0.111549, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:50:54][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:03:05", "gpu_mem": "27.09 GB", "iter": "850/2500", "time_diff": 0.112293, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:51:06][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:04:35", "gpu_mem": "27.09 GB", "iter": "900/2500", "time_diff": 0.172106, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:51:17][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:05:14", "gpu_mem": "27.09 GB", "iter": "950/2500", "time_diff": 0.203196, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:51:29][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:02:48", "gpu_mem": "27.09 GB", "iter": "1000/2500", "time_diff": 0.112029, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:51:41][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:02:46", "gpu_mem": "27.09 GB", "iter": "1050/2500", "time_diff": 0.114718, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:51:53][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:02:36", "gpu_mem": "27.09 GB", "iter": "1100/2500", "time_diff": 0.112047, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:52:05][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:02:29", "gpu_mem": "27.09 GB", "iter": "1150/2500", "time_diff": 0.111061, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:52:17][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:10:33", "gpu_mem": "27.09 GB", "iter": "1200/2500", "time_diff": 0.487579, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:52:28][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:02:18", "gpu_mem": "27.09 GB", "iter": "1250/2500", "time_diff": 0.111185, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:52:40][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:02:15", "gpu_mem": "27.09 GB", "iter": "1300/2500", "time_diff": 0.112725, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:52:52][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:02:17", "gpu_mem": "27.09 GB", "iter": "1350/2500", "time_diff": 0.119642, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:53:03][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:02:07", "gpu_mem": "27.09 GB", "iter": "1400/2500", "time_diff": 0.116044, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:53:15][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:01:55", "gpu_mem": "27.09 GB", "iter": "1450/2500", "time_diff": 0.109696, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:53:26][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:01:51", "gpu_mem": "27.09 GB", "iter": "1500/2500", "time_diff": 0.111350, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:53:38][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:08:43", "gpu_mem": "27.09 GB", "iter": "1550/2500", "time_diff": 0.550837, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:53:49][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:01:42", "gpu_mem": "27.09 GB", "iter": "1600/2500", "time_diff": 0.114408, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:54:01][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:01:37", "gpu_mem": "27.09 GB", "iter": "1650/2500", "time_diff": 0.114540, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:54:13][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:06:17", "gpu_mem": "27.09 GB", "iter": "1700/2500", "time_diff": 0.471832, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:54:24][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:01:26", "gpu_mem": "27.09 GB", "iter": "1750/2500", "time_diff": 0.115058, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:54:35][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:01:20", "gpu_mem": "27.09 GB", "iter": "1800/2500", "time_diff": 0.114749, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:54:47][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:01:12", "gpu_mem": "27.09 GB", "iter": "1850/2500", "time_diff": 0.112099, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:55:00][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:05:23", "gpu_mem": "27.09 GB", "iter": "1900/2500", "time_diff": 0.538911, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:55:12][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:01:01", "gpu_mem": "27.09 GB", "iter": "1950/2500", "time_diff": 0.112354, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:55:24][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:00:58", "gpu_mem": "27.09 GB", "iter": "2000/2500", "time_diff": 0.116810, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:55:36][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:00:52", "gpu_mem": "27.09 GB", "iter": "2050/2500", "time_diff": 0.117685, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:55:48][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:00:49", "gpu_mem": "27.09 GB", "iter": "2100/2500", "time_diff": 0.124157, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:56:00][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:00:41", "gpu_mem": "27.09 GB", "iter": "2150/2500", "time_diff": 0.119030, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:56:12][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:00:34", "gpu_mem": "27.09 GB", "iter": "2200/2500", "time_diff": 0.114651, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:56:24][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:02:35", "gpu_mem": "27.09 GB", "iter": "2250/2500", "time_diff": 0.622380, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:56:36][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:00:22", "gpu_mem": "27.09 GB", "iter": "2300/2500", "time_diff": 0.111999, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:56:48][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:00:16", "gpu_mem": "27.09 GB", "iter": "2350/2500", "time_diff": 0.111899, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:57:02][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:00:11", "gpu_mem": "27.09 GB", "iter": "2400/2500", "time_diff": 0.112014, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:57:15][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:00:05", "gpu_mem": "27.09 GB", "iter": "2450/2500", "time_diff": 0.116999, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:57:29][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:00:00", "gpu_mem": "27.09 GB", "iter": "2500/2500", "time_diff": 0.108913, "top1_err": 19.999998, "top5_err": 0.000000} [09/09 23:57:29][INFO] utils.logging: 90: {"RAM": "0.01/585.94 GB", "_type": "val_epoch", "epoch": "1/10", "gpu_mem": "27.09 GB", "min_top1_err": 19.007999, "min_top5_err": 0.000000, "time_diff": 0.567919, "top1_err": 19.007999, "top5_err": 0.000000} [09/09 23:57:30][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "1/10", "eta": "19347 days, 7:38:49", "iter": "300/250000", "loss": 2.774728, "lr": 0.000027, "time_diff": 668.723579, "top1_err": 14.285713, "top5_err": 0.000000} [09/09 23:59:34][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "1/10", "eta": "107 days, 5:54:31", "iter": "350/250000", "loss": 2.504072, "lr": 0.000030, "time_diff": 3.706947, "top1_err": 12.857142, "top5_err": 0.000000} [09/10 00:01:52][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "1/10", "eta": "34 days, 22:21:54", "iter": "400/250000", "loss": 1.967621, "lr": 0.000033, "time_diff": 1.207439, "top1_err": 8.571428, "top5_err": 0.000000} [09/10 00:04:09][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "1/10", "eta": "202 days, 1:07:04", "iter": "450/250000", "loss": 1.837473, "lr": 0.000036, "time_diff": 6.983987, "top1_err": 8.571428, "top5_err": 0.000000} [09/10 00:06:21][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "1/10", "eta": "35 days, 10:08:12", "iter": "500/250000", "loss": 1.635736, "lr": 0.000039, "time_diff": 1.224442, "top1_err": 10.000000, "top5_err": 0.000000} [09/10 00:08:35][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "1/10", "eta": "200 days, 6:11:49", "iter": "550/250000", "loss": 1.572966, "lr": 0.000041, "time_diff": 6.922447, "top1_err": 5.714285, "top5_err": 0.000000}

[09/10 00:10:58][INFO] utils.bucket: 60: putting 'output/test/checkpoints/checkpoint_epoch_00001.pyth' to 'model/few-shot/Few_shot_Relation-Attention_final/UCF/v1/checkpoint_epoch_00001.pyth' [09/10 00:11:26][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:05:03", "gpu_mem": "27.09 GB", "iter": "50/2500", "time_diff": 0.124000, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:11:38][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:04:27", "gpu_mem": "27.09 GB", "iter": "100/2500", "time_diff": 0.111529, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:11:50][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:16:34", "gpu_mem": "27.09 GB", "iter": "150/2500", "time_diff": 0.423382, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:12:04][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:04:15", "gpu_mem": "27.09 GB", "iter": "200/2500", "time_diff": 0.111018, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:12:17][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:04:19", "gpu_mem": "27.09 GB", "iter": "250/2500", "time_diff": 0.115280, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:12:30][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:04:08", "gpu_mem": "27.09 GB", "iter": "300/2500", "time_diff": 0.112777, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:12:44][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:04:01", "gpu_mem": "27.09 GB", "iter": "350/2500", "time_diff": 0.112360, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:12:58][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:03:58", "gpu_mem": "27.09 GB", "iter": "400/2500", "time_diff": 0.113510, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:13:12][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:12:33", "gpu_mem": "27.09 GB", "iter": "450/2500", "time_diff": 0.367542, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:13:26][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:20:43", "gpu_mem": "27.09 GB", "iter": "500/2500", "time_diff": 0.621785, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:13:40][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:26:42", "gpu_mem": "27.09 GB", "iter": "550/2500", "time_diff": 0.821962, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:13:53][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:03:30", "gpu_mem": "27.09 GB", "iter": "600/2500", "time_diff": 0.110844, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:14:07][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:22:41", "gpu_mem": "27.09 GB", "iter": "650/2500", "time_diff": 0.735973, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:14:21][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:10:02", "gpu_mem": "27.09 GB", "iter": "700/2500", "time_diff": 0.334623, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:14:36][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:19:07", "gpu_mem": "27.09 GB", "iter": "750/2500", "time_diff": 0.655700, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:14:48][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:06:23", "gpu_mem": "27.09 GB", "iter": "800/2500", "time_diff": 0.225872, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:15:01][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:18:42", "gpu_mem": "27.09 GB", "iter": "850/2500", "time_diff": 0.680268, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:15:14][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:03:10", "gpu_mem": "27.09 GB", "iter": "900/2500", "time_diff": 0.119142, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:15:26][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:02:55", "gpu_mem": "27.09 GB", "iter": "950/2500", "time_diff": 0.113500, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:15:40][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:04:24", "gpu_mem": "27.09 GB", "iter": "1000/2500", "time_diff": 0.176486, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:15:52][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:02:51", "gpu_mem": "27.09 GB", "iter": "1050/2500", "time_diff": 0.118468, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:16:06][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:02:38", "gpu_mem": "27.09 GB", "iter": "1100/2500", "time_diff": 0.113453, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:16:19][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:02:32", "gpu_mem": "27.09 GB", "iter": "1150/2500", "time_diff": 0.113313, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:16:32][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:02:31", "gpu_mem": "27.09 GB", "iter": "1200/2500", "time_diff": 0.116473, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:16:45][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:02:17", "gpu_mem": "27.09 GB", "iter": "1250/2500", "time_diff": 0.110247, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:16:58][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:02:25", "gpu_mem": "27.09 GB", "iter": "1300/2500", "time_diff": 0.121470, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:17:11][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:03:58", "gpu_mem": "27.09 GB", "iter": "1350/2500", "time_diff": 0.206987, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:17:24][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:02:04", "gpu_mem": "27.09 GB", "iter": "1400/2500", "time_diff": 0.112800, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:17:35][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:02:01", "gpu_mem": "27.09 GB", "iter": "1450/2500", "time_diff": 0.115862, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:17:47][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:01:56", "gpu_mem": "27.09 GB", "iter": "1500/2500", "time_diff": 0.116170, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:18:00][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:01:51", "gpu_mem": "27.09 GB", "iter": "1550/2500", "time_diff": 0.117865, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:18:12][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:01:40", "gpu_mem": "27.09 GB", "iter": "1600/2500", "time_diff": 0.111213, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:18:24][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:07:11", "gpu_mem": "27.09 GB", "iter": "1650/2500", "time_diff": 0.507727, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:18:36][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:01:30", "gpu_mem": "27.09 GB", "iter": "1700/2500", "time_diff": 0.113232, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:18:48][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:01:22", "gpu_mem": "27.09 GB", "iter": "1750/2500", "time_diff": 0.110607, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:18:59][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:02:15", "gpu_mem": "27.09 GB", "iter": "1800/2500", "time_diff": 0.192909, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:19:11][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:04:53", "gpu_mem": "27.09 GB", "iter": "1850/2500", "time_diff": 0.451823, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:19:23][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:03:12", "gpu_mem": "27.09 GB", "iter": "1900/2500", "time_diff": 0.320685, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:19:35][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:01:03", "gpu_mem": "27.09 GB", "iter": "1950/2500", "time_diff": 0.114679, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:19:48][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:04:12", "gpu_mem": "27.09 GB", "iter": "2000/2500", "time_diff": 0.505846, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:19:59][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:02:05", "gpu_mem": "27.09 GB", "iter": "2050/2500", "time_diff": 0.278028, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:20:11][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:00:45", "gpu_mem": "27.09 GB", "iter": "2100/2500", "time_diff": 0.113668, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:20:23][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:00:38", "gpu_mem": "27.09 GB", "iter": "2150/2500", "time_diff": 0.110916, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:20:34][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:00:36", "gpu_mem": "27.09 GB", "iter": "2200/2500", "time_diff": 0.120661, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:20:46][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:00:28", "gpu_mem": "27.09 GB", "iter": "2250/2500", "time_diff": 0.113459, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:20:58][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:00:22", "gpu_mem": "27.09 GB", "iter": "2300/2500", "time_diff": 0.113105, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:21:10][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:00:16", "gpu_mem": "27.09 GB", "iter": "2350/2500", "time_diff": 0.112240, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:21:21][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:00:11", "gpu_mem": "27.09 GB", "iter": "2400/2500", "time_diff": 0.116359, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:21:33][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:00:05", "gpu_mem": "27.09 GB", "iter": "2450/2500", "time_diff": 0.117999, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:21:46][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "1/10", "eta": "0:00:00", "gpu_mem": "27.09 GB", "iter": "2500/2500", "time_diff": 0.111291, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:21:46][INFO] utils.logging: 90: {"RAM": "0.01/585.94 GB", "_type": "val_epoch", "epoch": "1/10", "gpu_mem": "27.09 GB", "min_top1_err": 18.167999, "min_top5_err": 0.000000, "time_diff": 0.498578, "top1_err": 18.167999, "top5_err": 0.000000} [09/10 00:21:47][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "1/10", "eta": "19213 days, 0:59:15", "iter": "600/250000", "loss": 1.536787, "lr": 0.000044, "time_diff": 664.162101, "top1_err": 5.714285, "top5_err": 0.000000} [09/10 00:23:58][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "1/10", "eta": "36 days, 6:37:37", "iter": "650/250000", "loss": 1.300092, "lr": 0.000047, "time_diff": 1.254029, "top1_err": 5.714285, "top5_err": 0.000000} [09/10 00:26:20][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "1/10", "eta": "35 days, 1:12:37", "iter": "700/250000", "loss": 1.180781, "lr": 0.000050, "time_diff": 1.211682, "top1_err": 5.714285, "top5_err": 0.000000} [09/10 00:28:43][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "2/10", "eta": "140 days, 6:22:41", "iter": "750/250000", "loss": 1.003870, "lr": 0.000050, "time_diff": 5.388001, "top1_err": 2.857143, "top5_err": 0.000000} [09/10 00:30:59][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "2/10", "eta": "31 days, 15:32:54", "iter": "800/250000", "loss": 0.967570, "lr": 0.000050, "time_diff": 1.215710, "top1_err": 5.714285, "top5_err": 0.000000} [09/10 00:33:19][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "2/10", "eta": "158 days, 18:45:27", "iter": "850/250000", "loss": 0.794633, "lr": 0.000050, "time_diff": 6.099516, "top1_err": 2.857143, "top5_err": 0.000000}

[09/10 00:35:47][INFO] utils.bucket: 60: putting 'output/test/checkpoints/checkpoint_epoch_00002.pyth' to 'model/few-shot/Few_shot_Relation-Attention_final/UCF/v1/checkpoint_epoch_00002.pyth' [09/10 00:36:15][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:11:30", "gpu_mem": "27.09 GB", "iter": "50/2500", "time_diff": 0.281743, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:36:30][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:04:26", "gpu_mem": "27.09 GB", "iter": "100/2500", "time_diff": 0.110972, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:36:45][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:04:17", "gpu_mem": "27.09 GB", "iter": "150/2500", "time_diff": 0.109466, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:36:59][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:04:12", "gpu_mem": "27.09 GB", "iter": "200/2500", "time_diff": 0.109875, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:37:19][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:04:10", "gpu_mem": "27.09 GB", "iter": "250/2500", "time_diff": 0.111473, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:37:37][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:04:03", "gpu_mem": "27.09 GB", "iter": "300/2500", "time_diff": 0.110701, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:37:50][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:04:02", "gpu_mem": "27.09 GB", "iter": "350/2500", "time_diff": 0.112642, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:38:05][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:04:04", "gpu_mem": "27.09 GB", "iter": "400/2500", "time_diff": 0.116335, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:38:20][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:03:52", "gpu_mem": "27.09 GB", "iter": "450/2500", "time_diff": 0.113597, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:38:34][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:24:01", "gpu_mem": "27.09 GB", "iter": "500/2500", "time_diff": 0.720910, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:38:47][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:03:33", "gpu_mem": "27.09 GB", "iter": "550/2500", "time_diff": 0.109289, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:39:01][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:03:35", "gpu_mem": "27.09 GB", "iter": "600/2500", "time_diff": 0.113502, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:39:14][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:03:31", "gpu_mem": "27.09 GB", "iter": "650/2500", "time_diff": 0.114090, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:39:27][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:03:20", "gpu_mem": "27.09 GB", "iter": "700/2500", "time_diff": 0.111563, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:39:40][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:03:11", "gpu_mem": "27.09 GB", "iter": "750/2500", "time_diff": 0.109543, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:39:53][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:10:18", "gpu_mem": "27.09 GB", "iter": "800/2500", "time_diff": 0.363817, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:40:06][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:03:00", "gpu_mem": "27.09 GB", "iter": "850/2500", "time_diff": 0.109554, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:40:20][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:13:55", "gpu_mem": "27.09 GB", "iter": "900/2500", "time_diff": 0.521929, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:40:36][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:02:58", "gpu_mem": "27.09 GB", "iter": "950/2500", "time_diff": 0.114883, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:40:49][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:16:26", "gpu_mem": "27.09 GB", "iter": "1000/2500", "time_diff": 0.657358, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:41:01][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:02:41", "gpu_mem": "27.09 GB", "iter": "1050/2500", "time_diff": 0.111371, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:41:14][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:12:59", "gpu_mem": "27.09 GB", "iter": "1100/2500", "time_diff": 0.556479, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:41:26][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:02:30", "gpu_mem": "27.09 GB", "iter": "1150/2500", "time_diff": 0.111360, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:41:39][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:11:55", "gpu_mem": "27.09 GB", "iter": "1200/2500", "time_diff": 0.550547, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:41:51][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:11:38", "gpu_mem": "27.09 GB", "iter": "1250/2500", "time_diff": 0.559005, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:42:03][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:02:14", "gpu_mem": "27.09 GB", "iter": "1300/2500", "time_diff": 0.112258, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:42:15][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:07:09", "gpu_mem": "27.09 GB", "iter": "1350/2500", "time_diff": 0.373859, "top1_err": 9.999999, "top5_err": 0.000000} [09/10 00:42:28][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:02:09", "gpu_mem": "27.09 GB", "iter": "1400/2500", "time_diff": 0.117844, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:42:40][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:02:01", "gpu_mem": "27.09 GB", "iter": "1450/2500", "time_diff": 0.115822, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:42:51][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:01:54", "gpu_mem": "27.09 GB", "iter": "1500/2500", "time_diff": 0.114698, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:43:04][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:07:19", "gpu_mem": "27.09 GB", "iter": "1550/2500", "time_diff": 0.462845, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:43:16][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:05:50", "gpu_mem": "27.09 GB", "iter": "1600/2500", "time_diff": 0.389800, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:43:28][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:01:36", "gpu_mem": "27.09 GB", "iter": "1650/2500", "time_diff": 0.113978, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:43:41][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:01:29", "gpu_mem": "27.09 GB", "iter": "1700/2500", "time_diff": 0.112021, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:43:54][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:07:35", "gpu_mem": "27.09 GB", "iter": "1750/2500", "time_diff": 0.607052, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:44:06][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:01:18", "gpu_mem": "27.09 GB", "iter": "1800/2500", "time_diff": 0.112033, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:44:18][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:01:14", "gpu_mem": "27.09 GB", "iter": "1850/2500", "time_diff": 0.114623, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:44:32][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:01:11", "gpu_mem": "27.09 GB", "iter": "1900/2500", "time_diff": 0.119092, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:44:44][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:01:04", "gpu_mem": "27.09 GB", "iter": "1950/2500", "time_diff": 0.116920, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:44:57][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:03:20", "gpu_mem": "27.09 GB", "iter": "2000/2500", "time_diff": 0.401614, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:45:10][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:05:02", "gpu_mem": "27.09 GB", "iter": "2050/2500", "time_diff": 0.672555, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:45:22][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:00:46", "gpu_mem": "27.09 GB", "iter": "2100/2500", "time_diff": 0.117051, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:45:35][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:00:41", "gpu_mem": "27.09 GB", "iter": "2150/2500", "time_diff": 0.117863, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:45:49][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:02:44", "gpu_mem": "27.09 GB", "iter": "2200/2500", "time_diff": 0.547219, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:46:01][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:00:28", "gpu_mem": "27.09 GB", "iter": "2250/2500", "time_diff": 0.114735, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:46:15][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:01:41", "gpu_mem": "27.09 GB", "iter": "2300/2500", "time_diff": 0.507457, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:46:33][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:01:24", "gpu_mem": "27.09 GB", "iter": "2350/2500", "time_diff": 0.564393, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:46:48][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:00:48", "gpu_mem": "27.09 GB", "iter": "2400/2500", "time_diff": 0.483054, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:47:02][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:00:34", "gpu_mem": "27.09 GB", "iter": "2450/2500", "time_diff": 0.699943, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:47:17][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:00:00", "gpu_mem": "27.09 GB", "iter": "2500/2500", "time_diff": 0.110606, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 00:47:17][INFO] utils.logging: 90: {"RAM": "0.01/585.94 GB", "_type": "val_epoch", "epoch": "2/10", "gpu_mem": "27.09 GB", "min_top1_err": 17.079999, "min_top5_err": 0.000000, "time_diff": 0.464428, "top1_err": 17.079999, "top5_err": 0.000000} [09/10 00:47:18][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "2/10", "eta": "18387 days, 9:39:44", "iter": "900/250000", "loss": 0.769909, "lr": 0.000050, "time_diff": 706.358803, "top1_err": 2.857143, "top5_err": 0.000000} [09/10 00:49:34][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "2/10", "eta": "45 days, 1:29:38", "iter": "950/250000", "loss": 0.646096, "lr": 0.000050, "time_diff": 1.731121, "top1_err": 2.857143, "top5_err": 0.000000} [09/10 00:51:57][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "2/10", "eta": "31 days, 9:55:03", "iter": "1000/250000", "loss": 0.673324, "lr": 0.000050, "time_diff": 1.206805, "top1_err": 2.857143, "top5_err": 0.000000} [09/10 00:54:21][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "2/10", "eta": "47 days, 10:43:39", "iter": "1050/250000", "loss": 0.546911, "lr": 0.000050, "time_diff": 1.822815, "top1_err": 2.857143, "top5_err": 0.000000} [09/10 00:56:37][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "2/10", "eta": "31 days, 14:34:46", "iter": "1100/250000", "loss": 0.536771, "lr": 0.000050, "time_diff": 1.214321, "top1_err": 2.857143, "top5_err": 0.000000} [09/10 00:58:54][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "2/10", "eta": "85 days, 10:17:20", "iter": "1150/250000", "loss": 0.507801, "lr": 0.000050, "time_diff": 3.282140, "top1_err": 0.000000, "top5_err": 0.000000}

[09/10 01:01:25][INFO] utils.bucket: 60: putting 'output/test/checkpoints/checkpoint_epoch_00002.pyth' to 'model/few-shot/Few_shot_Relation-Attention_final/UCF/v1/checkpoint_epoch_00002.pyth' [09/10 01:01:57][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:33:10", "gpu_mem": "27.09 GB", "iter": "50/2500", "time_diff": 0.812588, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:02:12][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:04:24", "gpu_mem": "27.09 GB", "iter": "100/2500", "time_diff": 0.110273, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:02:29][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:14:42", "gpu_mem": "27.09 GB", "iter": "150/2500", "time_diff": 0.375395, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:02:46][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:04:23", "gpu_mem": "27.09 GB", "iter": "200/2500", "time_diff": 0.114374, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:03:02][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:17:56", "gpu_mem": "27.09 GB", "iter": "250/2500", "time_diff": 0.478575, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:03:19][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:04:04", "gpu_mem": "27.09 GB", "iter": "300/2500", "time_diff": 0.110971, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:03:35][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:04:00", "gpu_mem": "27.09 GB", "iter": "350/2500", "time_diff": 0.112058, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:03:53][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:03:51", "gpu_mem": "27.09 GB", "iter": "400/2500", "time_diff": 0.110413, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:04:09][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:06:50", "gpu_mem": "27.09 GB", "iter": "450/2500", "time_diff": 0.200397, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:04:27][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:03:48", "gpu_mem": "27.09 GB", "iter": "500/2500", "time_diff": 0.114463, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:04:43][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:24:12", "gpu_mem": "27.09 GB", "iter": "550/2500", "time_diff": 0.744617, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:05:01][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:03:39", "gpu_mem": "27.09 GB", "iter": "600/2500", "time_diff": 0.115337, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:05:20][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:07:12", "gpu_mem": "27.09 GB", "iter": "650/2500", "time_diff": 0.233758, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:05:37][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:10:34", "gpu_mem": "27.09 GB", "iter": "700/2500", "time_diff": 0.352775, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:05:54][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:03:16", "gpu_mem": "27.09 GB", "iter": "750/2500", "time_diff": 0.112076, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:06:12][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:03:11", "gpu_mem": "27.09 GB", "iter": "800/2500", "time_diff": 0.112447, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:06:30][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:03:01", "gpu_mem": "27.09 GB", "iter": "850/2500", "time_diff": 0.110010, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:06:48][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:02:59", "gpu_mem": "27.09 GB", "iter": "900/2500", "time_diff": 0.112156, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:07:05][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:02:49", "gpu_mem": "27.09 GB", "iter": "950/2500", "time_diff": 0.109158, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:07:23][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:02:46", "gpu_mem": "27.09 GB", "iter": "1000/2500", "time_diff": 0.111228, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:07:40][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:07:55", "gpu_mem": "27.09 GB", "iter": "1050/2500", "time_diff": 0.328250, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:07:57][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:19:59", "gpu_mem": "27.09 GB", "iter": "1100/2500", "time_diff": 0.856698, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:08:19][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:02:37", "gpu_mem": "27.09 GB", "iter": "1150/2500", "time_diff": 0.116929, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:08:39][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:10:45", "gpu_mem": "27.09 GB", "iter": "1200/2500", "time_diff": 0.496627, "top1_err": 9.999999, "top5_err": 0.000000} [09/10 01:08:57][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:11:33", "gpu_mem": "27.09 GB", "iter": "1250/2500", "time_diff": 0.555067, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:09:15][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:02:18", "gpu_mem": "27.09 GB", "iter": "1300/2500", "time_diff": 0.115367, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:09:31][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:08:32", "gpu_mem": "27.09 GB", "iter": "1350/2500", "time_diff": 0.445253, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:09:49][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:02:09", "gpu_mem": "27.09 GB", "iter": "1400/2500", "time_diff": 0.117283, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:10:06][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:15:16", "gpu_mem": "27.09 GB", "iter": "1450/2500", "time_diff": 0.873132, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:10:24][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:01:54", "gpu_mem": "27.09 GB", "iter": "1500/2500", "time_diff": 0.114955, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:10:41][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:01:52", "gpu_mem": "27.09 GB", "iter": "1550/2500", "time_diff": 0.118619, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:10:58][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:01:37", "gpu_mem": "27.09 GB", "iter": "1600/2500", "time_diff": 0.108649, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:11:15][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:01:43", "gpu_mem": "27.09 GB", "iter": "1650/2500", "time_diff": 0.121533, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:11:32][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:01:30", "gpu_mem": "27.09 GB", "iter": "1700/2500", "time_diff": 0.113690, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:11:48][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:01:24", "gpu_mem": "27.09 GB", "iter": "1750/2500", "time_diff": 0.112568, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:12:03][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:01:20", "gpu_mem": "27.09 GB", "iter": "1800/2500", "time_diff": 0.114333, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:12:19][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:01:11", "gpu_mem": "27.09 GB", "iter": "1850/2500", "time_diff": 0.110449, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:12:34][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:01:06", "gpu_mem": "27.09 GB", "iter": "1900/2500", "time_diff": 0.111121, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:12:50][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:01:03", "gpu_mem": "27.09 GB", "iter": "1950/2500", "time_diff": 0.115958, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:13:05][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:00:55", "gpu_mem": "27.09 GB", "iter": "2000/2500", "time_diff": 0.111817, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:13:19][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:00:52", "gpu_mem": "27.09 GB", "iter": "2050/2500", "time_diff": 0.117408, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:13:32][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:00:44", "gpu_mem": "27.09 GB", "iter": "2100/2500", "time_diff": 0.112329, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:13:46][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:00:41", "gpu_mem": "27.09 GB", "iter": "2150/2500", "time_diff": 0.119977, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:14:00][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:00:35", "gpu_mem": "27.09 GB", "iter": "2200/2500", "time_diff": 0.118929, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:14:12][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:00:28", "gpu_mem": "27.09 GB", "iter": "2250/2500", "time_diff": 0.114096, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:14:24][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:00:22", "gpu_mem": "27.09 GB", "iter": "2300/2500", "time_diff": 0.111872, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:14:37][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:00:20", "gpu_mem": "27.09 GB", "iter": "2350/2500", "time_diff": 0.133497, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:14:48][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:00:49", "gpu_mem": "27.09 GB", "iter": "2400/2500", "time_diff": 0.494269, "top1_err": 9.999999, "top5_err": 0.000000} [09/10 01:15:00][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:00:05", "gpu_mem": "27.09 GB", "iter": "2450/2500", "time_diff": 0.116044, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:15:13][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "2/10", "eta": "0:00:00", "gpu_mem": "27.09 GB", "iter": "2500/2500", "time_diff": 0.114413, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:15:14][INFO] utils.logging: 90: {"RAM": "0.01/585.94 GB", "_type": "val_epoch", "epoch": "2/10", "gpu_mem": "27.09 GB", "min_top1_err": 17.079999, "min_top5_err": 0.000000, "time_diff": 0.486240, "top1_err": 17.191999, "top5_err": 0.000000} [09/10 01:15:15][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "2/10", "eta": "22087 days, 21:18:38", "iter": "1200/250000", "loss": 0.570184, "lr": 0.000050, "time_diff": 848.627498, "top1_err": 2.857143, "top5_err": 0.000000} [09/10 01:17:30][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "2/10", "eta": "78 days, 2:49:33", "iter": "1250/250000", "loss": 0.458915, "lr": 0.000050, "time_diff": 3.001389, "top1_err": 1.428571, "top5_err": 0.000000} [09/10 01:19:50][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "2/10", "eta": "31 days, 11:19:19", "iter": "1300/250000", "loss": 0.483480, "lr": 0.000050, "time_diff": 1.209214, "top1_err": 0.000000, "top5_err": 0.000000} [09/10 01:22:14][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "2/10", "eta": "204 days, 22:35:27", "iter": "1350/250000", "loss": 0.408897, "lr": 0.000050, "time_diff": 7.874470, "top1_err": 0.000000, "top5_err": 0.000000} [09/10 01:24:28][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "2/10", "eta": "31 days, 23:56:34", "iter": "1400/250000", "loss": 0.430746, "lr": 0.000050, "time_diff": 1.229474, "top1_err": 2.857143, "top5_err": 0.000000} [09/10 01:26:47][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "3/10", "eta": "148 days, 16:40:46", "iter": "1450/250000", "loss": 0.337011, "lr": 0.000025, "time_diff": 6.428284, "top1_err": 0.000000, "top5_err": 0.000000}

[09/10 01:29:13][INFO] utils.bucket: 60: putting 'output/test/checkpoints/checkpoint_epoch_00003.pyth' to 'model/few-shot/Few_shot_Relation-Attention_final/UCF/v1/checkpoint_epoch_00003.pyth' [09/10 01:29:43][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:04:56", "gpu_mem": "27.09 GB", "iter": "50/2500", "time_diff": 0.121073, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:29:58][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:04:26", "gpu_mem": "27.09 GB", "iter": "100/2500", "time_diff": 0.111027, "top1_err": 9.999999, "top5_err": 0.000000} [09/10 01:30:12][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:04:23", "gpu_mem": "27.09 GB", "iter": "150/2500", "time_diff": 0.112056, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:30:25][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:04:34", "gpu_mem": "27.09 GB", "iter": "200/2500", "time_diff": 0.119195, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:30:40][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:04:16", "gpu_mem": "27.09 GB", "iter": "250/2500", "time_diff": 0.114003, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:30:54][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:26:11", "gpu_mem": "27.09 GB", "iter": "300/2500", "time_diff": 0.714451, "top1_err": 9.999999, "top5_err": 0.000000} [09/10 01:31:08][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:04:25", "gpu_mem": "27.09 GB", "iter": "350/2500", "time_diff": 0.123308, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:31:22][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:27:13", "gpu_mem": "27.09 GB", "iter": "400/2500", "time_diff": 0.777981, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:31:36][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:03:54", "gpu_mem": "27.09 GB", "iter": "450/2500", "time_diff": 0.114472, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:31:50][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:04:08", "gpu_mem": "27.09 GB", "iter": "500/2500", "time_diff": 0.124076, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:32:05][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:03:45", "gpu_mem": "27.09 GB", "iter": "550/2500", "time_diff": 0.115872, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:32:19][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:03:45", "gpu_mem": "27.09 GB", "iter": "600/2500", "time_diff": 0.118931, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:32:33][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:03:46", "gpu_mem": "27.09 GB", "iter": "650/2500", "time_diff": 0.122497, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:32:46][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:04:31", "gpu_mem": "27.09 GB", "iter": "700/2500", "time_diff": 0.150743, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:33:02][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:03:20", "gpu_mem": "27.09 GB", "iter": "750/2500", "time_diff": 0.114810, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:33:16][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:03:18", "gpu_mem": "27.09 GB", "iter": "800/2500", "time_diff": 0.117044, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:33:29][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:05:35", "gpu_mem": "27.09 GB", "iter": "850/2500", "time_diff": 0.203155, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:33:42][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:03:02", "gpu_mem": "27.09 GB", "iter": "900/2500", "time_diff": 0.113873, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:33:56][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:06:04", "gpu_mem": "27.09 GB", "iter": "950/2500", "time_diff": 0.235155, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:34:09][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:02:54", "gpu_mem": "27.09 GB", "iter": "1000/2500", "time_diff": 0.116651, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:34:22][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:12:17", "gpu_mem": "27.09 GB", "iter": "1050/2500", "time_diff": 0.508830, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:34:35][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:02:34", "gpu_mem": "27.09 GB", "iter": "1100/2500", "time_diff": 0.110240, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:34:49][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:02:32", "gpu_mem": "27.09 GB", "iter": "1150/2500", "time_diff": 0.112783, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:35:01][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:02:28", "gpu_mem": "27.09 GB", "iter": "1200/2500", "time_diff": 0.114152, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:35:15][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:13:56", "gpu_mem": "27.09 GB", "iter": "1250/2500", "time_diff": 0.669303, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:35:29][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:02:22", "gpu_mem": "27.09 GB", "iter": "1300/2500", "time_diff": 0.118751, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:35:43][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:13:31", "gpu_mem": "27.09 GB", "iter": "1350/2500", "time_diff": 0.706051, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:35:56][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:02:03", "gpu_mem": "27.09 GB", "iter": "1400/2500", "time_diff": 0.112357, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:36:10][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:02:15", "gpu_mem": "27.09 GB", "iter": "1450/2500", "time_diff": 0.129387, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:36:23][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:01:56", "gpu_mem": "27.09 GB", "iter": "1500/2500", "time_diff": 0.116790, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:36:36][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:06:32", "gpu_mem": "27.09 GB", "iter": "1550/2500", "time_diff": 0.412707, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:36:49][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:01:43", "gpu_mem": "27.09 GB", "iter": "1600/2500", "time_diff": 0.114524, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:37:03][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:01:39", "gpu_mem": "27.09 GB", "iter": "1650/2500", "time_diff": 0.117163, "top1_err": 0.000000, "top5_err": 0.000000} [09/10 01:37:14][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:01:31", "gpu_mem": "27.09 GB", "iter": "1700/2500", "time_diff": 0.113934, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:37:27][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:01:32", "gpu_mem": "27.09 GB", "iter": "1750/2500", "time_diff": 0.122820, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:37:40][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:08:07", "gpu_mem": "27.09 GB", "iter": "1800/2500", "time_diff": 0.696803, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:37:53][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:01:19", "gpu_mem": "27.09 GB", "iter": "1850/2500", "time_diff": 0.122196, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:38:06][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:06:21", "gpu_mem": "27.09 GB", "iter": "1900/2500", "time_diff": 0.635173, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:38:18][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:01:03", "gpu_mem": "27.09 GB", "iter": "1950/2500", "time_diff": 0.114748, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:38:31][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:04:42", "gpu_mem": "27.09 GB", "iter": "2000/2500", "time_diff": 0.564037, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:38:43][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:00:55", "gpu_mem": "27.09 GB", "iter": "2050/2500", "time_diff": 0.123376, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:38:56][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:04:10", "gpu_mem": "27.09 GB", "iter": "2100/2500", "time_diff": 0.626621, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:39:09][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:00:40", "gpu_mem": "27.09 GB", "iter": "2150/2500", "time_diff": 0.115702, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:39:23][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:00:35", "gpu_mem": "27.09 GB", "iter": "2200/2500", "time_diff": 0.118233, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:39:37][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:02:37", "gpu_mem": "27.09 GB", "iter": "2250/2500", "time_diff": 0.628721, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:39:52][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:00:25", "gpu_mem": "27.09 GB", "iter": "2300/2500", "time_diff": 0.127210, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:40:05][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:00:17", "gpu_mem": "27.09 GB", "iter": "2350/2500", "time_diff": 0.117312, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:40:19][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:00:32", "gpu_mem": "27.09 GB", "iter": "2400/2500", "time_diff": 0.328052, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:40:32][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:00:06", "gpu_mem": "27.09 GB", "iter": "2450/2500", "time_diff": 0.121894, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:40:44][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:00:00", "gpu_mem": "27.09 GB", "iter": "2500/2500", "time_diff": 0.336961, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:40:45][INFO] utils.logging: 90: {"RAM": "0.01/585.94 GB", "_type": "val_epoch", "epoch": "3/10", "gpu_mem": "27.09 GB", "min_top1_err": 17.079999, "min_top5_err": 0.000000, "time_diff": 0.473863, "top1_err": 17.335999, "top5_err": 0.000000} [09/10 01:40:46][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "3/10", "eta": "16376 days, 18:01:21", "iter": "1500/250000", "loss": 0.321338, "lr": 0.000025, "time_diff": 708.006646, "top1_err": 0.000000, "top5_err": 0.000000} [09/10 01:42:50][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "3/10", "eta": "55 days, 7:36:12", "iter": "1550/250000", "loss": 0.295718, "lr": 0.000025, "time_diff": 2.391540, "top1_err": 0.000000, "top5_err": 0.000000} [09/10 01:45:07][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "3/10", "eta": "28 days, 7:20:18", "iter": "1600/250000", "loss": 0.290204, "lr": 0.000025, "time_diff": 1.223788, "top1_err": 0.000000, "top5_err": 0.000000} [09/10 01:47:24][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "3/10", "eta": "91 days, 4:59:27", "iter": "1650/250000", "loss": 0.288918, "lr": 0.000025, "time_diff": 3.943437, "top1_err": 0.000000, "top5_err": 0.000000} [09/10 01:49:38][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "3/10", "eta": "28 days, 10:32:40", "iter": "1700/250000", "loss": 0.229426, "lr": 0.000025, "time_diff": 1.229626, "top1_err": 0.000000, "top5_err": 0.000000} [09/10 01:51:53][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "3/10", "eta": "161 days, 0:22:26", "iter": "1750/250000", "loss": 0.248469, "lr": 0.000025, "time_diff": 6.961965, "top1_err": 0.000000, "top5_err": 0.000000}

[09/10 01:54:15][INFO] utils.bucket: 60: putting 'output/test/checkpoints/checkpoint_epoch_00003.pyth' to 'model/few-shot/Few_shot_Relation-Attention_final/UCF/v1/checkpoint_epoch_00003.pyth' [09/10 01:54:43][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:05:42", "gpu_mem": "27.09 GB", "iter": "50/2500", "time_diff": 0.139627, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:54:56][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:04:25", "gpu_mem": "27.09 GB", "iter": "100/2500", "time_diff": 0.110527, "top1_err": 0.000000, "top5_err": 0.000000} [09/10 01:55:08][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:22:19", "gpu_mem": "27.09 GB", "iter": "150/2500", "time_diff": 0.569960, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:55:20][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:04:25", "gpu_mem": "27.09 GB", "iter": "200/2500", "time_diff": 0.115248, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:55:35][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:04:23", "gpu_mem": "27.09 GB", "iter": "250/2500", "time_diff": 0.117132, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:55:46][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:04:22", "gpu_mem": "27.09 GB", "iter": "300/2500", "time_diff": 0.119221, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:55:59][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:04:09", "gpu_mem": "27.09 GB", "iter": "350/2500", "time_diff": 0.116041, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:56:11][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:04:05", "gpu_mem": "27.09 GB", "iter": "400/2500", "time_diff": 0.116704, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:56:23][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:04:08", "gpu_mem": "27.09 GB", "iter": "450/2500", "time_diff": 0.121128, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:56:38][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:03:54", "gpu_mem": "27.09 GB", "iter": "500/2500", "time_diff": 0.117380, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:56:51][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:03:40", "gpu_mem": "27.09 GB", "iter": "550/2500", "time_diff": 0.113289, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:57:03][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:03:38", "gpu_mem": "27.09 GB", "iter": "600/2500", "time_diff": 0.115253, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:57:15][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:03:37", "gpu_mem": "27.09 GB", "iter": "650/2500", "time_diff": 0.117449, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:57:28][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:03:38", "gpu_mem": "27.09 GB", "iter": "700/2500", "time_diff": 0.121166, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:57:41][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:17:09", "gpu_mem": "27.09 GB", "iter": "750/2500", "time_diff": 0.588186, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:57:54][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:03:15", "gpu_mem": "27.09 GB", "iter": "800/2500", "time_diff": 0.115252, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:58:07][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:12:06", "gpu_mem": "27.09 GB", "iter": "850/2500", "time_diff": 0.440081, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:58:19][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:17:34", "gpu_mem": "27.09 GB", "iter": "900/2500", "time_diff": 0.659070, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:58:31][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:03:05", "gpu_mem": "27.09 GB", "iter": "950/2500", "time_diff": 0.119522, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:58:44][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:06:01", "gpu_mem": "27.09 GB", "iter": "1000/2500", "time_diff": 0.240914, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:58:57][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:12:18", "gpu_mem": "27.09 GB", "iter": "1050/2500", "time_diff": 0.509454, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:59:08][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:02:47", "gpu_mem": "27.09 GB", "iter": "1100/2500", "time_diff": 0.119625, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:59:21][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:02:49", "gpu_mem": "27.09 GB", "iter": "1150/2500", "time_diff": 0.125358, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:59:34][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:02:34", "gpu_mem": "27.09 GB", "iter": "1200/2500", "time_diff": 0.118984, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:59:46][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:02:24", "gpu_mem": "27.09 GB", "iter": "1250/2500", "time_diff": 0.115816, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 01:59:59][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:02:17", "gpu_mem": "27.09 GB", "iter": "1300/2500", "time_diff": 0.114755, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:00:11][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:02:26", "gpu_mem": "27.09 GB", "iter": "1350/2500", "time_diff": 0.127731, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:00:24][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:02:03", "gpu_mem": "27.09 GB", "iter": "1400/2500", "time_diff": 0.112529, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:00:37][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:02:10", "gpu_mem": "27.09 GB", "iter": "1450/2500", "time_diff": 0.124272, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:00:49][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:01:53", "gpu_mem": "27.09 GB", "iter": "1500/2500", "time_diff": 0.113928, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:01:03][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:01:57", "gpu_mem": "27.09 GB", "iter": "1550/2500", "time_diff": 0.124065, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:01:15][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:01:48", "gpu_mem": "27.09 GB", "iter": "1600/2500", "time_diff": 0.120286, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:01:27][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:01:41", "gpu_mem": "27.09 GB", "iter": "1650/2500", "time_diff": 0.119377, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:01:40][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:01:37", "gpu_mem": "27.09 GB", "iter": "1700/2500", "time_diff": 0.122152, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:01:55][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:31:30", "gpu_mem": "27.09 GB", "iter": "1750/2500", "time_diff": 2.521110, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:02:06][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:01:18", "gpu_mem": "27.09 GB", "iter": "1800/2500", "time_diff": 0.112772, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:02:18][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:06:09", "gpu_mem": "27.09 GB", "iter": "1850/2500", "time_diff": 0.568618, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:02:29][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:01:06", "gpu_mem": "27.09 GB", "iter": "1900/2500", "time_diff": 0.111378, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:02:41][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:01:04", "gpu_mem": "27.09 GB", "iter": "1950/2500", "time_diff": 0.116811, "top1_err": 0.000000, "top5_err": 0.000000} [09/10 02:02:57][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:04:07", "gpu_mem": "27.09 GB", "iter": "2000/2500", "time_diff": 0.495697, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:03:09][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:00:51", "gpu_mem": "27.09 GB", "iter": "2050/2500", "time_diff": 0.115317, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:03:22][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:03:54", "gpu_mem": "27.09 GB", "iter": "2100/2500", "time_diff": 0.585984, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:03:34][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:00:41", "gpu_mem": "27.09 GB", "iter": "2150/2500", "time_diff": 0.117808, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:03:46][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:03:24", "gpu_mem": "27.09 GB", "iter": "2200/2500", "time_diff": 0.682687, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:03:59][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:00:29", "gpu_mem": "27.09 GB", "iter": "2250/2500", "time_diff": 0.117860, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:04:11][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:00:23", "gpu_mem": "27.09 GB", "iter": "2300/2500", "time_diff": 0.118900, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:04:25][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:00:17", "gpu_mem": "27.09 GB", "iter": "2350/2500", "time_diff": 0.117943, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:04:36][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:00:11", "gpu_mem": "27.09 GB", "iter": "2400/2500", "time_diff": 0.112773, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:04:49][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:00:05", "gpu_mem": "27.09 GB", "iter": "2450/2500", "time_diff": 0.118096, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:05:01][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:00:00", "gpu_mem": "27.09 GB", "iter": "2500/2500", "time_diff": 0.111181, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:05:02][INFO] utils.logging: 90: {"RAM": "0.01/585.94 GB", "_type": "val_epoch", "epoch": "3/10", "gpu_mem": "27.09 GB", "min_top1_err": 17.079999, "min_top5_err": 0.000000, "time_diff": 0.515658, "top1_err": 17.855999, "top5_err": 0.000000} [09/10 02:05:03][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "3/10", "eta": "15332 days, 3:43:42", "iter": "1800/250000", "loss": 0.262196, "lr": 0.000025, "time_diff": 662.945762, "top1_err": 0.000000, "top5_err": 0.000000} [09/10 02:07:11][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "3/10", "eta": "28 days, 4:22:29", "iter": "1850/250000", "loss": 0.253639, "lr": 0.000025, "time_diff": 1.218602, "top1_err": 0.000000, "top5_err": 0.000000} [09/10 02:09:25][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "3/10", "eta": "28 days, 7:13:27", "iter": "1900/250000", "loss": 0.250613, "lr": 0.000025, "time_diff": 1.223766, "top1_err": 0.000000, "top5_err": 0.000000} [09/10 02:11:43][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "3/10", "eta": "50 days, 19:40:53", "iter": "1950/250000", "loss": 0.206233, "lr": 0.000025, "time_diff": 2.197569, "top1_err": 0.000000, "top5_err": 0.000000} [09/10 02:13:59][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "3/10", "eta": "30 days, 1:18:31", "iter": "2000/250000", "loss": 0.251375, "lr": 0.000025, "time_diff": 1.299655, "top1_err": 0.000000, "top5_err": 0.000000} [09/10 02:16:21][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "3/10", "eta": "33 days, 20:31:00", "iter": "2050/250000", "loss": 0.228880, "lr": 0.000025, "time_diff": 1.464031, "top1_err": 0.000000, "top5_err": 0.000000}

[09/10 02:18:48][INFO] utils.bucket: 60: putting 'output/test/checkpoints/checkpoint_epoch_00003.pyth' to 'model/few-shot/Few_shot_Relation-Attention_final/UCF/v1/checkpoint_epoch_00003.pyth' [09/10 02:19:15][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:22:33", "gpu_mem": "27.09 GB", "iter": "50/2500", "time_diff": 0.552333, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:19:28][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:04:35", "gpu_mem": "27.09 GB", "iter": "100/2500", "time_diff": 0.114988, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:19:40][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:24:47", "gpu_mem": "27.09 GB", "iter": "150/2500", "time_diff": 0.632791, "top1_err": 0.000000, "top5_err": 0.000000} [09/10 02:19:53][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:16:20", "gpu_mem": "27.09 GB", "iter": "200/2500", "time_diff": 0.426371, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:20:05][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:04:14", "gpu_mem": "27.09 GB", "iter": "250/2500", "time_diff": 0.113080, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:20:18][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:22:14", "gpu_mem": "27.09 GB", "iter": "300/2500", "time_diff": 0.606496, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:20:29][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:04:21", "gpu_mem": "27.09 GB", "iter": "350/2500", "time_diff": 0.121768, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:20:43][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:03:59", "gpu_mem": "27.09 GB", "iter": "400/2500", "time_diff": 0.113970, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:20:55][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:04:02", "gpu_mem": "27.09 GB", "iter": "450/2500", "time_diff": 0.118200, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:21:07][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:03:55", "gpu_mem": "27.09 GB", "iter": "500/2500", "time_diff": 0.117988, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:21:19][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:03:42", "gpu_mem": "27.09 GB", "iter": "550/2500", "time_diff": 0.114327, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:21:33][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:03:28", "gpu_mem": "27.09 GB", "iter": "600/2500", "time_diff": 0.109904, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:21:50][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:03:35", "gpu_mem": "27.09 GB", "iter": "650/2500", "time_diff": 0.116621, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:22:02][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:03:30", "gpu_mem": "27.09 GB", "iter": "700/2500", "time_diff": 0.117113, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:22:14][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:03:31", "gpu_mem": "27.09 GB", "iter": "750/2500", "time_diff": 0.120736, "top1_err": 9.999999, "top5_err": 0.000000} [09/10 02:22:26][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:03:24", "gpu_mem": "27.09 GB", "iter": "800/2500", "time_diff": 0.120167, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:22:38][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:03:13", "gpu_mem": "27.09 GB", "iter": "850/2500", "time_diff": 0.117252, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:22:50][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:03:18", "gpu_mem": "27.09 GB", "iter": "900/2500", "time_diff": 0.123992, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:23:03][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:19:09", "gpu_mem": "27.09 GB", "iter": "950/2500", "time_diff": 0.741713, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:23:16][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:02:49", "gpu_mem": "27.09 GB", "iter": "1000/2500", "time_diff": 0.113100, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:23:28][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:02:56", "gpu_mem": "27.09 GB", "iter": "1050/2500", "time_diff": 0.121500, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:23:41][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:02:46", "gpu_mem": "27.09 GB", "iter": "1100/2500", "time_diff": 0.118778, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:23:53][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:02:44", "gpu_mem": "27.09 GB", "iter": "1150/2500", "time_diff": 0.122099, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:24:06][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:02:34", "gpu_mem": "27.09 GB", "iter": "1200/2500", "time_diff": 0.118869, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:24:18][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:02:24", "gpu_mem": "27.09 GB", "iter": "1250/2500", "time_diff": 0.115399, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:24:31][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:02:40", "gpu_mem": "27.09 GB", "iter": "1300/2500", "time_diff": 0.133426, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:24:43][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:02:10", "gpu_mem": "27.09 GB", "iter": "1350/2500", "time_diff": 0.113151, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:24:57][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:02:18", "gpu_mem": "27.09 GB", "iter": "1400/2500", "time_diff": 0.126079, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:25:09][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:02:03", "gpu_mem": "27.09 GB", "iter": "1450/2500", "time_diff": 0.118026, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:25:22][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:01:54", "gpu_mem": "27.09 GB", "iter": "1500/2500", "time_diff": 0.114965, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:25:34][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:01:47", "gpu_mem": "27.09 GB", "iter": "1550/2500", "time_diff": 0.112948, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:25:47][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:01:42", "gpu_mem": "27.09 GB", "iter": "1600/2500", "time_diff": 0.114110, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:25:59][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:01:41", "gpu_mem": "27.09 GB", "iter": "1650/2500", "time_diff": 0.118951, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:26:12][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:08:47", "gpu_mem": "27.09 GB", "iter": "1700/2500", "time_diff": 0.658802, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:26:24][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:01:28", "gpu_mem": "27.09 GB", "iter": "1750/2500", "time_diff": 0.117777, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:26:36][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:01:19", "gpu_mem": "27.09 GB", "iter": "1800/2500", "time_diff": 0.114281, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:26:49][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:01:17", "gpu_mem": "27.09 GB", "iter": "1850/2500", "time_diff": 0.118519, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:27:01][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:05:22", "gpu_mem": "27.09 GB", "iter": "1900/2500", "time_diff": 0.536714, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:27:13][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:01:06", "gpu_mem": "27.09 GB", "iter": "1950/2500", "time_diff": 0.120406, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:27:25][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:01:01", "gpu_mem": "27.09 GB", "iter": "2000/2500", "time_diff": 0.122906, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:27:38][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:00:51", "gpu_mem": "27.09 GB", "iter": "2050/2500", "time_diff": 0.114100, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:27:49][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:00:44", "gpu_mem": "27.09 GB", "iter": "2100/2500", "time_diff": 0.111658, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:28:01][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:00:44", "gpu_mem": "27.09 GB", "iter": "2150/2500", "time_diff": 0.125961, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:28:13][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:00:35", "gpu_mem": "27.09 GB", "iter": "2200/2500", "time_diff": 0.118782, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:28:26][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:02:10", "gpu_mem": "27.09 GB", "iter": "2250/2500", "time_diff": 0.521870, "top1_err": 0.000000, "top5_err": 0.000000} [09/10 02:28:38][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:00:23", "gpu_mem": "27.09 GB", "iter": "2300/2500", "time_diff": 0.119442, "top1_err": 0.000000, "top5_err": 0.000000} [09/10 02:28:51][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:01:54", "gpu_mem": "27.09 GB", "iter": "2350/2500", "time_diff": 0.762279, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:29:02][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:00:11", "gpu_mem": "27.09 GB", "iter": "2400/2500", "time_diff": 0.116350, "top1_err": 9.999999, "top5_err": 0.000000} [09/10 02:29:15][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:00:06", "gpu_mem": "27.09 GB", "iter": "2450/2500", "time_diff": 0.120635, "top1_err": 19.999998, "top5_err": 0.000000} [09/10 02:29:27][INFO] utils.logging: 90: {"_type": "val_iter", "epoch": "3/10", "eta": "0:00:00", "gpu_mem": "27.09 GB", "iter": "2500/2500", "time_diff": 0.387555, "top1_err": 0.000000, "top5_err": 0.000000} [09/10 02:29:27][INFO] utils.logging: 90: {"RAM": "0.01/585.94 GB", "_type": "val_epoch", "epoch": "3/10", "gpu_mem": "27.09 GB", "min_top1_err": 16.119999, "min_top5_err": 0.000000, "time_diff": 0.473712, "top1_err": 16.119999, "top5_err": 0.000000} [09/10 02:29:29][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "3/10", "eta": "15167 days, 4:34:41", "iter": "2100/250000", "loss": 0.184344, "lr": 0.000025, "time_diff": 655.911348, "top1_err": 0.000000, "top5_err": 0.000000} [09/10 02:31:29][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "4/10", "eta": "105 days, 8:20:54", "iter": "2150/250000", "loss": 0.184435, "lr": 0.000005, "time_diff": 5.207572, "top1_err": 0.000000, "top5_err": 0.000000} [09/10 02:33:37][INFO] utils.logging: 90: {"_type": "train_iter", "epoch": "4/10", "eta": "25 days, 2:19:36", "iter": "2200/250000", "loss": 0.209148, "lr": 0.000005, "time_diff": 1.240632, "top1_err": 0.000000, "top5_err": 0.000000} [09/10 02:33:46][INFO] utils.logging: 90: {"RAM": "0.01/585.94 GB", "_type": "train_epoch", "epoch": "4/10", "eta": "170 days, 15:28:13", "gpu_mem": "27.09 GB", "loss": 1.288251, "lr": 0.000005, "time_diff": 9.829164, "top1_err": 6.754545, "top5_err": 0.000000}

wangxiang1230 commented 1 year ago

I downloaded the pretrained Resnet-50 and loaded it before training. I tried 1-shot UCF and I set the query_num = 1 due to limited CUDA memory, the results: (val_epoch "min_top1_err": 33.02 , "top1_err": 66.11). I found that when I set the query_num = 3, the results becomes (val epoch "min_top1_err": 26.56, "top1_err": 49.84). Does the number of query make such a big difference？ Besides, I read the log and found that the top1_err is best at first, then becomes worse during training. This is shown in the results above (26.56% at first and 49.84% at final). Also, the NUM_TRAIN_TASKS is only 2500 and the lr is still warming up at the end of training. Are these phenomena normal? or maybe you could share your training logs with me?

Hi, yczhan, referring to this issue, when we rewrote the open source version, there was a small bug, that is, the learning rate was adjusted in a wrong way, which has been corrected now (here).