brjathu / iTAML

Official implementation of "iTAML : An Incremental Task-Agnostic Meta-learning Approach". CVPR 2020
96 stars 16 forks source link

train_mnist loss go to nan at Sess 3 #7

Closed ghost closed 4 years ago

ghost commented 4 years ago

command : python train_mnist 0 (start sess 0)

In my case, training goes well until Sess 2, Sess 2 best accuracy was 81.06450013224115

And here is my logs for Sess 3

[2, 2, 2, 2, 2] {'min_class': 6, 'max_class': 8, 'task': 3, 'max_task': 5, 'n_train_data': 14181, 'n_test_data': 8017} {0: 2115, 1: 2042, 2: 1874, 3: 1986}

Epoch: [1 | 20] LR: 0.100000 Sess: 3 Processing |################################| (56/56) | Total: 0:00:03 | Loss: 0.0897 | top1: 52.8383 | top5: 52.8383 Processing |################################| (32/32) Total: 0:00:08 | Loss: 1.9010 | top1: 60.0225 | top1_task: 60.0225 50.638297872340424 46.86581782566112 83.51120597652081 61.37965760322256 {1: 993, 0: 78, 2: 492, 3: 465, 5: 777, 4: 788, 6: 298, 7: 921}

Epoch: [2 | 20] LR: 0.100000 Sess: 3 Processing |################################| (56/56) | Total: 0:00:03 | Loss: 0.0334 | top1: 57.3866 | top5: 57.3866 Processing |################################| (32/32) Total: 0:00:08 | Loss: 1.7808 | top1: 72.2714 | top1_task: 72.2714 74.18439716312056 67.58080313418218 76.46744930629669 71.09768378650554 {1: 1001, 0: 568, 3: 947, 2: 433, 4: 934, 5: 499, 6: 509, 7: 903}

Epoch: [3 | 20] LR: 0.100000 Sess: 3 Processing |################################| (56/56) | Total: 0:00:03 | Loss: 0.0134 | top1: 57.7886 | top5: 57.7886 Processing |################################| (32/32) Total: 0:00:08 | Loss: 2.3776 | top1: 50.8544 | top1_task: 50.8544 47.84869976359338 63.07541625857003 40.821771611526145 50.95669687814703 {1: 977, 0: 35, 3: 993, 2: 295, 4: 765, 7: 626, 6: 386}

Epoch: [4 | 20] LR: 0.100000 Sess: 3 Processing |################################| (56/56) | Total: 0:00:03 | Loss: nan | top1: 37.5855 | top5: 37.5855 Processing |###### | (6/32) Total: 0:00:02 | Loss: nan | top1: 45.8333 | top1_task: 45.8333^CTraceback (most recent call last):