search

aigzhusmart / Slim-UNETR

An Efficient, High-Quality 3D Segmentation for Medical Image Analysis with Constrained Computational Resources
MIT License
33 stars 4 forks source link

model collapse on BTCV #1

Open FengheTan9 opened 9 months ago

FengheTan9 commented 9 months ago

reference on MONAI/BTCV get NAN loss.

Network HyperParamter: model = SlimUNETR(in_channels=args.in_channels, out_channels=args.out_channels, embed_dim=96, embedding_dim=27, channels=(24, 48, 60), blocks=(1, 2, 3, 2), heads=(1, 2, 4, 4), r=(4, 2, 2, 1), dropout=0.3 )

training log: dataset_1.json Total parameters count 1792194 Writing Tensorboard logs to ./runs/slimunetr_fold1 0 Mon Dec 18 12:58:28 2023 Epoch: 0 Epoch 0/5000 23/24 loss: 3.9039 time 6.56s Final training 0/4999 loss: 3.9039 time 6.57s 0 Mon Dec 18 12:58:35 2023 Epoch: 1 Epoch 1/5000 23/24 loss: 3.8993 time 3.74s Final training 1/4999 loss: 3.8993 time 3.74s 0 Mon Dec 18 12:58:39 2023 Epoch: 2 Epoch 2/5000 23/24 loss: 3.8803 time 4.28s Final training 2/4999 loss: 3.8803 time 4.29s 0 Mon Dec 18 12:58:43 2023 Epoch: 3 Epoch 3/5000 23/24 loss: 3.8489 time 4.08s Final training 3/4999 loss: 3.8489 time 4.09s 0 Mon Dec 18 12:58:47 2023 Epoch: 4 Epoch 4/5000 23/24 loss: 3.7970 time 4.09s Final training 4/4999 loss: 3.7970 time 4.09s 0 Mon Dec 18 12:58:51 2023 Epoch: 5 Epoch 5/5000 23/24 loss: 3.7307 time 3.83s Final training 5/4999 loss: 3.7307 time 3.83s 0 Mon Dec 18 12:58:55 2023 Epoch: 6 Epoch 6/5000 23/24 loss: 3.6377 time 3.84s Final training 6/4999 loss: 3.6377 time 3.85s 0 Mon Dec 18 12:58:59 2023 Epoch: 7 Epoch 7/5000 23/24 loss: 3.5136 time 3.87s Final training 7/4999 loss: 3.5136 time 3.87s 0 Mon Dec 18 12:59:03 2023 Epoch: 8 Epoch 8/5000 23/24 loss: 3.3679 time 4.16s Final training 8/4999 loss: 3.3679 time 4.17s 0 Mon Dec 18 12:59:07 2023 Epoch: 9 Epoch 9/5000 23/24 loss: 3.1723 time 3.95s Final training 9/4999 loss: 3.1723 time 3.96s 0 Mon Dec 18 12:59:11 2023 Epoch: 10 Epoch 10/5000 23/24 loss: 2.9591 time 3.88s Final training 10/4999 loss: 2.9591 time 3.89s 0 Mon Dec 18 12:59:15 2023 Epoch: 11 Epoch 11/5000 23/24 loss: 2.7003 time 3.76s Final training 11/4999 loss: 2.7003 time 3.77s 0 Mon Dec 18 12:59:18 2023 Epoch: 12 Epoch 12/5000 23/24 loss: 2.4783 time 3.98s Final training 12/4999 loss: 2.4783 time 3.98s 0 Mon Dec 18 12:59:22 2023 Epoch: 13 Epoch 13/5000 23/24 loss: 2.2511 time 4.08s Final training 13/4999 loss: 2.2511 time 4.08s 0 Mon Dec 18 12:59:26 2023 Epoch: 14 Epoch 14/5000 23/24 loss: 2.0465 time 4.27s Final training 14/4999 loss: 2.0465 time 4.27s 0 Mon Dec 18 12:59:31 2023 Epoch: 15 Epoch 15/5000 23/24 loss: 1.9442 time 4.01s Final training 15/4999 loss: 1.9442 time 4.02s 0 Mon Dec 18 12:59:35 2023 Epoch: 16 Epoch 16/5000 23/24 loss: 1.9374 time 3.87s Final training 16/4999 loss: 1.9374 time 3.87s 0 Mon Dec 18 12:59:39 2023 Epoch: 17 Epoch 17/5000 23/24 loss: 1.8336 time 3.86s Final training 17/4999 loss: 1.8336 time 3.86s 0 Mon Dec 18 12:59:42 2023 Epoch: 18 Epoch 18/5000 23/24 loss: 1.8595 time 4.03s Final training 18/4999 loss: 1.8595 time 4.04s 0 Mon Dec 18 12:59:46 2023 Epoch: 19 Epoch 19/5000 23/24 loss: 1.7936 time 4.19s Final training 19/4999 loss: 1.7936 time 4.19s 0 Mon Dec 18 12:59:51 2023 Epoch: 20 Epoch 20/5000 23/24 loss: 1.8144 time 3.92s Final training 20/4999 loss: 1.8144 time 3.93s 0 Mon Dec 18 12:59:55 2023 Epoch: 21 Epoch 21/5000 23/24 loss: 1.7370 time 3.92s Final training 21/4999 loss: 1.7370 time 3.92s 0 Mon Dec 18 12:59:59 2023 Epoch: 22 Epoch 22/5000 23/24 loss: 1.6169 time 4.21s Final training 22/4999 loss: 1.6169 time 4.22s 0 Mon Dec 18 13:00:03 2023 Epoch: 23 Epoch 23/5000 23/24 loss: 1.7327 time 3.73s Final training 23/4999 loss: 1.7327 time 3.73s 0 Mon Dec 18 13:00:06 2023 Epoch: 24 Epoch 24/5000 23/24 loss: 1.7386 time 4.10s Final training 24/4999 loss: 1.7386 time 4.10s 0 Mon Dec 18 13:00:11 2023 Epoch: 25 Epoch 25/5000 23/24 loss: 1.7416 time 3.89s Final training 25/4999 loss: 1.7416 time 3.89s 0 Mon Dec 18 13:00:14 2023 Epoch: 26 Epoch 26/5000 23/24 loss: 1.8222 time 4.02s Final training 26/4999 loss: 1.8222 time 4.02s 0 Mon Dec 18 13:00:18 2023 Epoch: 27 Epoch 27/5000 23/24 loss: 1.6779 time 3.77s Final training 27/4999 loss: 1.6779 time 3.78s 0 Mon Dec 18 13:00:22 2023 Epoch: 28 Epoch 28/5000 23/24 loss: 1.6813 time 3.87s Final training 28/4999 loss: 1.6813 time 3.87s 0 Mon Dec 18 13:00:26 2023 Epoch: 29 Epoch 29/5000 23/24 loss: 1.6859 time 4.02s Final training 29/4999 loss: 1.6859 time 4.02s 0 Mon Dec 18 13:00:30 2023 Epoch: 30 Epoch 30/5000 23/24 loss: 1.6389 time 3.99s Final training 30/4999 loss: 1.6389 time 3.99s 0 Mon Dec 18 13:00:34 2023 Epoch: 31 Epoch 31/5000 23/24 loss: 1.6203 time 3.90s Final training 31/4999 loss: 1.6203 time 3.91s 0 Mon Dec 18 13:00:38 2023 Epoch: 32 Epoch 32/5000 23/24 loss: 1.6520 time 3.75s Final training 32/4999 loss: 1.6520 time 3.76s 0 Mon Dec 18 13:00:42 2023 Epoch: 33 Epoch 33/5000 23/24 loss: 1.5805 time 3.82s Final training 33/4999 loss: 1.5805 time 3.83s 0 Mon Dec 18 13:00:46 2023 Epoch: 34 Epoch 34/5000 23/24 loss: 1.5516 time 4.22s Final training 34/4999 loss: 1.5516 time 4.23s 0 Mon Dec 18 13:00:50 2023 Epoch: 35 Epoch 35/5000 23/24 loss: 1.5662 time 4.31s Final training 35/4999 loss: 1.5662 time 4.32s 0 Mon Dec 18 13:00:54 2023 Epoch: 36 Epoch 36/5000 23/24 loss: 1.5042 time 4.28s Final training 36/4999 loss: 1.5042 time 4.29s 0 Mon Dec 18 13:00:58 2023 Epoch: 37 Epoch 37/5000 23/24 loss: 1.5496 time 4.08s Final training 37/4999 loss: 1.5496 time 4.08s 0 Mon Dec 18 13:01:03 2023 Epoch: 38 Epoch 38/5000 23/24 loss: 1.5307 time 4.05s Final training 38/4999 loss: 1.5307 time 4.05s 0 Mon Dec 18 13:01:07 2023 Epoch: 39 Epoch 39/5000 23/24 loss: 1.5068 time 4.06s Final training 39/4999 loss: 1.5068 time 4.06s 0 Mon Dec 18 13:01:11 2023 Epoch: 40 Epoch 40/5000 23/24 loss: 1.5126 time 3.95s Final training 40/4999 loss: 1.5126 time 3.95s 0 Mon Dec 18 13:01:15 2023 Epoch: 41 Epoch 41/5000 23/24 loss: 1.5159 time 3.86s Final training 41/4999 loss: 1.5159 time 3.86s 0 Mon Dec 18 13:01:18 2023 Epoch: 42 Epoch 42/5000 23/24 loss: 1.4761 time 4.00s Final training 42/4999 loss: 1.4761 time 4.01s 0 Mon Dec 18 13:01:23 2023 Epoch: 43 Epoch 43/5000 23/24 loss: 1.4955 time 4.13s Final training 43/4999 loss: 1.4955 time 4.13s 0 Mon Dec 18 13:01:27 2023 Epoch: 44 Epoch 44/5000 23/24 loss: 1.4819 time 4.04s Final training 44/4999 loss: 1.4819 time 4.04s 0 Mon Dec 18 13:01:31 2023 Epoch: 45 Epoch 45/5000 23/24 loss: 1.4713 time 4.08s Final training 45/4999 loss: 1.4713 time 4.09s 0 Mon Dec 18 13:01:35 2023 Epoch: 46 Epoch 46/5000 23/24 loss: 1.4352 time 3.92s Final training 46/4999 loss: 1.4352 time 3.93s 0 Mon Dec 18 13:01:39 2023 Epoch: 47 Epoch 47/5000 23/24 loss: 1.4795 time 3.88s Final training 47/4999 loss: 1.4795 time 3.88s 0 Mon Dec 18 13:01:43 2023 Epoch: 48 Epoch 48/5000 23/24 loss: 1.4605 time 3.89s Final training 48/4999 loss: 1.4605 time 3.90s 0 Mon Dec 18 13:01:46 2023 Epoch: 49 Epoch 49/5000 23/24 loss: 1.4490 time 3.93s Final training 49/4999 loss: 1.4490 time 3.94s Val 49/5000 0/6 acc 0.12112933 time 6.02s Val 49/5000 1/6 acc 0.12366394 time 1.95s Val 49/5000 2/6 acc 0.12092439 time 3.27s Val 49/5000 3/6 acc 0.11538899 time 1.53s Val 49/5000 4/6 acc 0.12162123 time 1.95s Val 49/5000 5/6 acc 0.11878981 time 1.04s Final validation 49/4999 acc 0.120252945 time 15.77s new best (0.000000 --> 0.120253). Saving checkpoint ./runs/slimunetr_fold1/model.pt Saving checkpoint ./runs/slimunetr_fold1/model_final.pt Copying to model.pt new best model!!!! 0 Mon Dec 18 13:02:06 2023 Epoch: 50 Epoch 50/5000 23/24 loss: 1.4712 time 3.70s Final training 50/4999 loss: 1.4712 time 3.71s 0 Mon Dec 18 13:02:10 2023 Epoch: 51 Epoch 51/5000 23/24 loss: 1.4246 time 4.52s Final training 51/4999 loss: 1.4246 time 4.53s 0 Mon Dec 18 13:02:15 2023 Epoch: 52 Epoch 52/5000 23/24 loss: 1.4418 time 4.55s Final training 52/4999 loss: 1.4418 time 4.55s 0 Mon Dec 18 13:02:19 2023 Epoch: 53 Epoch 53/5000 23/24 loss: 1.4542 time 4.39s Final training 53/4999 loss: 1.4542 time 4.40s 0 Mon Dec 18 13:02:24 2023 Epoch: 54 Epoch 54/5000 23/24 loss: 1.4683 time 4.02s Final training 54/4999 loss: 1.4683 time 4.02s 0 Mon Dec 18 13:02:28 2023 Epoch: 55 Epoch 55/5000 23/24 loss: 1.3980 time 3.87s Final training 55/4999 loss: 1.3980 time 3.88s 0 Mon Dec 18 13:02:31 2023 Epoch: 56 Epoch 56/5000 23/24 loss: 1.4433 time 4.15s Final training 56/4999 loss: 1.4433 time 4.15s 0 Mon Dec 18 13:02:36 2023 Epoch: 57 Epoch 57/5000 23/24 loss: 1.4014 time 3.76s Final training 57/4999 loss: 1.4014 time 3.76s 0 Mon Dec 18 13:02:39 2023 Epoch: 58 Epoch 58/5000 23/24 loss: 1.4691 time 3.79s Final training 58/4999 loss: 1.4691 time 3.79s 0 Mon Dec 18 13:02:43 2023 Epoch: 59 Epoch 59/5000 23/24 loss: 1.4424 time 3.71s Final training 59/4999 loss: 1.4424 time 3.72s 0 Mon Dec 18 13:02:47 2023 Epoch: 60 Epoch 60/5000 23/24 loss: 1.4457 time 4.10s Final training 60/4999 loss: 1.4457 time 4.10s 0 Mon Dec 18 13:02:51 2023 Epoch: 61 Epoch 61/5000 23/24 loss: 1.4131 time 3.89s Final training 61/4999 loss: 1.4131 time 3.90s 0 Mon Dec 18 13:02:55 2023 Epoch: 62 Epoch 62/5000 23/24 loss: 1.4413 time 3.76s Final training 62/4999 loss: 1.4413 time 3.77s 0 Mon Dec 18 13:02:59 2023 Epoch: 63 Epoch 63/5000 23/24 loss: 1.4581 time 3.94s Final training 63/4999 loss: 1.4581 time 3.94s 0 Mon Dec 18 13:03:03 2023 Epoch: 64 Epoch 64/5000 23/24 loss: 1.4063 time 4.17s Final training 64/4999 loss: 1.4063 time 4.18s 0 Mon Dec 18 13:03:07 2023 Epoch: 65 Epoch 65/5000 23/24 loss: 1.3442 time 4.02s Final training 65/4999 loss: 1.3442 time 4.03s 0 Mon Dec 18 13:03:11 2023 Epoch: 66 Epoch 66/5000 23/24 loss: 1.4005 time 4.07s Final training 66/4999 loss: 1.4005 time 4.08s 0 Mon Dec 18 13:03:15 2023 Epoch: 67 Epoch 67/5000 23/24 loss: 1.3812 time 4.07s Final training 67/4999 loss: 1.3812 time 4.07s 0 Mon Dec 18 13:03:19 2023 Epoch: 68 Epoch 68/5000 23/24 loss: 1.4296 time 4.00s Final training 68/4999 loss: 1.4296 time 4.00s 0 Mon Dec 18 13:03:23 2023 Epoch: 69 Epoch 69/5000 23/24 loss: 1.4273 time 3.86s Final training 69/4999 loss: 1.4273 time 3.87s 0 Mon Dec 18 13:03:27 2023 Epoch: 70 Epoch 70/5000 23/24 loss: 1.4052 time 4.07s Final training 70/4999 loss: 1.4052 time 4.07s 0 Mon Dec 18 13:03:31 2023 Epoch: 71 Epoch 71/5000 23/24 loss: 1.4038 time 3.88s Final training 71/4999 loss: 1.4038 time 3.88s 0 Mon Dec 18 13:03:35 2023 Epoch: 72 Epoch 72/5000 23/24 loss: 1.3859 time 3.85s Final training 72/4999 loss: 1.3859 time 3.85s 0 Mon Dec 18 13:03:39 2023 Epoch: 73 Epoch 73/5000 23/24 loss: 1.3829 time 4.07s Final training 73/4999 loss: 1.3829 time 4.07s 0 Mon Dec 18 13:03:43 2023 Epoch: 74 Epoch 74/5000 23/24 loss: 1.4215 time 3.98s Final training 74/4999 loss: 1.4215 time 3.99s 0 Mon Dec 18 13:03:47 2023 Epoch: 75 Epoch 75/5000 23/24 loss: 1.3949 time 3.94s Final training 75/4999 loss: 1.3949 time 3.95s 0 Mon Dec 18 13:03:51 2023 Epoch: 76 Epoch 76/5000 23/24 loss: 1.4314 time 4.06s Final training 76/4999 loss: 1.4314 time 4.07s 0 Mon Dec 18 13:03:55 2023 Epoch: 77 Epoch 77/5000 23/24 loss: 1.4113 time 3.95s Final training 77/4999 loss: 1.4113 time 3.95s 0 Mon Dec 18 13:03:59 2023 Epoch: 78 Epoch 78/5000 23/24 loss: 1.3981 time 3.70s Final training 78/4999 loss: 1.3981 time 3.71s 0 Mon Dec 18 13:04:02 2023 Epoch: 79 Epoch 79/5000 23/24 loss: 1.4001 time 3.96s Final training 79/4999 loss: 1.4001 time 3.96s 0 Mon Dec 18 13:04:06 2023 Epoch: 80 Epoch 80/5000 23/24 loss: 1.3874 time 3.88s Final training 80/4999 loss: 1.3874 time 3.88s 0 Mon Dec 18 13:04:10 2023 Epoch: 81 Epoch 81/5000 23/24 loss: 1.3728 time 3.97s Final training 81/4999 loss: 1.3728 time 3.98s 0 Mon Dec 18 13:04:14 2023 Epoch: 82 Epoch 82/5000 23/24 loss: 1.4110 time 3.84s Final training 82/4999 loss: 1.4110 time 3.85s 0 Mon Dec 18 13:04:18 2023 Epoch: 83 Epoch 83/5000 23/24 loss: 1.3782 time 3.93s Final training 83/4999 loss: 1.3782 time 3.93s 0 Mon Dec 18 13:04:22 2023 Epoch: 84 Epoch 84/5000 23/24 loss: 1.3575 time 3.87s Final training 84/4999 loss: 1.3575 time 3.87s 0 Mon Dec 18 13:04:26 2023 Epoch: 85 Epoch 85/5000 23/24 loss: 1.3792 time 4.18s Final training 85/4999 loss: 1.3792 time 4.19s 0 Mon Dec 18 13:04:30 2023 Epoch: 86 Epoch 86/5000 23/24 loss: 1.3703 time 4.16s Final training 86/4999 loss: 1.3703 time 4.16s 0 Mon Dec 18 13:04:34 2023 Epoch: 87 Epoch 87/5000 23/24 loss: 1.3673 time 3.97s Final training 87/4999 loss: 1.3673 time 3.98s 0 Mon Dec 18 13:04:38 2023 Epoch: 88 Epoch 88/5000 23/24 loss: 1.3543 time 4.00s Final training 88/4999 loss: 1.3543 time 4.01s 0 Mon Dec 18 13:04:42 2023 Epoch: 89 Epoch 89/5000 23/24 loss: 1.3559 time 3.94s Final training 89/4999 loss: 1.3559 time 3.94s 0 Mon Dec 18 13:04:46 2023 Epoch: 90 Epoch 90/5000 23/24 loss: 1.3961 time 3.95s Final training 90/4999 loss: 1.3961 time 3.95s 0 Mon Dec 18 13:04:50 2023 Epoch: 91 Epoch 91/5000 23/24 loss: 1.3108 time 4.09s Final training 91/4999 loss: 1.3108 time 4.10s 0 Mon Dec 18 13:04:54 2023 Epoch: 92 Epoch 92/5000 23/24 loss: 1.3829 time 3.91s Final training 92/4999 loss: 1.3829 time 3.92s 0 Mon Dec 18 13:04:58 2023 Epoch: 93 Epoch 93/5000 23/24 loss: 1.3490 time 3.92s Final training 93/4999 loss: 1.3490 time 3.92s 0 Mon Dec 18 13:05:02 2023 Epoch: 94 Epoch 94/5000 23/24 loss: 1.3521 time 3.81s Final training 94/4999 loss: 1.3521 time 3.82s 0 Mon Dec 18 13:05:06 2023 Epoch: 95 Epoch 95/5000 23/24 loss: 1.3804 time 3.77s Final training 95/4999 loss: 1.3804 time 3.77s 0 Mon Dec 18 13:05:10 2023 Epoch: 96 Epoch 96/5000 23/24 loss: 1.3550 time 3.95s Final training 96/4999 loss: 1.3550 time 3.95s 0 Mon Dec 18 13:05:14 2023 Epoch: 97 Epoch 97/5000 23/24 loss: 1.4092 time 4.04s Final training 97/4999 loss: 1.4092 time 4.04s 0 Mon Dec 18 13:05:18 2023 Epoch: 98 Epoch 98/5000 23/24 loss: 1.3943 time 3.93s Final training 98/4999 loss: 1.3943 time 3.94s 0 Mon Dec 18 13:05:21 2023 Epoch: 99 Epoch 99/5000 23/24 loss: 1.3859 time 3.73s Final training 99/4999 loss: 1.3859 time 3.73s Val 99/5000 0/6 acc 0.12706093 time 5.19s Val 99/5000 1/6 acc 0.128118 time 1.87s Val 99/5000 2/6 acc 0.12313781 time 3.11s Val 99/5000 3/6 acc 0.123975776 time 1.43s Val 99/5000 4/6 acc 0.12464811 time 2.03s Val 99/5000 5/6 acc 0.12816174 time 1.13s Final validation 99/4999 acc 0.1258504 time 14.76s new best (0.120253 --> 0.125850). Saving checkpoint ./runs/slimunetr_fold1/model.pt Saving checkpoint ./runs/slimunetr_fold1/model_final.pt Copying to model.pt new best model!!!! 0 Mon Dec 18 13:05:40 2023 Epoch: 100 Epoch 100/5000 23/24 loss: 1.3842 time 3.95s Final training 100/4999 loss: 1.3842 time 3.95s 0 Mon Dec 18 13:05:44 2023 Epoch: 101 Epoch 101/5000 23/24 loss: 1.3467 time 3.61s Final training 101/4999 loss: 1.3467 time 3.62s 0 Mon Dec 18 13:05:48 2023 Epoch: 102 Epoch 102/5000 23/24 loss: 1.3654 time 3.83s Final training 102/4999 loss: 1.3654 time 3.83s 0 Mon Dec 18 13:05:52 2023 Epoch: 103 Epoch 103/5000 23/24 loss: 1.3429 time 4.51s Final training 103/4999 loss: 1.3429 time 4.52s 0 Mon Dec 18 13:05:56 2023 Epoch: 104 Epoch 104/5000 23/24 loss: 1.3753 time 4.40s Final training 104/4999 loss: 1.3753 time 4.40s 0 Mon Dec 18 13:06:00 2023 Epoch: 105 Epoch 105/5000 23/24 loss: 1.2972 time 4.04s Final training 105/4999 loss: 1.2972 time 4.04s 0 Mon Dec 18 13:06:05 2023 Epoch: 106 Epoch 106/5000 23/24 loss: 1.3728 time 3.77s Final training 106/4999 loss: 1.3728 time 3.77s 0 Mon Dec 18 13:06:08 2023 Epoch: 107 Epoch 107/5000 23/24 loss: 1.2986 time 4.03s Final training 107/4999 loss: 1.2986 time 4.03s 0 Mon Dec 18 13:06:12 2023 Epoch: 108 Epoch 108/5000 23/24 loss: 1.3446 time 3.98s Final training 108/4999 loss: 1.3446 time 3.99s 0 Mon Dec 18 13:06:16 2023 Epoch: 109 Epoch 109/5000 23/24 loss: 1.3734 time 3.83s Final training 109/4999 loss: 1.3734 time 3.83s 0 Mon Dec 18 13:06:20 2023 Epoch: 110 Epoch 110/5000 23/24 loss: 1.3460 time 4.05s Final training 110/4999 loss: 1.3460 time 4.05s 0 Mon Dec 18 13:06:24 2023 Epoch: 111 Epoch 111/5000 23/24 loss: 1.3408 time 3.78s Final training 111/4999 loss: 1.3408 time 3.78s 0 Mon Dec 18 13:06:28 2023 Epoch: 112 Epoch 112/5000 23/24 loss: 1.3373 time 4.07s Final training 112/4999 loss: 1.3373 time 4.07s 0 Mon Dec 18 13:06:32 2023 Epoch: 113 Epoch 113/5000 23/24 loss: 1.3388 time 3.89s Final training 113/4999 loss: 1.3388 time 3.89s 0 Mon Dec 18 13:06:36 2023 Epoch: 114 Epoch 114/5000 23/24 loss: 1.3410 time 3.83s Final training 114/4999 loss: 1.3410 time 3.84s 0 Mon Dec 18 13:06:40 2023 Epoch: 115 Epoch 115/5000 23/24 loss: 1.3484 time 3.83s Final training 115/4999 loss: 1.3484 time 3.84s 0 Mon Dec 18 13:06:44 2023 Epoch: 116 Epoch 116/5000 23/24 loss: 1.3916 time 3.99s Final training 116/4999 loss: 1.3916 time 3.99s 0 Mon Dec 18 13:06:48 2023 Epoch: 117 Epoch 117/5000 23/24 loss: 1.3207 time 3.99s Final training 117/4999 loss: 1.3207 time 3.99s 0 Mon Dec 18 13:06:52 2023 Epoch: 118 Epoch 118/5000 23/24 loss: 1.3582 time 4.11s Final training 118/4999 loss: 1.3582 time 4.12s 0 Mon Dec 18 13:06:56 2023 Epoch: 119 Epoch 119/5000 23/24 loss: 1.3134 time 3.94s Final training 119/4999 loss: 1.3134 time 3.95s 0 Mon Dec 18 13:07:00 2023 Epoch: 120 Epoch 120/5000 23/24 loss: 1.4016 time 3.68s Final training 120/4999 loss: 1.4016 time 3.69s 0 Mon Dec 18 13:07:03 2023 Epoch: 121 Epoch 121/5000 23/24 loss: 1.3368 time 3.77s Final training 121/4999 loss: 1.3368 time 3.78s 0 Mon Dec 18 13:07:07 2023 Epoch: 122 Epoch 122/5000 23/24 loss: 1.3176 time 3.91s Final training 122/4999 loss: 1.3176 time 3.92s 0 Mon Dec 18 13:07:11 2023 Epoch: 123 Epoch 123/5000 23/24 loss: 1.3164 time 4.01s Final training 123/4999 loss: 1.3164 time 4.01s 0 Mon Dec 18 13:07:15 2023 Epoch: 124 Epoch 124/5000 23/24 loss: 1.3443 time 3.78s Final training 124/4999 loss: 1.3443 time 3.79s 0 Mon Dec 18 13:07:19 2023 Epoch: 125 Epoch 125/5000 23/24 loss: 1.3390 time 4.02s Final training 125/4999 loss: 1.3390 time 4.03s 0 Mon Dec 18 13:07:23 2023 Epoch: 126 Epoch 126/5000 23/24 loss: 1.3012 time 3.71s Final training 126/4999 loss: 1.3012 time 3.71s 0 Mon Dec 18 13:07:27 2023 Epoch: 127 Epoch 127/5000 23/24 loss: 1.3266 time 3.82s Final training 127/4999 loss: 1.3266 time 3.83s 0 Mon Dec 18 13:07:30 2023 Epoch: 128 Epoch 128/5000 23/24 loss: 1.3516 time 4.27s Final training 128/4999 loss: 1.3516 time 4.28s 0 Mon Dec 18 13:07:35 2023 Epoch: 129 Epoch 129/5000 23/24 loss: 1.3524 time 3.78s Final training 129/4999 loss: 1.3524 time 3.78s 0 Mon Dec 18 13:07:38 2023 Epoch: 130 Epoch 130/5000 23/24 loss: 1.3409 time 3.66s Final training 130/4999 loss: 1.3409 time 3.66s 0 Mon Dec 18 13:07:42 2023 Epoch: 131 Epoch 131/5000 23/24 loss: 1.3556 time 3.93s Final training 131/4999 loss: 1.3556 time 3.93s 0 Mon Dec 18 13:07:46 2023 Epoch: 132 Epoch 132/5000 23/24 loss: 1.3043 time 3.90s Final training 132/4999 loss: 1.3043 time 3.90s 0 Mon Dec 18 13:07:50 2023 Epoch: 133 Epoch 133/5000 23/24 loss: 1.3121 time 3.89s Final training 133/4999 loss: 1.3121 time 3.89s 0 Mon Dec 18 13:07:54 2023 Epoch: 134 Epoch 134/5000 23/24 loss: 1.2981 time 4.02s Final training 134/4999 loss: 1.2981 time 4.03s 0 Mon Dec 18 13:07:58 2023 Epoch: 135 Epoch 135/5000 23/24 loss: 1.2940 time 4.03s Final training 135/4999 loss: 1.2940 time 4.03s 0 Mon Dec 18 13:08:02 2023 Epoch: 136 Epoch 136/5000 23/24 loss: 1.2933 time 4.11s Final training 136/4999 loss: 1.2933 time 4.12s 0 Mon Dec 18 13:08:06 2023 Epoch: 137 Epoch 137/5000 23/24 loss: 1.2849 time 4.23s Final training 137/4999 loss: 1.2849 time 4.24s 0 Mon Dec 18 13:08:10 2023 Epoch: 138 Epoch 138/5000 23/24 loss: 1.3244 time 3.84s Final training 138/4999 loss: 1.3244 time 3.85s 0 Mon Dec 18 13:08:14 2023 Epoch: 139 Epoch 139/5000 23/24 loss: 1.2749 time 3.78s Final training 139/4999 loss: 1.2749 time 3.79s 0 Mon Dec 18 13:08:18 2023 Epoch: 140 Epoch 140/5000 23/24 loss: 1.3335 time 4.08s Final training 140/4999 loss: 1.3335 time 4.08s 0 Mon Dec 18 13:08:22 2023 Epoch: 141 Epoch 141/5000 23/24 loss: 1.3095 time 3.81s Final training 141/4999 loss: 1.3095 time 3.82s 0 Mon Dec 18 13:08:26 2023 Epoch: 142 Epoch 142/5000 23/24 loss: 1.2469 time 4.00s Final training 142/4999 loss: 1.2469 time 4.01s 0 Mon Dec 18 13:08:30 2023 Epoch: 143 Epoch 143/5000 23/24 loss: 1.3011 time 3.93s Final training 143/4999 loss: 1.3011 time 3.94s 0 Mon Dec 18 13:08:34 2023 Epoch: 144 Epoch 144/5000 23/24 loss: 1.2832 time 4.12s Final training 144/4999 loss: 1.2832 time 4.13s 0 Mon Dec 18 13:08:38 2023 Epoch: 145 Epoch 145/5000 23/24 loss: 1.2864 time 3.85s Final training 145/4999 loss: 1.2864 time 3.86s 0 Mon Dec 18 13:08:42 2023 Epoch: 146 Epoch 146/5000 23/24 loss: 1.2976 time 4.02s Final training 146/4999 loss: 1.2976 time 4.03s 0 Mon Dec 18 13:08:46 2023 Epoch: 147 Epoch 147/5000 23/24 loss: 1.2650 time 3.81s Final training 147/4999 loss: 1.2650 time 3.82s 0 Mon Dec 18 13:08:50 2023 Epoch: 148 Epoch 148/5000 23/24 loss: 1.2687 time 3.69s Final training 148/4999 loss: 1.2687 time 3.70s 0 Mon Dec 18 13:08:53 2023 Epoch: 149 Epoch 149/5000 23/24 loss: 1.2906 time 3.72s Final training 149/4999 loss: 1.2906 time 3.72s Val 149/5000 0/6 acc 0.16100001 time 5.40s Val 149/5000 1/6 acc 0.15518428 time 1.80s Val 149/5000 2/6 acc 0.13687502 time 2.72s Val 149/5000 3/6 acc 0.12824659 time 1.46s Val 149/5000 4/6 acc 0.15630746 time 2.05s Val 149/5000 5/6 acc 0.15781267 time 1.11s Final validation 149/4999 acc 0.14923768 time 14.55s new best (0.125850 --> 0.149238). Saving checkpoint ./runs/slimunetr_fold1/model.pt Saving checkpoint ./runs/slimunetr_fold1/model_final.pt Copying to model.pt new best model!!!! 0 Mon Dec 18 13:09:12 2023 Epoch: 150 Epoch 150/5000 23/24 loss: 1.2865 time 3.89s Final training 150/4999 loss: 1.2865 time 3.90s 0 Mon Dec 18 13:09:16 2023 Epoch: 151 Epoch 151/5000 23/24 loss: 1.2941 time 3.87s Final training 151/4999 loss: 1.2941 time 3.87s 0 Mon Dec 18 13:09:20 2023 Epoch: 152 Epoch 152/5000 23/24 loss: 1.2657 time 3.72s Final training 152/4999 loss: 1.2657 time 3.73s 0 Mon Dec 18 13:09:23 2023 Epoch: 153 Epoch 153/5000 23/24 loss: 1.2694 time 3.67s Final training 153/4999 loss: 1.2694 time 3.67s 0 Mon Dec 18 13:09:27 2023 Epoch: 154 Epoch 154/5000 23/24 loss: 1.2840 time 3.69s Final training 154/4999 loss: 1.2840 time 3.70s 0 Mon Dec 18 13:09:31 2023 Epoch: 155 Epoch 155/5000 23/24 loss: 1.2807 time 4.30s Final training 155/4999 loss: 1.2807 time 4.30s 0 Mon Dec 18 13:09:35 2023 Epoch: 156 Epoch 156/5000 23/24 loss: 1.2703 time 4.44s Final training 156/4999 loss: 1.2703 time 4.44s 0 Mon Dec 18 13:09:39 2023 Epoch: 157 Epoch 157/5000 23/24 loss: 1.3015 time 4.43s Final training 157/4999 loss: 1.3015 time 4.44s 0 Mon Dec 18 13:09:44 2023 Epoch: 158 Epoch 158/5000 23/24 loss: 1.2749 time 3.93s Final training 158/4999 loss: 1.2749 time 3.93s 0 Mon Dec 18 13:09:48 2023 Epoch: 159 Epoch 159/5000 23/24 loss: 1.2109 time 3.86s Final training 159/4999 loss: 1.2109 time 3.86s 0 Mon Dec 18 13:09:52 2023 Epoch: 160 Epoch 160/5000 23/24 loss: 1.2966 time 3.90s Final training 160/4999 loss: 1.2966 time 3.90s 0 Mon Dec 18 13:09:55 2023 Epoch: 161 Epoch 161/5000 23/24 loss: 1.2571 time 3.94s Final training 161/4999 loss: 1.2571 time 3.95s 0 Mon Dec 18 13:09:59 2023 Epoch: 162 Epoch 162/5000 23/24 loss: 1.2688 time 4.05s Final training 162/4999 loss: 1.2688 time 4.05s 0 Mon Dec 18 13:10:03 2023 Epoch: 163 Epoch 163/5000 23/24 loss: 1.2729 time 3.95s Final training 163/4999 loss: 1.2729 time 3.95s 0 Mon Dec 18 13:10:07 2023 Epoch: 164 Epoch 164/5000 23/24 loss: 1.3035 time 3.87s Final training 164/4999 loss: 1.3035 time 3.88s 0 Mon Dec 18 13:10:11 2023 Epoch: 165 Epoch 165/5000 23/24 loss: 1.2494 time 3.75s Final training 165/4999 loss: 1.2494 time 3.76s 0 Mon Dec 18 13:10:15 2023 Epoch: 166 Epoch 166/5000 23/24 loss: 1.2634 time 4.08s Final training 166/4999 loss: 1.2634 time 4.09s 0 Mon Dec 18 13:10:19 2023 Epoch: 167 Epoch 167/5000 23/24 loss: 1.2676 time 3.91s Final training 167/4999 loss: 1.2676 time 3.91s 0 Mon Dec 18 13:10:23 2023 Epoch: 168 Epoch 168/5000 23/24 loss: 1.2629 time 3.83s Final training 168/4999 loss: 1.2629 time 3.83s 0 Mon Dec 18 13:10:27 2023 Epoch: 169 Epoch 169/5000 23/24 loss: 1.2411 time 4.04s Final training 169/4999 loss: 1.2411 time 4.04s 0 Mon Dec 18 13:10:31 2023 Epoch: 170 Epoch 170/5000 23/24 loss: 1.2484 time 3.95s Final training 170/4999 loss: 1.2484 time 3.95s 0 Mon Dec 18 13:10:35 2023 Epoch: 171 Epoch 171/5000 23/24 loss: 1.2497 time 4.02s Final training 171/4999 loss: 1.2497 time 4.02s 0 Mon Dec 18 13:10:39 2023 Epoch: 172 Epoch 172/5000 23/24 loss: 1.2878 time 4.03s Final training 172/4999 loss: 1.2878 time 4.03s 0 Mon Dec 18 13:10:43 2023 Epoch: 173 Epoch 173/5000 23/24 loss: 1.2449 time 4.00s Final training 173/4999 loss: 1.2449 time 4.00s 0 Mon Dec 18 13:10:47 2023 Epoch: 174 Epoch 174/5000 23/24 loss: 1.2373 time 3.88s Final training 174/4999 loss: 1.2373 time 3.89s 0 Mon Dec 18 13:10:51 2023 Epoch: 175 Epoch 175/5000 23/24 loss: 1.2329 time 3.76s Final training 175/4999 loss: 1.2329 time 3.76s 0 Mon Dec 18 13:10:55 2023 Epoch: 176 Epoch 176/5000 23/24 loss: 1.2758 time 3.90s Final training 176/4999 loss: 1.2758 time 3.90s 0 Mon Dec 18 13:10:59 2023 Epoch: 177 Epoch 177/5000 23/24 loss: 1.2545 time 4.08s Final training 177/4999 loss: 1.2545 time 4.09s 0 Mon Dec 18 13:11:03 2023 Epoch: 178 Epoch 178/5000 23/24 loss: 1.2532 time 3.97s Final training 178/4999 loss: 1.2532 time 3.98s 0 Mon Dec 18 13:11:07 2023 Epoch: 179 Epoch 179/5000 23/24 loss: 1.2786 time 3.92s Final training 179/4999 loss: 1.2786 time 3.92s 0 Mon Dec 18 13:11:11 2023 Epoch: 180 Epoch 180/5000 23/24 loss: 1.2312 time 3.70s Final training 180/4999 loss: 1.2312 time 3.70s 0 Mon Dec 18 13:11:14 2023 Epoch: 181 Epoch 181/5000 23/24 loss: 1.2192 time 3.89s Final training 181/4999 loss: 1.2192 time 3.89s 0 Mon Dec 18 13:11:18 2023 Epoch: 182 Epoch 182/5000 23/24 loss: 1.2879 time 3.94s Final training 182/4999 loss: 1.2879 time 3.95s 0 Mon Dec 18 13:11:22 2023 Epoch: 183 Epoch 183/5000 23/24 loss: 1.2713 time 3.99s Final training 183/4999 loss: 1.2713 time 4.00s 0 Mon Dec 18 13:11:26 2023 Epoch: 184 Epoch 184/5000 23/24 loss: 1.2483 time 3.74s Final training 184/4999 loss: 1.2483 time 3.75s 0 Mon Dec 18 13:11:30 2023 Epoch: 185 Epoch 185/5000 23/24 loss: 1.2427 time 3.74s Final training 185/4999 loss: 1.2427 time 3.74s 0 Mon Dec 18 13:11:34 2023 Epoch: 186 Epoch 186/5000 23/24 loss: 1.2640 time 3.66s Final training 186/4999 loss: 1.2640 time 3.66s 0 Mon Dec 18 13:11:37 2023 Epoch: 187 Epoch 187/5000 23/24 loss: 1.2474 time 3.87s Final training 187/4999 loss: 1.2474 time 3.87s 0 Mon Dec 18 13:11:41 2023 Epoch: 188 Epoch 188/5000 23/24 loss: 1.2315 time 4.06s Final training 188/4999 loss: 1.2315 time 4.07s 0 Mon Dec 18 13:11:45 2023 Epoch: 189 Epoch 189/5000 23/24 loss: 1.2350 time 3.88s Final training 189/4999 loss: 1.2350 time 3.88s 0 Mon Dec 18 13:11:49 2023 Epoch: 190 Epoch 190/5000 23/24 loss: 1.2122 time 3.60s Final training 190/4999 loss: 1.2122 time 3.60s 0 Mon Dec 18 13:11:53 2023 Epoch: 191 Epoch 191/5000 23/24 loss: 1.2498 time 3.92s Final training 191/4999 loss: 1.2498 time 3.93s 0 Mon Dec 18 13:11:57 2023 Epoch: 192 Epoch 192/5000 23/24 loss: 1.1939 time 3.67s Final training 192/4999 loss: 1.1939 time 3.67s 0 Mon Dec 18 13:12:00 2023 Epoch: 193 Epoch 193/5000 23/24 loss: 1.2043 time 3.83s Final training 193/4999 loss: 1.2043 time 3.83s 0 Mon Dec 18 13:12:04 2023 Epoch: 194 Epoch 194/5000 23/24 loss: 1.2315 time 3.73s Final training 194/4999 loss: 1.2315 time 3.73s 0 Mon Dec 18 13:12:08 2023 Epoch: 195 Epoch 195/5000 23/24 loss: 1.2213 time 3.91s Final training 195/4999 loss: 1.2213 time 3.91s 0 Mon Dec 18 13:12:12 2023 Epoch: 196 Epoch 196/5000 23/24 loss: 1.2053 time 3.68s Final training 196/4999 loss: 1.2053 time 3.68s 0 Mon Dec 18 13:12:15 2023 Epoch: 197 Epoch 197/5000 23/24 loss: 1.2251 time 3.68s Final training 197/4999 loss: 1.2251 time 3.68s 0 Mon Dec 18 13:12:19 2023 Epoch: 198 Epoch 198/5000 23/24 loss: 1.2075 time 3.89s Final training 198/4999 loss: 1.2075 time 3.89s 0 Mon Dec 18 13:12:23 2023 Epoch: 199 Epoch 199/5000 23/24 loss: 1.2070 time 3.94s Final training 199/4999 loss: 1.2070 time 3.94s Val 199/5000 0/6 acc 0.20446874 time 5.06s Val 199/5000 1/6 acc 0.22045086 time 1.82s Val 199/5000 2/6 acc 0.19995542 time 2.73s Val 199/5000 3/6 acc 0.1642402 time 1.46s Val 199/5000 4/6 acc 0.19617088 time 2.03s Val 199/5000 5/6 acc 0.1849853 time 1.09s Final validation 199/4999 acc 0.19504523 time 14.21s new best (0.149238 --> 0.195045). Saving checkpoint ./runs/slimunetr_fold1/model.pt Saving checkpoint ./runs/slimunetr_fold1/model_final.pt Copying to model.pt new best model!!!! 0 Mon Dec 18 13:12:41 2023 Epoch: 200 Epoch 200/5000 23/24 loss: 1.2224 time 3.87s Final training 200/4999 loss: 1.2224 time 3.87s 0 Mon Dec 18 13:12:45 2023 Epoch: 201 Epoch 201/5000 23/24 loss: 1.1885 time 3.91s Final training 201/4999 loss: 1.1885 time 3.91s 0 Mon Dec 18 13:12:49 2023 Epoch: 202 Epoch 202/5000 23/24 loss: 1.1983 time 3.59s Final training 202/4999 loss: 1.1983 time 3.59s 0 Mon Dec 18 13:12:53 2023 Epoch: 203 Epoch 203/5000 23/24 loss: 1.2150 time 3.92s Final training 203/4999 loss: 1.2150 time 3.92s 0 Mon Dec 18 13:12:57 2023 Epoch: 204 Epoch 204/5000 23/24 loss: 1.1973 time 3.94s Final training 204/4999 loss: 1.1973 time 3.94s 0 Mon Dec 18 13:13:01 2023 Epoch: 205 Epoch 205/5000 23/24 loss: 1.1772 time 3.90s Final training 205/4999 loss: 1.1772 time 3.91s 0 Mon Dec 18 13:13:04 2023 Epoch: 206 Epoch 206/5000 23/24 loss: 1.2064 time 3.85s Final training 206/4999 loss: 1.2064 time 3.86s 0 Mon Dec 18 13:13:08 2023 Epoch: 207 Epoch 207/5000 23/24 loss: 1.2046 time 3.54s Final training 207/4999 loss: 1.2046 time 3.54s 0 Mon Dec 18 13:13:12 2023 Epoch: 208 Epoch 208/5000 23/24 loss: 1.1816 time 4.19s Final training 208/4999 loss: 1.1816 time 4.20s 0 Mon Dec 18 13:13:16 2023 Epoch: 209 Epoch 209/5000 23/24 loss: 1.2124 time 4.56s Final training 209/4999 loss: 1.2124 time 4.57s 0 Mon Dec 18 13:13:21 2023 Epoch: 210 Epoch 210/5000 23/24 loss: 1.2079 time 4.26s Final training 210/4999 loss: 1.2079 time 4.26s 0 Mon Dec 18 13:13:25 2023 Epoch: 211 Epoch 211/5000 23/24 loss: 1.1953 time 4.08s Final training 211/4999 loss: 1.1953 time 4.08s 0 Mon Dec 18 13:13:29 2023 Epoch: 212 Epoch 212/5000 23/24 loss: 1.1964 time 3.96s Final training 212/4999 loss: 1.1964 time 3.96s 0 Mon Dec 18 13:13:33 2023 Epoch: 213 Epoch 213/5000 23/24 loss: 1.1894 time 3.83s Final training 213/4999 loss: 1.1894 time 3.84s 0 Mon Dec 18 13:13:37 2023 Epoch: 214 Epoch 214/5000 23/24 loss: 1.2448 time 3.99s Final training 214/4999 loss: 1.2448 time 3.99s 0 Mon Dec 18 13:13:41 2023 Epoch: 215 Epoch 215/5000 23/24 loss: 1.2297 time 3.90s Final training 215/4999 loss: 1.2297 time 3.91s 0 Mon Dec 18 13:13:45 2023 Epoch: 216 Epoch 216/5000 23/24 loss: 1.1912 time 4.02s Final training 216/4999 loss: 1.1912 time 4.02s 0 Mon Dec 18 13:13:49 2023 Epoch: 217 Epoch 217/5000 23/24 loss: 1.1754 time 4.14s Final training 217/4999 loss: 1.1754 time 4.15s 0 Mon Dec 18 13:13:53 2023 Epoch: 218 Epoch 218/5000 23/24 loss: 1.1947 time 3.80s Final training 218/4999 loss: 1.1947 time 3.80s 0 Mon Dec 18 13:13:57 2023 Epoch: 219 Epoch 219/5000 23/24 loss: 1.1694 time 4.00s Final training 219/4999 loss: 1.1694 time 4.01s 0 Mon Dec 18 13:14:01 2023 Epoch: 220 Epoch 220/5000 23/24 loss: 1.1915 time 4.02s Final training 220/4999 loss: 1.1915 time 4.03s 0 Mon Dec 18 13:14:05 2023 Epoch: 221 Epoch 221/5000 23/24 loss: 1.1477 time 3.85s Final training 221/4999 loss: 1.1477 time 3.85s 0 Mon Dec 18 13:14:08 2023 Epoch: 222 Epoch 222/5000 23/24 loss: 1.1481 time 3.91s Final training 222/4999 loss: 1.1481 time 3.91s 0 Mon Dec 18 13:14:12 2023 Epoch: 223 Epoch 223/5000 23/24 loss: 1.1755 time 3.86s Final training 223/4999 loss: 1.1755 time 3.86s 0 Mon Dec 18 13:14:16 2023 Epoch: 224 Epoch 224/5000 23/24 loss: 1.1622 time 3.98s Final training 224/4999 loss: 1.1622 time 3.98s 0 Mon Dec 18 13:14:20 2023 Epoch: 225 Epoch 225/5000 23/24 loss: 1.2029 time 4.00s Final training 225/4999 loss: 1.2029 time 4.00s 0 Mon Dec 18 13:14:24 2023 Epoch: 226 Epoch 226/5000 23/24 loss: 1.1960 time 3.78s Final training 226/4999 loss: 1.1960 time 3.78s 0 Mon Dec 18 13:14:28 2023 Epoch: 227 Epoch 227/5000 23/24 loss: 1.1579 time 3.69s Final training 227/4999 loss: 1.1579 time 3.70s 0 Mon Dec 18 13:14:32 2023 Epoch: 228 Epoch 228/5000 23/24 loss: 1.1743 time 3.74s Final training 228/4999 loss: 1.1743 time 3.75s 0 Mon Dec 18 13:14:35 2023 Epoch: 229 Epoch 229/5000 23/24 loss: 1.1395 time 3.97s Final training 229/4999 loss: 1.1395 time 3.97s 0 Mon Dec 18 13:14:39 2023 Epoch: 230 Epoch 230/5000 23/24 loss: 1.1570 time 3.87s Final training 230/4999 loss: 1.1570 time 3.87s 0 Mon Dec 18 13:14:43 2023 Epoch: 231 Epoch 231/5000 23/24 loss: 1.1702 time 3.76s Final training 231/4999 loss: 1.1702 time 3.76s 0 Mon Dec 18 13:14:47 2023 Epoch: 232 Epoch 232/5000 23/24 loss: 1.1969 time 3.94s Final training 232/4999 loss: 1.1969 time 3.95s 0 Mon Dec 18 13:14:51 2023 Epoch: 233 Epoch 233/5000 23/24 loss: 1.1642 time 3.91s Final training 233/4999 loss: 1.1642 time 3.92s 0 Mon Dec 18 13:14:55 2023 Epoch: 234 Epoch 234/5000 23/24 loss: 1.1365 time 4.10s Final training 234/4999 loss: 1.1365 time 4.11s 0 Mon Dec 18 13:14:59 2023 Epoch: 235 Epoch 235/5000 23/24 loss: 1.1689 time 4.05s Final training 235/4999 loss: 1.1689 time 4.05s 0 Mon Dec 18 13:15:03 2023 Epoch: 236 Epoch 236/5000 23/24 loss: 1.1830 time 3.65s Final training 236/4999 loss: 1.1830 time 3.66s 0 Mon Dec 18 13:15:07 2023 Epoch: 237 Epoch 237/5000 23/24 loss: 1.1509 time 3.93s Final training 237/4999 loss: 1.1509 time 3.93s 0 Mon Dec 18 13:15:11 2023 Epoch: 238 Epoch 238/5000 23/24 loss: 1.1579 time 3.73s Final training 238/4999 loss: 1.1579 time 3.73s 0 Mon Dec 18 13:15:14 2023 Epoch: 239 Epoch 239/5000 23/24 loss: 1.1391 time 3.98s Final training 239/4999 loss: 1.1391 time 3.99s 0 Mon Dec 18 13:15:18 2023 Epoch: 240 Epoch 240/5000 23/24 loss: 1.1730 time 3.91s Final training 240/4999 loss: 1.1730 time 3.91s 0 Mon Dec 18 13:15:22 2023 Epoch: 241 Epoch 241/5000 23/24 loss: 1.1146 time 3.95s Final training 241/4999 loss: 1.1146 time 3.96s 0 Mon Dec 18 13:15:26 2023 Epoch: 242 Epoch 242/5000 23/24 loss: 1.1367 time 3.91s Final training 242/4999 loss: 1.1367 time 3.91s 0 Mon Dec 18 13:15:30 2023 Epoch: 243 Epoch 243/5000 23/24 loss: 1.1433 time 3.85s Final training 243/4999 loss: 1.1433 time 3.86s 0 Mon Dec 18 13:15:34 2023 Epoch: 244 Epoch 244/5000 23/24 loss: 1.1064 time 3.71s Final training 244/4999 loss: 1.1064 time 3.72s 0 Mon Dec 18 13:15:38 2023 Epoch: 245 Epoch 245/5000 23/24 loss: 1.1606 time 3.86s Final training 245/4999 loss: 1.1606 time 3.87s 0 Mon Dec 18 13:15:42 2023 Epoch: 246 Epoch 246/5000 23/24 loss: 1.1294 time 4.01s Final training 246/4999 loss: 1.1294 time 4.01s 0 Mon Dec 18 13:15:46 2023 Epoch: 247 Epoch 247/5000 23/24 loss: 1.1688 time 3.66s Final training 247/4999 loss: 1.1688 time 3.66s 0 Mon Dec 18 13:15:49 2023 Epoch: 248 Epoch 248/5000 23/24 loss: 1.1272 time 3.65s Final training 248/4999 loss: 1.1272 time 3.66s 0 Mon Dec 18 13:15:53 2023 Epoch: 249 Epoch 249/5000 23/24 loss: 1.1132 time 3.87s Final training 249/4999 loss: 1.1132 time 3.88s Val 249/5000 0/6 acc 0.37154913 time 4.80s Val 249/5000 1/6 acc 0.3773623 time 1.76s Val 249/5000 2/6 acc 0.29707712 time 2.85s Val 249/5000 3/6 acc 0.2712974 time 1.43s Val 249/5000 4/6 acc 0.31494597 time 2.06s Val 249/5000 5/6 acc 0.3335623 time 1.11s Final validation 249/4999 acc 0.3276324 time 14.02s new best (0.195045 --> 0.327632). Saving checkpoint ./runs/slimunetr_fold1/model.pt Saving checkpoint ./runs/slimunetr_fold1/model_final.pt Copying to model.pt new best model!!!! 0 Mon Dec 18 13:16:11 2023 Epoch: 250 Epoch 250/5000 23/24 loss: 1.1023 time 3.82s Final training 250/4999 loss: 1.1023 time 3.83s 0 Mon Dec 18 13:16:15 2023 Epoch: 251 Epoch 251/5000 23/24 loss: 1.0964 time 3.91s Final training 251/4999 loss: 1.0964 time 3.92s 0 Mon Dec 18 13:16:19 2023 Epoch: 252 Epoch 252/5000 23/24 loss: 1.1340 time 3.60s Final training 252/4999 loss: 1.1340 time 3.60s 0 Mon Dec 18 13:16:22 2023 Epoch: 253 Epoch 253/5000 23/24 loss: 1.1874 time 3.95s Final training 253/4999 loss: 1.1874 time 3.95s 0 Mon Dec 18 13:16:26 2023 Epoch: 254 Epoch 254/5000 23/24 loss: 1.1403 time 3.75s Final training 254/4999 loss: 1.1403 time 3.75s 0 Mon Dec 18 13:16:30 2023 Epoch: 255 Epoch 255/5000 23/24 loss: 1.1595 time 4.02s Final training 255/4999 loss: 1.1595 time 4.02s 0 Mon Dec 18 13:16:34 2023 Epoch: 256 Epoch 256/5000 23/24 loss: 1.0976 time 3.98s Final training 256/4999 loss: 1.0976 time 3.98s 0 Mon Dec 18 13:16:38 2023 Epoch: 257 Epoch 257/5000 23/24 loss: 1.1245 time 3.72s Final training 257/4999 loss: 1.1245 time 3.73s 0 Mon Dec 18 13:16:42 2023 Epoch: 258 Epoch 258/5000 23/24 loss: 1.1533 time 3.84s Final training 258/4999 loss: 1.1533 time 3.84s 0 Mon Dec 18 13:16:46 2023 Epoch: 259 Epoch 259/5000 23/24 loss: 1.1083 time 3.50s Final training 259/4999 loss: 1.1083 time 3.51s 0 Mon Dec 18 13:16:49 2023 Epoch: 260 Epoch 260/5000 23/24 loss: 1.1216 time 3.72s Final training 260/4999 loss: 1.1216 time 3.72s 0 Mon Dec 18 13:16:53 2023 Epoch: 261 Epoch 261/5000 23/24 loss: 1.1216 time 4.64s Final training 261/4999 loss: 1.1216 time 4.65s 0 Mon Dec 18 13:16:58 2023 Epoch: 262 Epoch 262/5000 23/24 loss: 1.0809 time 4.43s Final training 262/4999 loss: 1.0809 time 4.44s 0 Mon Dec 18 13:17:02 2023 Epoch: 263 Epoch 263/5000 23/24 loss: 1.1126 time 3.84s Final training 263/4999 loss: 1.1126 time 3.85s 0 Mon Dec 18 13:17:06 2023 Epoch: 264 Epoch 264/5000 23/24 loss: 1.1263 time 3.81s Final training 264/4999 loss: 1.1263 time 3.81s 0 Mon Dec 18 13:17:10 2023 Epoch: 265 Epoch 265/5000 23/24 loss: 1.0867 time 3.98s Final training 265/4999 loss: 1.0867 time 3.98s 0 Mon Dec 18 13:17:14 2023 Epoch: 266 Epoch 266/5000 23/24 loss: 1.0927 time 3.90s Final training 266/4999 loss: 1.0927 time 3.91s 0 Mon Dec 18 13:17:18 2023 Epoch: 267 Epoch 267/5000 23/24 loss: 1.1121 time 3.88s Final training 267/4999 loss: 1.1121 time 3.89s 0 Mon Dec 18 13:17:21 2023 Epoch: 268 Epoch 268/5000 23/24 loss: 1.1381 time 3.76s Final training 268/4999 loss: 1.1381 time 3.76s 0 Mon Dec 18 13:17:25 2023 Epoch: 269 Epoch 269/5000 23/24 loss: 1.1050 time 3.77s Final training 269/4999 loss: 1.1050 time 3.78s 0 Mon Dec 18 13:17:29 2023 Epoch: 270 Epoch 270/5000 23/24 loss: 1.1046 time 3.81s Final training 270/4999 loss: 1.1046 time 3.81s 0 Mon Dec 18 13:17:33 2023 Epoch: 271 Epoch 271/5000 23/24 loss: 1.1192 time 3.91s Final training 271/4999 loss: 1.1192 time 3.91s 0 Mon Dec 18 13:17:37 2023 Epoch: 272 Epoch 272/5000 23/24 loss: 1.1055 time 3.74s Final training 272/4999 loss: 1.1055 time 3.74s 0 Mon Dec 18 13:17:40 2023 Epoch: 273 Epoch 273/5000 23/24 loss: 1.0668 time 3.75s Final training 273/4999 loss: 1.0668 time 3.76s 0 Mon Dec 18 13:17:44 2023 Epoch: 274 Epoch 274/5000 23/24 loss: 1.0820 time 4.05s Final training 274/4999 loss: 1.0820 time 4.05s 0 Mon Dec 18 13:17:48 2023 Epoch: 275 Epoch 275/5000 23/24 loss: 1.0732 time 4.00s Final training 275/4999 loss: 1.0732 time 4.01s 0 Mon Dec 18 13:17:52 2023 Epoch: 276 Epoch 276/5000 23/24 loss: 1.0987 time 3.87s Final training 276/4999 loss: 1.0987 time 3.87s 0 Mon Dec 18 13:17:56 2023 Epoch: 277 Epoch 277/5000 23/24 loss: 1.0955 time 3.84s Final training 277/4999 loss: 1.0955 time 3.84s 0 Mon Dec 18 13:18:00 2023 Epoch: 278 Epoch 278/5000 23/24 loss: 1.0993 time 3.87s Final training 278/4999 loss: 1.0993 time 3.88s 0 Mon Dec 18 13:18:04 2023 Epoch: 279 Epoch 279/5000 23/24 loss: 1.0706 time 3.73s Final training 279/4999 loss: 1.0706 time 3.74s 0 Mon Dec 18 13:18:08 2023 Epoch: 280 Epoch 280/5000 23/24 loss: 1.0999 time 4.08s Final training 280/4999 loss: 1.0999 time 4.09s 0 Mon Dec 18 13:18:12 2023 Epoch: 281 Epoch 281/5000 23/24 loss: 1.0631 time 3.79s Final training 281/4999 loss: 1.0631 time 3.79s 0 Mon Dec 18 13:18:15 2023 Epoch: 282 Epoch 282/5000 23/24 loss: 1.0868 time 3.76s Final training 282/4999 loss: 1.0868 time 3.77s 0 Mon Dec 18 13:18:19 2023 Epoch: 283 Epoch 283/5000 23/24 loss: 1.0885 time 4.09s Final training 283/4999 loss: 1.0885 time 4.10s 0 Mon Dec 18 13:18:23 2023 Epoch: 284 Epoch 284/5000 23/24 loss: 1.0863 time 3.94s Final training 284/4999 loss: 1.0863 time 3.94s 0 Mon Dec 18 13:18:27 2023 Epoch: 285 Epoch 285/5000 23/24 loss: 1.0688 time 3.91s Final training 285/4999 loss: 1.0688 time 3.92s 0 Mon Dec 18 13:18:31 2023 Epoch: 286 Epoch 286/5000 23/24 loss: 1.0701 time 3.85s Final training 286/4999 loss: 1.0701 time 3.86s 0 Mon Dec 18 13:18:35 2023 Epoch: 287 Epoch 287/5000 23/24 loss: nan time 3.64s Final training 287/4999 loss: nan time 3.64s 0 Mon Dec 18 13:18:39 2023 Epoch: 288 Epoch 288/5000 23/24 loss: 1.0548 time 4.01s Final training 288/4999 loss: 1.0548 time 4.02s 0 Mon Dec 18 13:18:43 2023 Epoch: 289 Epoch 289/5000 23/24 loss: 1.0489 time 4.10s Final training 289/4999 loss: 1.0489 time 4.10s 0 Mon Dec 18 13:18:47 2023 Epoch: 290 Epoch 290/5000 23/24 loss: 1.0551 time 3.94s Final training 290/4999 loss: 1.0551 time 3.95s 0 Mon Dec 18 13:18:51 2023 Epoch: 291 Epoch 291/5000 23/24 loss: 1.0528 time 3.79s Final training 291/4999 loss: 1.0528 time 3.80s 0 Mon Dec 18 13:18:55 2023 Epoch: 292 Epoch 292/5000 23/24 loss: 1.0423 time 3.70s Final training 292/4999 loss: 1.0423 time 3.70s 0 Mon Dec 18 13:18:58 2023 Epoch: 293 Epoch 293/5000 23/24 loss: nan time 3.63s Final training 293/4999 loss: nan time 3.64s 0 Mon Dec 18 13:19:02 2023 Epoch: 294 Epoch 294/5000 23/24 loss: 1.0555 time 3.64s Final training 294/4999 loss: 1.0555 time 3.64s 0 Mon Dec 18 13:19:06 2023 Epoch: 295 Epoch 295/5000 23/24 loss: 1.0987 time 3.98s Final training 295/4999 loss: 1.0987 time 3.99s 0 Mon Dec 18 13:19:10 2023 Epoch: 296 Epoch 296/5000 23/24 loss: 1.0526 time 3.94s Final training 296/4999 loss: 1.0526 time 3.95s 0 Mon Dec 18 13:19:13 2023 Epoch: 297 Epoch 297/5000 23/24 loss: 1.0682 time 3.70s Final training 297/4999 loss: 1.0682 time 3.71s 0 Mon Dec 18 13:19:17 2023 Epoch: 298 Epoch 298/5000 23/24 loss: 1.0515 time 4.04s Final training 298/4999 loss: 1.0515 time 4.05s 0 Mon Dec 18 13:19:21 2023 Epoch: 299 Epoch 299/5000 23/24 loss: 1.0719 time 3.87s Final training 299/4999 loss: 1.0719 time 3.87s Val 299/5000 0/6 acc 0.068658285 time 4.89s Val 299/5000 1/6 acc 0.0699427 time 2.01s Val 299/5000 2/6 acc 0.06958269 time 3.80s Val 299/5000 3/6 acc 0.06975234 time 1.63s Val 299/5000 4/6 acc 0.068783894 time 2.15s Val 299/5000 5/6 acc 0.06931136 time 1.22s Final validation 299/4999 acc 0.069338545 time 15.71s Saving checkpoint ./runs/slimunetr_fold1/model_final.pt 0 Mon Dec 18 13:19:41 2023 Epoch: 300 Epoch 300/5000 23/24 loss: 1.0531 time 4.00s Final training 300/4999 loss: 1.0531 time 4.00s 0 Mon Dec 18 13:19:45 2023 Epoch: 301 Epoch 301/5000 23/24 loss: 1.0573 time 3.79s Final training 301/4999 loss: 1.0573 time 3.79s 0 Mon Dec 18 13:19:49 2023 Epoch: 302 Epoch 302/5000 23/24 loss: 1.0737 time 3.83s Final training 302/4999 loss: 1.0737 time 3.83s 0 Mon Dec 18 13:19:52 2023 Epoch: 303 Epoch 303/5000 23/24 loss: 1.0434 time 3.79s Final training 303/4999 loss: 1.0434 time 3.80s 0 Mon Dec 18 13:19:56 2023 Epoch: 304 Epoch 304/5000 23/24 loss: 1.0497 time 3.87s Final training 304/4999 loss: 1.0497 time 3.87s 0 Mon Dec 18 13:20:00 2023 Epoch: 305 Epoch 305/5000 23/24 loss: 1.0395 time 3.96s Final training 305/4999 loss: 1.0395 time 3.96s 0 Mon Dec 18 13:20:04 2023 Epoch: 306 Epoch 306/5000 23/24 loss: 1.0536 time 4.05s Final training 306/4999 loss: 1.0536 time 4.05s 0 Mon Dec 18 13:20:08 2023 Epoch: 307 Epoch 307/5000 23/24 loss: 1.0768 time 3.92s Final training 307/4999 loss: 1.0768 time 3.93s 0 Mon Dec 18 13:20:12 2023 Epoch: 308 Epoch 308/5000 23/24 loss: 1.0595 time 3.87s Final training 308/4999 loss: 1.0595 time 3.87s 0 Mon Dec 18 13:20:16 2023 Epoch: 309 Epoch 309/5000 23/24 loss: 1.0679 time 4.15s Final training 309/4999 loss: 1.0679 time 4.16s 0 Mon Dec 18 13:20:20 2023 Epoch: 310 Epoch 310/5000 23/24 loss: 1.0214 time 3.64s Final training 310/4999 loss: 1.0214 time 3.64s 0 Mon Dec 18 13:20:24 2023 Epoch: 311 Epoch 311/5000 23/24 loss: 1.0273 time 3.66s Final training 311/4999 loss: 1.0273 time 3.66s 0 Mon Dec 18 13:20:27 2023 Epoch: 312 Epoch 312/5000 23/24 loss: 1.0439 time 4.39s Final training 312/4999 loss: 1.0439 time 4.39s 0 Mon Dec 18 13:20:32 2023 Epoch: 313 Epoch 313/5000 23/24 loss: 1.0391 time 4.34s Final training 313/4999 loss: 1.0391 time 4.35s 0 Mon Dec 18 13:20:36 2023 Epoch: 314 Epoch 314/5000 23/24 loss: 1.0424 time 4.12s Final training 314/4999 loss: 1.0424 time 4.12s 0 Mon Dec 18 13:20:40 2023 Epoch: 315 Epoch 315/5000 23/24 loss: 1.0455 time 3.60s Final training 315/4999 loss: 1.0455 time 3.61s 0 Mon Dec 18 13:20:44 2023 Epoch: 316 Epoch 316/5000 23/24 loss: 1.0292 time 3.90s Final training 316/4999 loss: 1.0292 time 3.90s 0 Mon Dec 18 13:20:48 2023 Epoch: 317 Epoch 317/5000 23/24 loss: 1.0379 time 3.88s Final training 317/4999 loss: 1.0379 time 3.89s 0 Mon Dec 18 13:20:52 2023 Epoch: 318 Epoch 318/5000 23/24 loss: 1.0263 time 4.08s Final training 318/4999 loss: 1.0263 time 4.09s 0 Mon Dec 18 13:20:56 2023 Epoch: 319 Epoch 319/5000 23/24 loss: 1.0363 time 3.95s Final training 319/4999 loss: 1.0363 time 3.95s 0 Mon Dec 18 13:21:00 2023 Epoch: 320 Epoch 320/5000 23/24 loss: 1.0497 time 3.76s Final training 320/4999 loss: 1.0497 time 3.77s 0 Mon Dec 18 13:21:04 2023 Epoch: 321 Epoch 321/5000 23/24 loss: 1.0412 time 3.83s Final training 321/4999 loss: 1.0412 time 3.83s 0 Mon Dec 18 13:21:07 2023 Epoch: 322 Epoch 322/5000 23/24 loss: 1.0327 time 3.64s Final training 322/4999 loss: 1.0327 time 3.64s 0 Mon Dec 18 13:21:11 2023 Epoch: 323 Epoch 323/5000 23/24 loss: 1.0382 time 3.76s Final training 323/4999 loss: 1.0382 time 3.76s 0 Mon Dec 18 13:21:15 2023 Epoch: 324 Epoch 324/5000 23/24 loss: 1.0458 time 3.84s Final training 324/4999 loss: 1.0458 time 3.85s 0 Mon Dec 18 13:21:19 2023 Epoch: 325 Epoch 325/5000 23/24 loss: 1.0835 time 3.73s Final training 325/4999 loss: 1.0835 time 3.74s 0 Mon Dec 18 13:21:22 2023 Epoch: 326 Epoch 326/5000 23/24 loss: 1.0335 time 3.80s Final training 326/4999 loss: 1.0335 time 3.81s 0 Mon Dec 18 13:21:26 2023 Epoch: 327 Epoch 327/5000 23/24 loss: 1.0163 time 3.92s Final training 327/4999 loss: 1.0163 time 3.92s 0 Mon Dec 18 13:21:30 2023 Epoch: 328 Epoch 328/5000 23/24 loss: 1.0337 time 3.93s Final training 328/4999 loss: 1.0337 time 3.94s 0 Mon Dec 18 13:21:34 2023 Epoch: 329 Epoch 329/5000 23/24 loss: 1.0366 time 3.68s Final training 329/4999 loss: 1.0366 time 3.69s 0 Mon Dec 18 13:21:38 2023 Epoch: 330 Epoch 330/5000 23/24 loss: 1.0527 time 4.08s Final training 330/4999 loss: 1.0527 time 4.08s 0 Mon Dec 18 13:21:42 2023 Epoch: 331 Epoch 331/5000 23/24 loss: 1.0209 time 3.83s Final training 331/4999 loss: 1.0209 time 3.83s 0 Mon Dec 18 13:21:46 2023 Epoch: 332 Epoch 332/5000 23/24 loss: 1.0355 time 3.92s Final training 332/4999 loss: 1.0355 time 3.93s 0 Mon Dec 18 13:21:50 2023 Epoch: 333 Epoch 333/5000 23/24 loss: 1.0388 time 3.89s Final training 333/4999 loss: 1.0388 time 3.89s 0 Mon Dec 18 13:21:53 2023 Epoch: 334 Epoch 334/5000 23/24 loss: 1.0219 time 3.93s Final training 334/4999 loss: 1.0219 time 3.94s 0 Mon Dec 18 13:21:57 2023 Epoch: 335 Epoch 335/5000 23/24 loss: 1.0268 time 3.87s Final training 335/4999 loss: 1.0268 time 3.87s 0 Mon Dec 18 13:22:01 2023 Epoch: 336 Epoch 336/5000 23/24 loss: nan time 4.02s Final training 336/4999 loss: nan time 4.02s 0 Mon Dec 18 13:22:05 2023 Epoch: 337 Epoch 337/5000 23/24 loss: 0.9979 time 3.72s Final training 337/4999 loss: 0.9979 time 3.72s 0 Mon Dec 18 13:22:09 2023 Epoch: 338 Epoch 338/5000 23/24 loss: nan time 3.78s Final training 338/4999 loss: nan time 3.78s 0 Mon Dec 18 13:22:13 2023 Epoch: 339 Epoch 339/5000 23/24 loss: nan time 3.90s Final training 339/4999 loss: nan time 3.90s 0 Mon Dec 18 13:22:17 2023 Epoch: 340 Epoch 340/5000 23/24 loss: nan time 3.64s Final training 340/4999 loss: nan time 3.65s 0 Mon Dec 18 13:22:20 2023 Epoch: 341 Epoch 341/5000 23/24 loss: nan time 3.84s Final training 341/4999 loss: nan time 3.85s 0 Mon Dec 18 13:22:24 2023 Epoch: 342 Epoch 342/5000 23/24 loss: nan time 3.69s Final training 342/4999 loss: nan time 3.70s 0 Mon Dec 18 13:22:28 2023 Epoch: 343 Epoch 343/5000 23/24 loss: nan time 3.91s Final training 343/4999 loss: nan time 3.92s 0 Mon Dec 18 13:22:32 2023 Epoch: 344 Epoch 344/5000 23/24 loss: nan time 3.65s Final training 344/4999 loss: nan time 3.66s 0 Mon Dec 18 13:22:35 2023 Epoch: 345 Epoch 345/5000 23/24 loss: nan time 3.62s Final training 345/4999 loss: nan time 3.63s 0 Mon Dec 18 13:22:39 2023 Epoch: 346 Epoch 346/5000 23/24 loss: nan time 3.68s Final training 346/4999 loss: nan time 3.69s 0 Mon Dec 18 13:22:43 2023 Epoch: 347 Epoch 347/5000 23/24 loss: nan time 3.63s Final training 347/4999 loss: nan time 3.64s 0 Mon Dec 18 13:22:46 2023 Epoch: 348 Epoch 348/5000 23/24 loss: nan time 3.88s Final training 348/4999 loss: nan time 3.88s 0 Mon Dec 18 13:22:50 2023 Epoch: 349 Epoch 349/5000 23/24 loss: nan time 3.71s Final training 349/4999 loss: nan time 3.72s Val 349/5000 0/6 acc 0.068658285 time 4.78s Val 349/5000 1/6 acc 0.0699427 time 1.92s Val 349/5000 2/6 acc 0.06958269 time 3.18s Val 349/5000 3/6 acc 0.06975234 time 1.57s Val 349/5000 4/6 acc 0.068783894 time 2.28s Val 349/5000 5/6 acc 0.06931136 time 1.16s Final validation 349/4999 acc 0.069338545 time 14.88s Saving checkpoint ./runs/slimunetr_fold1/model_final.pt

JJW20230321 commented 4 months ago

out of memory？