Open danishansari opened 1 year ago
Why does the memory requirement keep on increasing for several iterations? Is this an expected behaviour for transformers? Even with resnet50 as backbone, on 16gb GPU I am able to run with only 2 batch-size?
Epoch: [0] [ 0/8819] eta: 6:34:13 lr: 0.000100 loss: 33.1388 (33.1388) time: 2.6822 data: 1.0370 max mem: 3331 Epoch: [0] [ 10/8819] eta: 1:06:08 lr: 0.000100 loss: 28.7849 (29.3637) time: 0.4505 data: 0.1035 max mem: 3409 Epoch: [0] [ 20/8819] eta: 0:52:00 lr: 0.000100 loss: 25.1816 (26.4967) time: 0.2383 data: 0.0167 max mem: 4454 Epoch: [0] [ 30/8819] eta: 0:46:17 lr: 0.000100 loss: 22.1077 (24.6563) time: 0.2421 data: 0.0178 max mem: 4454 Epoch: [0] [ 40/8819] eta: 0:43:19 lr: 0.000100 loss: 19.9209 (23.0615) time: 0.2347 data: 0.0120 max mem: 4454 Epoch: [0] [ 50/8819] eta: 0:43:07 lr: 0.000100 loss: 16.4599 (21.6173) time: 0.2626 data: 0.0404 max mem: 4454 Epoch: [0] [ 60/8819] eta: 0:42:41 lr: 0.000100 loss: 13.7710 (20.2928) time: 0.2848 data: 0.0647 max mem: 4454 Epoch: [0] [ 70/8819] eta: 0:41:28 lr: 0.000100 loss: 12.7223 (19.3061) time: 0.2573 data: 0.0365 max mem: 5913 Epoch: [0] [ 80/8819] eta: 0:40:41 lr: 0.000100 loss: 12.4632 (18.5024) time: 0.2398 data: 0.0171 max mem: 5913 Epoch: [0] [ 90/8819] eta: 0:40:52 lr: 0.000100 loss: 12.4809 (17.8598) time: 0.2686 data: 0.0488 max mem: 5913
Why does the memory requirement keep on increasing for several iterations? Is this an expected behaviour for transformers? Even with resnet50 as backbone, on 16gb GPU I am able to run with only 2 batch-size?
Epoch: [0] [ 0/8819] eta: 6:34:13 lr: 0.000100 loss: 33.1388 (33.1388) time: 2.6822 data: 1.0370 max mem: 3331 Epoch: [0] [ 10/8819] eta: 1:06:08 lr: 0.000100 loss: 28.7849 (29.3637) time: 0.4505 data: 0.1035 max mem: 3409 Epoch: [0] [ 20/8819] eta: 0:52:00 lr: 0.000100 loss: 25.1816 (26.4967) time: 0.2383 data: 0.0167 max mem: 4454 Epoch: [0] [ 30/8819] eta: 0:46:17 lr: 0.000100 loss: 22.1077 (24.6563) time: 0.2421 data: 0.0178 max mem: 4454 Epoch: [0] [ 40/8819] eta: 0:43:19 lr: 0.000100 loss: 19.9209 (23.0615) time: 0.2347 data: 0.0120 max mem: 4454 Epoch: [0] [ 50/8819] eta: 0:43:07 lr: 0.000100 loss: 16.4599 (21.6173) time: 0.2626 data: 0.0404 max mem: 4454 Epoch: [0] [ 60/8819] eta: 0:42:41 lr: 0.000100 loss: 13.7710 (20.2928) time: 0.2848 data: 0.0647 max mem: 4454 Epoch: [0] [ 70/8819] eta: 0:41:28 lr: 0.000100 loss: 12.7223 (19.3061) time: 0.2573 data: 0.0365 max mem: 5913 Epoch: [0] [ 80/8819] eta: 0:40:41 lr: 0.000100 loss: 12.4632 (18.5024) time: 0.2398 data: 0.0171 max mem: 5913 Epoch: [0] [ 90/8819] eta: 0:40:52 lr: 0.000100 loss: 12.4809 (17.8598) time: 0.2686 data: 0.0488 max mem: 5913