Open shimen opened 5 years ago
I have a huge difference in the available GPU memory.
When using the command: nvidia smi | 2 Tesla V100-SXM2... On | 00000000:0A:00.0 Off | 0 | | N/A 43C P0 70W / 300W | 16101MiB / 16160MiB | 0% Default |
When using the cutorch command: cutorch.getMemoryUsage(cutorch.getDevice()) Epoch: [1][2/5] Time 0.724 tripErr 7.65e-02 -------- exit epoch 7911792640 16945512448 -------- start epoch 7915701248 16945512448 (nTrips, nTripsFound) = (5206, 369) Epoch: [1][3/5] Time 0.505 tripErr 8.91e-02 -------- exit epoch 7923902976 16945512448 -------- start epoch 7927670784 16945512448 (nTrips, nTripsFound) = (7141, 265) Epoch: [1][4/5] Time 0.623 tripErr 7.40e-02 -------- exit epoch 7867307008 16945512448 -------- start epoch 7918349824 16945512448 (nTrips, nTripsFound) = (5658, 378) Epoch: [1][5/5] Time 0.469 tripErr 7.18e-02 -------- exit epoch 7922003968 16945512448
How come there is such a huge difference? Almost 0MB free with "nvidia smi" command while the "cutorch" command showing that there is almost 8000mb free.
I have a huge difference in the available GPU memory.
When using the command: nvidia smi | 2 Tesla V100-SXM2... On | 00000000:0A:00.0 Off | 0 | | N/A 43C P0 70W / 300W | 16101MiB / 16160MiB | 0% Default |
When using the cutorch command: cutorch.getMemoryUsage(cutorch.getDevice()) Epoch: [1][2/5] Time 0.724 tripErr 7.65e-02 -------- exit epoch 7911792640 16945512448 -------- start epoch 7915701248 16945512448 (nTrips, nTripsFound) = (5206, 369) Epoch: [1][3/5] Time 0.505 tripErr 8.91e-02 -------- exit epoch 7923902976 16945512448 -------- start epoch 7927670784 16945512448 (nTrips, nTripsFound) = (7141, 265) Epoch: [1][4/5] Time 0.623 tripErr 7.40e-02 -------- exit epoch 7867307008 16945512448 -------- start epoch 7918349824 16945512448 (nTrips, nTripsFound) = (5658, 378) Epoch: [1][5/5] Time 0.469 tripErr 7.18e-02 -------- exit epoch 7922003968 16945512448
How come there is such a huge difference? Almost 0MB free with "nvidia smi" command while the "cutorch" command showing that there is almost 8000mb free.