GPU utilization patterns

szilard commented 5 years ago

xgboost:

1M:

[0] Tesla V100-SXM2-16GB | 29'C,   0 % |     0 / 16160 MB |
[0] Tesla V100-SXM2-16GB | 29'C,  62 % |  2278 / 16160 MB | root(2268M)
[0] Tesla V100-SXM2-16GB | 30'C,  80 % |  1096 / 16160 MB | root(1086M)
[0] Tesla V100-SXM2-16GB | 30'C,  80 % |  1096 / 16160 MB | root(1086M)
[0] Tesla V100-SXM2-16GB | 30'C,  79 % |  1096 / 16160 MB | root(1086M)
[0] Tesla V100-SXM2-16GB | 30'C,   0 % |  1098 / 16160 MB | root(1088M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % |     0 / 16160 MB |

Screen Shot 2019-04-30 at 8 28 10 AM

03:10:48 PM  all    1.38    0.00    0.25    0.00    0.00    0.00    0.00    0.00    0.00   98.37
03:10:49 PM  all   10.50    0.00    3.38    0.00    0.00    0.00    0.00    0.00    0.00   86.12
03:10:50 PM  all   10.26    0.00    3.88    0.00    0.00    0.00    0.00    0.00    0.00   85.86
03:10:51 PM  all   10.78    0.00    2.13    0.00    0.00    0.00    0.00    0.00    0.00   87.09
03:10:52 PM  all   11.36    0.00    2.50    0.00    0.00    0.00    0.00    0.00    0.00   86.14
03:10:53 PM  all   11.26    0.00    3.00    0.00    0.00    0.00    0.00    0.00    0.00   85.73
03:10:54 PM  all    7.18    0.00    2.52    0.13    0.00    0.00    0.13    0.00    0.00   90.05

Screen Shot 2019-04-30 at 8 27 56 AM

10M:

[0] Tesla V100-SXM2-16GB | 26'C,   0 % |     0 / 16160 MB |
[0] Tesla V100-SXM2-16GB | 27'C,  42 % |  2624 / 16160 MB | root(2614M)
[0] Tesla V100-SXM2-16GB | 27'C,  77 % |  2624 / 16160 MB | root(2614M)
[0] Tesla V100-SXM2-16GB | 28'C,  76 % |  2624 / 16160 MB | root(2614M)
[0] Tesla V100-SXM2-16GB | 28'C,  73 % |  2624 / 16160 MB | root(2614M)
[0] Tesla V100-SXM2-16GB | 28'C,  80 % |  2624 / 16160 MB | root(2614M)
[0] Tesla V100-SXM2-16GB | 29'C,  86 % |  1612 / 16160 MB | root(1602M)
[0] Tesla V100-SXM2-16GB | 30'C,  86 % |  1612 / 16160 MB | root(1602M)
[0] Tesla V100-SXM2-16GB | 30'C,  87 % |  1612 / 16160 MB | root(1602M)
[0] Tesla V100-SXM2-16GB | 30'C,  86 % |  1612 / 16160 MB | root(1602M)
[0] Tesla V100-SXM2-16GB | 30'C,  86 % |  1612 / 16160 MB | root(1602M)
[0] Tesla V100-SXM2-16GB | 30'C,  85 % |  1612 / 16160 MB | root(1602M)
[0] Tesla V100-SXM2-16GB | 31'C,  86 % |  1612 / 16160 MB | root(1602M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % |     0 / 16160 MB |

08:41:22 PM  all    1.38    0.00    0.25    0.00    0.00    0.00    0.00    0.00    0.00   98.37
08:41:23 PM  all    6.75    0.00    0.38    0.00    0.00    0.00    0.12    0.00    0.00   92.75
08:41:24 PM  all   11.50    0.00    2.00    0.00    0.00    0.00    0.00    0.00    0.00   86.50
08:41:25 PM  all   10.62    0.00    2.88    0.00    0.00    0.00    0.00    0.00    0.00   86.50
08:41:26 PM  all   11.53    0.00    2.63    0.00    0.00    0.00    0.00    0.00    0.00   85.84
08:41:27 PM  all   11.62    0.00    2.38    0.00    0.00    0.00    0.00    0.00    0.00   86.00
08:41:28 PM  all   11.79    0.00    2.26    0.00    0.00    0.00    0.00    0.00    0.00   85.95
08:41:29 PM  all   11.62    0.00    2.62    0.00    0.00    0.00    0.00    0.00    0.00   85.75
08:41:30 PM  all   11.38    0.00    2.75    0.00    0.00    0.00    0.00    0.00    0.00   85.88
08:41:31 PM  all   10.28    0.00    3.76    0.00    0.00    0.00    0.00    0.00    0.00   85.96

08:41:31 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
08:41:32 PM  all    9.75    0.00    4.50    0.00    0.00    0.00    0.00    0.00    0.00   85.75
08:41:33 PM  all    9.64    0.00    3.50    0.00    0.00    0.00    0.00    0.00    0.00   86.86
08:41:34 PM  all    9.91    0.00    3.39    0.00    0.00    0.00    0.00    0.00    0.00   86.70
08:41:35 PM  all   10.74    0.00    3.50    0.00    0.00    0.00    0.00    0.00    0.00   85.77
08:41:36 PM  all   10.14    0.00    4.01    0.00    0.00    0.00    0.00    0.00    0.00   85.86
08:41:37 PM  all   10.34    0.00    3.78    0.00    0.00    0.00    0.00    0.00    0.00   85.88
08:41:38 PM  all    1.39    0.00    1.01    0.13    0.00    0.00    0.00    0.00    0.00   97.48

szilard commented 5 years ago

h2o xgboost:

1M:

[0] Tesla V100-SXM2-16GB | 29'C,   0 % |     0 / 16160 MB |
[0] Tesla V100-SXM2-16GB | 29'C,   0 % |   454 / 16160 MB | root(444M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % |   454 / 16160 MB | root(444M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % |   510 / 16160 MB | root(500M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % |   510 / 16160 MB | root(500M)
[0] Tesla V100-SXM2-16GB | 29'C,  24 % |   510 / 16160 MB | root(500M)
[0] Tesla V100-SXM2-16GB | 30'C,  78 % |   510 / 16160 MB | root(500M)
[0] Tesla V100-SXM2-16GB | 31'C,  79 % |   510 / 16160 MB | root(500M)
[0] Tesla V100-SXM2-16GB | 31'C,  80 % |   510 / 16160 MB | root(500M)
[0] Tesla V100-SXM2-16GB | 30'C,   0 % |   510 / 16160 MB | root(500M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % |   510 / 16160 MB | root(500M)
[0] Tesla V100-SXM2-16GB | 30'C,   0 % |   510 / 16160 MB | root(500M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % |   510 / 16160 MB | root(500M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % |   444 / 16160 MB | root(434M)
[0] Tesla V100-SXM2-16GB | 29'C,   6 % |     0 / 16160 MB |

Screen Shot 2019-04-30 at 8 35 40 AM

03:22:51 PM  all    1.50    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00   98.12
03:22:52 PM  all   17.74    0.00    1.64    0.00    0.00    0.00    0.13    0.00    0.00   80.50
03:22:53 PM  all   51.19    0.00    1.38    0.00    0.00    0.00    0.00    0.00    0.00   47.42
03:22:54 PM  all   57.02    0.00    2.53    0.00    0.00    0.00    0.13    0.00    0.00   40.33
03:22:55 PM  all   85.82    0.00    2.03    0.00    0.00    0.00    0.13    0.00    0.00   12.03

03:22:55 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
03:22:56 PM  all   77.50    0.00    2.28    0.00    0.00    0.00    0.00    0.00    0.00   20.23
03:22:57 PM  all   78.63    0.00    1.39    0.00    0.00    0.00    0.00    0.00    0.00   19.97
03:22:58 PM  all   34.47    0.00    5.41    0.00    0.00    0.00    0.13    0.00    0.00   60.00
03:22:59 PM  all   17.00    0.00    5.16    0.00    0.00    0.00    0.00    0.00    0.00   77.83
03:23:00 PM  all   17.96    0.00    4.61    0.00    0.00    0.00    0.00    0.00    0.00   77.43
03:23:01 PM  all   37.44    0.00    4.02    0.00    0.00    0.00    0.00    0.00    0.00   58.54
03:23:02 PM  all   99.25    0.00    0.75    0.00    0.00    0.00    0.00    0.00    0.00    0.00
03:23:03 PM  all   88.51    0.00    1.26    0.00    0.00    0.00    0.00    0.00    0.00   10.23
03:23:04 PM  all   51.31    0.00    2.63    0.00    0.00    0.00    0.00    0.00    0.00   46.06
03:23:05 PM  all   99.12    0.00    0.63    0.00    0.00    0.00    0.00    0.00    0.00    0.25
03:23:06 PM  all   96.00    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00    3.62
03:23:07 PM  all   38.74    0.00    1.64    0.88    0.00    0.00    0.00    0.00    0.00   58.74
03:23:08 PM  all   24.02    0.00    1.64    0.13    0.00    0.13    0.13    0.00    0.00   73.96
03:23:09 PM  all    1.25    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00   98.25

Screen Shot 2019-04-30 at 8 36 45 AM

10M:

[0] Tesla V100-SXM2-16GB | 26'C,   0 % |     0 / 16160 MB |
[0] Tesla V100-SXM2-16GB | 27'C,   4 % |   454 / 16160 MB | root(444M)
[0] Tesla V100-SXM2-16GB | 27'C,   0 % |   444 / 16160 MB | root(434M)
[0] Tesla V100-SXM2-16GB | 27'C,   0 % |   444 / 16160 MB | root(434M)
[0] Tesla V100-SXM2-16GB | 27'C,   0 % |   444 / 16160 MB | root(434M)
[0] Tesla V100-SXM2-16GB | 27'C,   0 % |   444 / 16160 MB | root(434M)
[0] Tesla V100-SXM2-16GB | 27'C,   0 % |   444 / 16160 MB | root(434M)
[0] Tesla V100-SXM2-16GB | 27'C,   0 % |   444 / 16160 MB | root(434M)
[0] Tesla V100-SXM2-16GB | 27'C,   0 % |   452 / 16160 MB | root(442M)
[0] Tesla V100-SXM2-16GB | 27'C,   0 % |   452 / 16160 MB | root(442M)
[0] Tesla V100-SXM2-16GB | 28'C,  58 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 29'C,  69 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 29'C,  72 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 31'C,  69 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 30'C,  70 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 31'C,  60 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 30'C,  69 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 32'C,  67 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 31'C,  68 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 31'C,  69 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 31'C,  63 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 31'C,  58 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 32'C,  64 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 32'C,  63 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 31'C,  69 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 30'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 30'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 28'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 28'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 28'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 28'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 28'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 28'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 28'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 28'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 28'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 28'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 28'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 28'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 28'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 28'C,   0 % |   850 / 16160 MB | root(840M)
[0] Tesla V100-SXM2-16GB | 28'C,   0 % |   444 / 16160 MB | root(434M)
[0] Tesla V100-SXM2-16GB | 28'C,   0 % |     0 / 16160 MB |

08:49:38 PM  all    0.88    0.00    0.13    0.00    0.00    0.00    0.00    0.00    0.00   99.00
08:49:39 PM  all   46.39    0.00    1.77    0.00    0.00    0.00    0.38    0.00    0.00   51.46
08:49:40 PM  all   34.13    0.00    2.91    0.00    0.00    0.00    0.00    0.00    0.00   62.96
08:49:41 PM  all   13.88    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00   85.62
08:49:42 PM  all   12.16    0.00    2.26    0.00    0.00    0.00    0.00    0.00    0.00   85.59
08:49:43 PM  all   10.88    0.00    3.50    0.00    0.00    0.00    0.00    0.00    0.00   85.62
08:49:44 PM  all   74.34    0.00    2.89    0.00    0.00    0.00    0.13    0.00    0.00   22.64
08:49:45 PM  all   92.69    0.00    1.89    0.00    0.00    0.00    0.00    0.00    0.00    5.42
08:49:46 PM  all   80.90    0.00    2.76    0.00    0.00    0.00    0.00    0.00    0.00   16.33
08:49:47 PM  all   46.88    0.00    1.00    0.00    0.00    0.00    0.00    0.00    0.00   52.12
08:49:48 PM  all   28.07    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00   71.43
08:49:49 PM  all   40.28    0.00    2.01    0.00    0.00    0.00    0.00    0.00    0.00   57.72
08:49:50 PM  all   32.75    0.00    2.88    0.00    0.00    0.00    0.00    0.00    0.00   64.38
08:49:51 PM  all   26.53    0.00    5.51    0.25    0.00    0.00    0.13    0.00    0.00   67.58
08:49:52 PM  all   30.61    0.00    4.64    0.00    0.00    0.00    0.00    0.00    0.00   64.74
08:49:53 PM  all   28.57    0.00    4.76    0.00    0.00    0.00    0.00    0.00    0.00   66.67
08:49:54 PM  all   28.45    0.00    4.14    0.00    0.00    0.00    0.00    0.00    0.00   67.42
08:49:55 PM  all   31.45    0.00    3.65    0.00    0.00    0.00    0.00    0.00    0.00   64.91
08:49:56 PM  all   31.45    0.00    3.88    0.00    0.00    0.00    0.00    0.00    0.00   64.66
08:49:57 PM  all   31.03    0.00    3.89    0.00    0.00    0.00    0.00    0.00    0.00   65.08
08:49:58 PM  all   30.61    0.00    3.76    0.00    0.00    0.00    0.00    0.00    0.00   65.62
08:49:59 PM  all   31.37    0.00    4.39    0.00    0.00    0.00    0.00    0.00    0.00   64.24
08:50:00 PM  all   31.53    0.00    3.77    0.00    0.00    0.00    0.00    0.00    0.00   64.70
08:50:01 PM  all   30.33    0.00    3.63    0.00    0.00    0.00    0.00    0.00    0.00   66.04
08:50:02 PM  all   30.65    0.00    4.40    0.00    0.00    0.00    0.00    0.00    0.00   64.95
08:50:03 PM  all   32.46    0.00    4.14    0.00    0.00    0.00    0.00    0.00    0.00   63.41
08:50:04 PM  all   34.00    0.00    3.50    0.00    0.00    0.00    0.00    0.00    0.00   62.50
08:50:05 PM  all   30.70    0.00    4.26    0.00    0.00    0.00    0.00    0.00    0.00   65.04
08:50:06 PM  all   36.51    0.00    4.27    0.00    0.00    0.00    0.00    0.00    0.00   59.22
08:50:07 PM  all   98.88    0.00    1.12    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:08 PM  all   99.75    0.00    0.25    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:09 PM  all   99.62    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:10 PM  all   99.38    0.00    0.62    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:11 PM  all   99.25    0.00    0.62    0.00    0.00    0.00    0.00    0.00    0.00    0.12
08:50:12 PM  all   96.12    0.00    2.00    0.00    0.00    0.00    0.00    0.00    0.00    1.88
08:50:13 PM  all   99.88    0.00    0.12    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:14 PM  all   99.62    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:15 PM  all   99.38    0.00    0.62    0.00    0.00    0.00    0.00    0.00    0.00    0.00

08:50:15 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
08:50:16 PM  all   99.62    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:17 PM  all   98.75    0.00    0.63    0.00    0.00    0.00    0.00    0.00    0.00    0.63
08:50:18 PM  all   99.75    0.00    0.25    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:19 PM  all   99.88    0.00    0.12    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:20 PM  all   99.50    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:21 PM  all   99.75    0.00    0.25    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:22 PM  all   99.38    0.00    0.62    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:23 PM  all   99.50    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:24 PM  all   99.62    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:25 PM  all   99.50    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:26 PM  all   99.38    0.00    0.62    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:27 PM  all   99.50    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:28 PM  all   85.98    0.00    1.63    0.00    0.00    0.00    0.00    0.00    0.00   12.39
08:50:29 PM  all   86.36    0.00    0.25    0.00    0.00    0.00    0.00    0.00    0.00   13.39
08:50:30 PM  all   44.65    0.00    2.52    0.00    0.00    0.00    0.00    0.00    0.00   52.83
08:50:31 PM  all   99.50    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:32 PM  all   99.75    0.00    0.25    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:33 PM  all   99.62    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:34 PM  all   99.62    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:35 PM  all   99.87    0.00    0.13    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:36 PM  all   98.75    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00    0.75
08:50:37 PM  all   99.75    0.00    0.25    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:38 PM  all   99.50    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:39 PM  all   99.50    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:40 PM  all   99.75    0.00    0.25    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:41 PM  all   99.12    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00    0.50
08:50:42 PM  all   98.13    0.00    1.87    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:43 PM  all   99.62    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:44 PM  all   99.62    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:45 PM  all   99.75    0.00    0.25    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:46 PM  all   99.75    0.00    0.25    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:47 PM  all   99.50    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00    0.12
08:50:48 PM  all   99.62    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:49 PM  all   99.50    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:50 PM  all   99.88    0.00    0.12    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:51 PM  all   99.75    0.00    0.25    0.00    0.00    0.00    0.00    0.00    0.00    0.00
08:50:52 PM  all   90.99    0.00    0.75    0.00    0.00    0.00    0.00    0.00    0.00    8.26
08:50:53 PM  all   89.92    0.00    0.50    0.00    0.00    0.00    0.13    0.00    0.00    9.45
08:50:54 PM  all   19.90    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00   79.60
08:50:55 PM  all   53.69    0.00    2.67    0.00    0.00    0.00    0.13    0.00    0.00   43.51
08:50:56 PM  all    1.51    0.00    1.13    0.00    0.00    0.00    0.00    0.00    0.00   97.36

szilard commented 5 years ago

catboost:

1M:

[0] Tesla V100-SXM2-16GB | 28'C,   0 % |     0 / 16160 MB |
[0] Tesla V100-SXM2-16GB | 28'C,   6 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 28'C,   4 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 30'C,  85 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 30'C,  86 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 30'C,  81 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 31'C,  82 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 31'C,  84 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 30'C,   0 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 29'C,  92 % |   452 / 16160 MB | root(442M)
[0] Tesla V100-SXM2-16GB | 29'C,   7 % |     0 / 16160 MB |

Screen Shot 2019-05-01 at 11 58 16 AM

08:35:45 PM  all    1.25    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00   98.38
08:35:46 PM  all   15.31    0.00    2.01    0.00    0.00    0.00    0.00    0.00    0.00   82.69
08:35:47 PM  all   19.15    0.00   23.90    0.00    0.00    0.00    0.00    0.00    0.00   56.95
08:35:48 PM  all   20.05    0.00   10.47    0.00    0.00    0.00    0.00    0.00    0.00   69.48
08:35:49 PM  all   14.14    0.00    8.96    0.00    0.00    0.00    0.13    0.00    0.00   76.77
08:35:50 PM  all    8.33    0.00    6.15    0.00    0.00    0.00    0.13    0.00    0.00   85.38
08:35:51 PM  all    9.71    0.00    6.26    0.00    0.00    0.00    0.13    0.00    0.00   83.91
08:35:52 PM  all    9.24    0.00    7.45    0.00    0.00    0.00    0.13    0.00    0.00   83.18
08:35:53 PM  all    8.25    0.00    7.60    0.00    0.00    0.00    0.13    0.00    0.00   84.02
08:35:54 PM  all    8.71    0.00    7.68    0.00    0.00    0.00    0.00    0.00    0.00   83.61
08:35:55 PM  all   54.24    0.00   10.24    0.00    0.00    0.00    0.25    0.00    0.00   35.27
08:35:56 PM  all   73.56    0.00    8.77    0.00    0.00    0.00    0.00    0.00    0.00   17.67
08:35:57 PM  all   30.36    0.00   10.04    0.00    0.00    0.00    0.00    0.00    0.00   59.60
08:35:58 PM  all    1.65    0.00    0.76    0.25    0.00    0.00    0.00    0.00    0.00   97.34

Screen Shot 2019-05-01 at 11 58 53 AM

10M:

[0] Tesla V100-SXM2-16GB | 26'C,   0 % |     0 / 16160 MB |
[0] Tesla V100-SXM2-16GB | 27'C,  16 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 27'C,   8 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 27'C,  10 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 27'C,  13 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 27'C,   7 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 27'C,   3 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 27'C,   4 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 27'C,   7 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 27'C,   2 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 27'C,   0 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 27'C,   7 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 27'C,   2 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 27'C,   0 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 27'C,   4 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 27'C,   2 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 28'C,  11 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 29'C,  71 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 30'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 30'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 30'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 30'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 30'C,  97 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 30'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 31'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 31'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 31'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 31'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 31'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 31'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 31'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 31'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 31'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  97 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 33'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  97 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  96 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 33'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 33'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 33'C,  97 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 33'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 33'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 33'C,  99 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 33'C,  98 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 32'C, 100 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 31'C,   0 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 31'C,   0 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 30'C,   0 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 30'C,   0 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 30'C,   0 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % | 15376 / 16160 MB | root(15366M)
[0] Tesla V100-SXM2-16GB | 29'C, 100 % |   452 / 16160 MB | root(442M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % |   452 / 16160 MB | root(442M)
[0] Tesla V100-SXM2-16GB | 29'C,   0 % |     0 / 16160 MB |

Screen Shot 2019-05-01 at 12 01 56 PM

09:06:58 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
09:06:59 PM  all    1.25    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00   98.37
09:07:00 PM  all    5.00    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00   94.62
09:07:01 PM  all   67.12    0.00    4.25    0.00    0.00    0.00    0.00    0.00    0.00   28.62
09:07:02 PM  all   94.63    0.00    4.49    0.00    0.00    0.00    0.00    0.00    0.00    0.87
09:07:03 PM  all   25.97    0.00    2.89    0.00    0.00    0.00    0.00    0.00    0.00   71.14
09:07:04 PM  all   27.00    0.00   32.12    0.00    0.00    0.00    0.00    0.00    0.00   40.88
09:07:05 PM  all   41.43    0.00   58.32    0.00    0.00    0.00    0.00    0.00    0.00    0.25
09:07:06 PM  all   18.02    0.00    1.75    0.00    0.00    0.00    0.00    0.00    0.00   80.23
09:07:07 PM  all   19.00    0.00   29.12    0.00    0.00    0.00    0.00    0.00    0.00   51.88
09:07:08 PM  all   27.78    0.00   65.08    0.00    0.00    0.00    0.00    0.00    0.00    7.13
09:07:09 PM  all   12.17    0.00    2.01    0.00    0.00    0.00    0.00    0.00    0.00   85.82
09:07:10 PM  all   21.31    0.00   11.98    0.00    0.00    0.00    0.00    0.00    0.00   66.71
09:07:11 PM  all   19.10    0.00   10.24    0.00    0.00    0.00    0.12    0.00    0.00   70.54
09:07:12 PM  all   19.02    0.00   10.26    0.00    0.00    0.00    0.00    0.00    0.00   70.71
09:07:13 PM  all   18.52    0.00   10.51    0.00    0.00    0.00    0.00    0.00    0.00   70.96
09:07:14 PM  all   21.03    0.00   10.26    0.00    0.00    0.00    0.00    0.00    0.00   68.71
09:07:15 PM  all   18.57    0.00   10.29    0.00    0.00    0.00    0.00    0.00    0.00   71.14
09:07:16 PM  all   19.40    0.00   10.01    0.00    0.00    0.00    0.00    0.00    0.00   70.59
09:07:17 PM  all   20.73    0.00    9.55    0.00    0.00    0.00    0.00    0.00    0.00   69.72
09:07:18 PM  all   21.03    0.00   10.45    0.00    0.00    0.00    0.00    0.00    0.00   68.51
09:07:19 PM  all   22.00    0.00   10.12    0.00    0.00    0.00    0.00    0.00    0.00   67.88
09:07:20 PM  all   20.08    0.00   10.41    0.00    0.00    0.00    0.00    0.00    0.00   69.51
09:07:21 PM  all   22.31    0.00   10.40    0.00    0.00    0.00    0.00    0.00    0.00   67.29
09:07:22 PM  all   20.43    0.00    9.77    0.00    0.00    0.00    0.00    0.00    0.00   69.80
09:07:23 PM  all   22.22    0.00   10.24    0.00    0.00    0.00    0.00    0.00    0.00   67.54
09:07:24 PM  all   21.96    0.00    9.91    0.00    0.00    0.00    0.00    0.00    0.00   68.13
09:07:25 PM  all   20.57    0.00    9.98    0.00    0.00    0.00    0.12    0.00    0.00   69.33
09:07:26 PM  all   18.47    0.00   10.05    0.00    0.00    0.00    0.00    0.00    0.00   71.48
09:07:27 PM  all   12.92    0.00    8.66    0.00    0.00    0.00    0.00    0.00    0.00   78.42
09:07:28 PM  all    6.78    0.00    7.54    0.00    0.00    0.00    0.00    0.00    0.00   85.68
09:07:29 PM  all    7.89    0.00    6.77    0.00    0.00    0.00    0.00    0.00    0.00   85.34
09:07:30 PM  all    7.13    0.00    7.38    0.00    0.00    0.00    0.00    0.00    0.00   85.48
09:07:31 PM  all    7.65    0.00    6.52    0.00    0.00    0.00    0.00    0.00    0.00   85.82
09:07:32 PM  all    7.76    0.00    6.63    0.00    0.00    0.00    0.00    0.00    0.00   85.61
09:07:33 PM  all    7.15    0.00    7.28    0.00    0.00    0.00    0.00    0.00    0.00   85.57
09:07:34 PM  all    6.64    0.00    6.52    0.00    0.00    0.00    0.00    0.00    0.00   86.84
09:07:35 PM  all    8.76    0.00    6.13    0.00    0.00    0.00    0.00    0.00    0.00   85.11
09:07:36 PM  all    7.28    0.00    7.28    0.00    0.00    0.00    0.00    0.00    0.00   85.45
09:07:37 PM  all    6.88    0.00    7.51    0.00    0.00    0.00    0.00    0.00    0.00   85.61
09:07:38 PM  all    6.89    0.00    7.52    0.00    0.00    0.00    0.00    0.00    0.00   85.59
09:07:39 PM  all    7.38    0.00    7.01    0.00    0.00    0.00    0.13    0.00    0.00   85.48
09:07:40 PM  all    7.28    0.00    7.40    0.00    0.00    0.00    0.00    0.00    0.00   85.32
09:07:41 PM  all    7.38    0.00    7.01    0.00    0.00    0.00    0.00    0.00    0.00   85.61
09:07:42 PM  all    8.15    0.00    6.27    0.00    0.00    0.00    0.00    0.00    0.00   85.59
09:07:43 PM  all    7.13    0.00    6.13    0.00    0.00    0.00    0.00    0.00    0.00   86.73
09:07:44 PM  all    7.14    0.00    7.27    0.00    0.00    0.00    0.00    0.00    0.00   85.59
09:07:45 PM  all    7.24    0.00    7.49    0.00    0.00    0.00    0.00    0.00    0.00   85.27
09:07:46 PM  all    7.39    0.00    7.02    0.00    0.00    0.00    0.00    0.00    0.00   85.59
09:07:47 PM  all    7.40    0.00    6.90    0.00    0.00    0.00    0.00    0.00    0.00   85.70
09:07:48 PM  all    7.88    0.00    6.51    0.00    0.00    0.00    0.00    0.00    0.00   85.61
09:07:49 PM  all    7.13    0.00    7.26    0.00    0.00    0.00    0.00    0.00    0.00   85.61
09:07:50 PM  all    6.90    0.00    7.53    0.00    0.00    0.00    0.00    0.00    0.00   85.57
09:07:51 PM  all    7.39    0.00    6.77    0.00    0.00    0.00    0.00    0.00    0.00   85.84

09:07:51 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
09:07:52 PM  all    6.76    0.00    7.38    0.00    0.00    0.00    0.00    0.00    0.00   85.86
09:07:53 PM  all    8.27    0.00    6.14    0.00    0.00    0.00    0.13    0.00    0.00   85.46
09:07:54 PM  all    7.77    0.00    6.52    0.00    0.00    0.00    0.00    0.00    0.00   85.71
09:07:55 PM  all    7.14    0.00    7.52    0.00    0.00    0.00    0.00    0.00    0.00   85.34
09:07:56 PM  all    8.14    0.00    6.38    0.00    0.00    0.00    0.00    0.00    0.00   85.48
09:07:57 PM  all    8.27    0.00    6.02    0.00    0.00    0.00    0.13    0.00    0.00   85.59
09:07:58 PM  all    7.17    0.00    7.30    0.00    0.00    0.00    0.00    0.00    0.00   85.53
09:07:59 PM  all    7.27    0.00    7.27    0.00    0.00    0.00    0.00    0.00    0.00   85.46
09:08:00 PM  all    6.52    0.00    6.89    0.00    0.00    0.00    0.00    0.00    0.00   86.59
09:08:01 PM  all    7.39    0.00    6.52    0.00    0.00    0.00    0.00    0.00    0.00   86.09
09:08:02 PM  all    7.53    0.00    7.03    0.00    0.00    0.00    0.00    0.00    0.00   85.45
09:08:03 PM  all    7.88    0.00    6.63    0.00    0.00    0.00    0.00    0.00    0.00   85.48
09:08:04 PM  all    8.66    0.00    6.15    0.00    0.00    0.00    0.00    0.00    0.00   85.19
09:08:05 PM  all    7.01    0.00    8.26    0.00    0.00    0.00    0.00    0.00    0.00   84.73
09:08:06 PM  all    8.65    0.00    5.76    0.00    0.00    0.00    0.00    0.00    0.00   85.59
09:08:07 PM  all    7.89    0.00    6.64    0.00    0.00    0.00    0.00    0.00    0.00   85.46
09:08:08 PM  all    7.27    0.00    7.27    0.00    0.00    0.00    0.13    0.00    0.00   85.34
09:08:09 PM  all    7.14    0.00    6.39    0.00    0.00    0.00    0.00    0.00    0.00   86.47
09:08:10 PM  all    7.27    0.00    6.89    0.00    0.00    0.00    0.00    0.00    0.00   85.84
09:08:11 PM  all    7.63    0.00    7.13    0.00    0.00    0.00    0.00    0.00    0.00   85.23
09:08:12 PM  all    7.64    0.00    6.89    0.00    0.00    0.00    0.00    0.00    0.00   85.46
09:08:13 PM  all    7.77    0.00    7.02    0.00    0.00    0.00    0.00    0.00    0.00   85.21
09:08:14 PM  all    8.40    0.00    6.39    0.00    0.00    0.00    0.00    0.00    0.00   85.21
09:08:15 PM  all    7.89    0.00    6.77    0.00    0.00    0.00    0.00    0.00    0.00   85.34
09:08:16 PM  all    7.66    0.00    6.78    0.00    0.00    0.00    0.00    0.00    0.00   85.55
09:08:17 PM  all    6.90    0.00    7.53    0.00    0.00    0.00    0.00    0.00    0.00   85.57
09:08:18 PM  all    7.15    0.00    6.27    0.00    0.00    0.00    0.00    0.00    0.00   86.57
09:08:19 PM  all    7.01    0.00    7.38    0.00    0.00    0.00    0.13    0.00    0.00   85.48
09:08:20 PM  all    8.78    0.00    5.65    0.00    0.00    0.00    0.00    0.00    0.00   85.57
09:08:21 PM  all    7.52    0.00    7.02    0.00    0.00    0.00    0.00    0.00    0.00   85.46
09:08:22 PM  all    7.52    0.00    7.52    0.00    0.00    0.00    0.00    0.00    0.00   84.96
09:08:23 PM  all    7.15    0.00    7.65    0.00    0.00    0.00    0.00    0.00    0.00   85.19
09:08:24 PM  all    7.02    0.00    7.39    0.00    0.00    0.00    0.00    0.00    0.00   85.59
09:08:25 PM  all    7.90    0.00    6.90    0.00    0.00    0.00    0.00    0.00    0.00   85.19
09:08:26 PM  all    6.78    0.00    7.65    0.00    0.00    0.00    0.00    0.00    0.00   85.57
09:08:27 PM  all    6.74    0.00    6.99    0.00    0.00    0.00    0.00    0.00    0.00   86.27
09:08:28 PM  all    7.76    0.00    7.26    0.00    0.00    0.00    0.13    0.00    0.00   84.86
09:08:29 PM  all    7.52    0.00    7.14    0.00    0.00    0.00    0.00    0.00    0.00   85.34
09:08:30 PM  all    7.78    0.00    7.03    0.00    0.00    0.00    0.00    0.00    0.00   85.19
09:08:31 PM  all    7.52    0.00    7.39    0.00    0.00    0.00    0.00    0.00    0.00   85.09
09:08:32 PM  all    7.76    0.00    7.01    0.00    0.00    0.00    0.00    0.00    0.00   85.23
09:08:33 PM  all    7.30    0.00    7.30    0.00    0.00    0.00    0.00    0.00    0.00   85.41
09:08:34 PM  all    7.74    0.00    7.12    0.00    0.00    0.00    0.00    0.00    0.00   85.14
09:08:35 PM  all    7.25    0.00    6.75    0.00    0.00    0.00    0.00    0.00    0.00   86.00
09:08:36 PM  all    6.78    0.00    7.40    0.00    0.00    0.00    0.00    0.00    0.00   85.82
09:08:37 PM  all    7.28    0.00    7.40    0.00    0.00    0.00    0.00    0.00    0.00   85.32
09:08:38 PM  all    7.64    0.00    7.39    0.00    0.00    0.00    0.00    0.00    0.00   84.96
09:08:39 PM  all    7.78    0.00    7.28    0.00    0.00    0.00    0.00    0.00    0.00   84.94
09:08:40 PM  all    7.77    0.00    7.02    0.00    0.00    0.00    0.00    0.00    0.00   85.21
09:08:41 PM  all    7.40    0.00    7.28    0.00    0.00    0.00    0.00    0.00    0.00   85.32
09:08:42 PM  all    8.26    0.00    6.38    0.00    0.00    0.00    0.13    0.00    0.00   85.23
09:08:43 PM  all    7.66    0.00    7.41    0.00    0.00    0.00    0.00    0.00    0.00   84.92
09:08:44 PM  all    6.88    0.00    6.88    0.00    0.00    0.00    0.12    0.00    0.00   86.12

09:08:44 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
09:08:45 PM  all    7.15    0.00    7.40    0.00    0.00    0.00    0.00    0.00    0.00   85.45
09:08:46 PM  all    7.76    0.00    7.38    0.00    0.00    0.00    0.00    0.00    0.00   84.86
09:08:47 PM  all    7.78    0.00    7.15    0.00    0.00    0.00    0.00    0.00    0.00   85.07
09:08:48 PM  all    7.63    0.00    7.26    0.00    0.00    0.00    0.00    0.00    0.00   85.11
09:08:49 PM  all    7.79    0.00    7.04    0.00    0.00    0.00    0.00    0.00    0.00   85.18
09:08:50 PM  all    7.54    0.00    7.54    0.00    0.00    0.00    0.00    0.00    0.00   84.92
09:08:51 PM  all    7.27    0.00    7.52    0.00    0.00    0.00    0.00    0.00    0.00   85.21
09:08:52 PM  all    7.39    0.00    7.39    0.00    0.00    0.00    0.00    0.00    0.00   85.21
09:08:53 PM  all    6.40    0.00    7.15    0.00    0.00    0.00    0.00    0.00    0.00   86.45
09:08:54 PM  all    7.77    0.00    6.89    0.00    0.00    0.00    0.00    0.00    0.00   85.34
09:08:55 PM  all    7.90    0.00    7.28    0.00    0.00    0.00    0.00    0.00    0.00   84.82
09:08:56 PM  all    8.77    0.00    6.39    0.00    0.00    0.00    0.00    0.00    0.00   84.84
09:08:57 PM  all    7.77    0.00    7.02    0.00    0.00    0.00    0.00    0.00    0.00   85.21
09:08:58 PM  all    8.02    0.00    7.14    0.00    0.00    0.00    0.13    0.00    0.00   84.71
09:08:59 PM  all    7.90    0.00    7.40    0.00    0.00    0.00    0.00    0.00    0.00   84.69
09:09:00 PM  all    7.65    0.00    7.28    0.00    0.00    0.00    0.00    0.00    0.00   85.07
09:09:01 PM  all    7.14    0.00    7.27    0.00    0.00    0.00    0.00    0.00    0.00   85.59
09:09:02 PM  all   73.93    0.00    9.90    0.13    0.00    0.00    0.00    0.00    0.00   16.04
09:09:03 PM  all   98.75    0.00    1.12    0.00    0.00    0.00    0.00    0.00    0.00    0.12
09:09:04 PM  all   98.50    0.00    1.50    0.00    0.00    0.00    0.00    0.00    0.00    0.00
09:09:05 PM  all   99.00    0.00    0.75    0.00    0.00    0.00    0.00    0.00    0.00    0.25
09:09:06 PM  all   98.87    0.00    1.13    0.00    0.00    0.00    0.00    0.00    0.00    0.00
09:09:07 PM  all   99.50    0.00    0.38    0.00    0.00    0.00    0.00    0.00    0.00    0.12
09:09:08 PM  all   99.38    0.00    0.62    0.00    0.00    0.00    0.00    0.00    0.00    0.00
09:09:09 PM  all   99.00    0.00    0.62    0.00    0.00    0.00    0.00    0.00    0.00    0.37
09:09:10 PM  all   98.50    0.00    1.50    0.00    0.00    0.00    0.00    0.00    0.00    0.00
09:09:11 PM  all   99.25    0.00    0.75    0.00    0.00    0.00    0.00    0.00    0.00    0.00
09:09:12 PM  all   99.13    0.00    0.62    0.00    0.00    0.00    0.00    0.00    0.00    0.25
09:09:13 PM  all   99.12    0.00    0.88    0.00    0.00    0.00    0.00    0.00    0.00    0.00
09:09:14 PM  all   99.25    0.00    0.75    0.00    0.00    0.00    0.00    0.00    0.00    0.00
09:09:15 PM  all   48.31    0.00    6.76    0.00    0.00    0.00    0.00    0.00    0.00   44.93
09:09:16 PM  all   31.99    0.00   10.79    0.00    0.00    0.00    0.00    0.00    0.00   57.21
09:09:17 PM  all    5.92    0.00    2.14    0.13    0.00    0.00    0.13    0.00    0.00   91.69
09:09:18 PM  all    0.88    0.00    0.63    0.00    0.00    0.00    0.00    0.00    0.00   98.50

Screen Shot 2019-05-01 at 12 00 46 PM

Screen Shot 2019-05-01 at 12 01 05 PM

szilard commented 5 years ago

lightgbm

1M

[0] Tesla V100-SXM2-16GB | 26'C,   0 % |     0 / 16160 MB |
[0] Tesla V100-SXM2-16GB | 27'C,   5 % |   422 / 16160 MB | root(412M)
[0] Tesla V100-SXM2-16GB | 27'C,   7 % |   422 / 16160 MB | root(412M)
[0] Tesla V100-SXM2-16GB | 27'C,   9 % |   422 / 16160 MB | root(412M)
[0] Tesla V100-SXM2-16GB | 27'C,  10 % |   422 / 16160 MB | root(412M)
[0] Tesla V100-SXM2-16GB | 27'C,  12 % |   422 / 16160 MB | root(412M)
[0] Tesla V100-SXM2-16GB | 27'C,  12 % |   422 / 16160 MB | root(412M)
[0] Tesla V100-SXM2-16GB | 27'C,   2 % |   422 / 16160 MB | root(412M)
[0] Tesla V100-SXM2-16GB | 27'C,   3 % |   422 / 16160 MB | root(412M)
[0] Tesla V100-SXM2-16GB | 27'C,   5 % |   422 / 16160 MB | root(412M)
[0] Tesla V100-SXM2-16GB | 27'C,   6 % |   422 / 16160 MB | root(412M)
[0] Tesla V100-SXM2-16GB | 27'C,   8 % |   422 / 16160 MB | root(412M)
[0] Tesla V100-SXM2-16GB | 27'C,  10 % |   422 / 16160 MB | root(412M)
[0] Tesla V100-SXM2-16GB | 27'C,  11 % |   422 / 16160 MB | root(412M)
[0] Tesla V100-SXM2-16GB | 27'C,   2 % |   422 / 16160 MB | root(412M)
[0] Tesla V100-SXM2-16GB | 27'C,   0 % |   422 / 16160 MB | root(412M)
[0] Tesla V100-SXM2-16GB | 27'C,   0 % |     0 / 16160 MB |

02:07:04 PM  all    2.13    0.00    0.25    0.00    0.00    0.00    0.00    0.00    0.00   97.62
02:07:05 PM  all   45.54    0.00    1.66    0.00    0.00    0.00    0.00    0.00    0.00   52.81
02:07:06 PM  all   95.22    0.00    1.76    0.00    0.00    0.00    0.00    0.00    0.00    3.02
02:07:07 PM  all   97.87    0.00    1.38    0.00    0.00    0.00    0.13    0.00    0.00    0.63
02:07:08 PM  all   97.37    0.00    1.00    0.00    0.00    0.00    0.00    0.00    0.00    1.63
02:07:09 PM  all   97.62    0.00    1.25    0.00    0.00    0.00    0.00    0.00    0.00    1.13
02:07:10 PM  all   97.62    0.00    1.38    0.00    0.00    0.00    0.00    0.00    0.00    1.00
02:07:11 PM  all   97.62    0.00    1.38    0.00    0.00    0.00    0.00    0.00    0.00    1.00
02:07:12 PM  all   97.61    0.00    0.75    0.00    0.00    0.00    0.13    0.00    0.00    1.51
02:07:13 PM  all   97.87    0.00    1.51    0.00    0.00    0.00    0.00    0.00    0.00    0.63
02:07:14 PM  all   98.49    0.00    0.75    0.00    0.00    0.00    0.00    0.00    0.00    0.75
02:07:15 PM  all   97.22    0.00    1.77    0.00    0.00    0.00    0.00    0.00    0.00    1.01
02:07:16 PM  all   98.24    0.00    1.13    0.00    0.00    0.00    0.00    0.00    0.00    0.63
02:07:17 PM  all   97.99    0.00    1.50    0.00    0.00    0.00    0.00    0.00    0.00    0.50
02:07:18 PM  all   98.22    0.00    1.02    0.00    0.00    0.00    0.00    0.00    0.00    0.76
02:07:19 PM  all   98.12    0.00    1.13    0.00    0.00    0.00    0.00    0.00    0.00    0.75
02:07:20 PM  all   97.60    0.00    1.26    0.00    0.00    0.00    0.13    0.00    0.00    1.01
02:07:21 PM  all   97.37    0.00    1.75    0.00    0.00    0.00    0.00    0.00    0.00    0.88
02:07:22 PM  all   59.32    0.00    1.00    0.00    0.00    0.00    0.00    0.00    0.00   39.67
02:07:23 PM  all    2.02    0.00    1.26    0.25    0.00    0.00    0.00    0.00    0.00   96.47

10M:

[0] Tesla V100-SXM2-16GB | 29'C,   0 % |     0 / 16160 MB |
[0] Tesla V100-SXM2-16GB | 30'C,   5 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 30'C,  10 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 30'C,   6 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 30'C,   8 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 30'C,   6 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 30'C,   6 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 30'C,   7 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 30'C,   8 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 30'C,   6 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 30'C,   1 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   2 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 30'C,   2 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   2 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 30'C,   4 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   3 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   8 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   5 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   6 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   7 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   6 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,  11 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   8 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   8 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   1 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   6 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   1 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   1 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   2 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   3 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   4 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   4 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   4 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   4 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   6 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   7 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   3 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   6 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   7 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   1 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   0 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   2 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   2 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   6 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   5 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   6 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   7 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   7 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   7 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   7 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   7 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   9 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   9 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   0 % |   662 / 16160 MB | root(652M)
[0] Tesla V100-SXM2-16GB | 31'C,   0 % |     0 / 16160 MB |

06:54:58 PM  all   90.21    0.00    0.25    0.00    0.00    0.00    0.00    0.00    0.00    9.54
06:54:59 PM  all   93.39    0.00    0.62    0.00    0.00    0.00    0.00    0.00    0.00    5.99
06:55:00 PM  all   93.61    0.00    0.63    0.00    0.00    0.00    0.00    0.00    0.00    5.76
06:55:01 PM  all   88.71    0.00    1.25    0.00    0.00    0.00    0.00    0.00    0.00   10.04
06:55:02 PM  all   92.67    0.00    0.87    0.00    0.00    0.00    0.00    0.00    0.00    6.46
06:55:03 PM  all   94.49    0.00    0.75    0.00    0.00    0.00    0.00    0.00    0.00    4.76
06:55:04 PM  all   93.73    0.00    0.75    0.00    0.00    0.00    0.00    0.00    0.00    5.52
06:55:05 PM  all   92.60    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00    6.90
06:55:06 PM  all   93.94    0.00    0.63    0.00    0.00    0.00    0.00    0.00    0.00    5.43
06:55:07 PM  all   88.57    0.00    1.01    0.00    0.00    0.00    0.00    0.00    0.00   10.43
06:55:08 PM  all   95.21    0.00    0.88    0.00    0.00    0.00    0.00    0.00    0.00    3.91
06:55:09 PM  all   91.59    0.00    0.63    0.00    0.00    0.00    0.13    0.00    0.00    7.65
06:55:10 PM  all   91.48    0.00    0.75    0.00    0.00    0.00    0.00    0.00    0.00    7.77
06:55:11 PM  all   95.64    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00    3.86
06:55:12 PM  all   91.91    0.00    0.63    0.00    0.00    0.00    0.00    0.00    0.00    7.46
06:55:13 PM  all   94.70    0.00    0.51    0.00    0.00    0.00    0.00    0.00    0.00    4.80
06:55:14 PM  all   89.56    0.00    0.75    0.00    0.00    0.00    0.00    0.00    0.00    9.69
06:55:15 PM  all   94.10    0.00    0.88    0.00    0.00    0.00    0.00    0.00    0.00    5.02
06:55:16 PM  all   91.01    0.00    0.75    0.00    0.00    0.00    0.00    0.00    0.00    8.24

06:55:16 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
06:55:17 PM  all   95.21    0.00    1.01    0.00    0.00    0.00    0.00    0.00    0.00    3.78
06:55:18 PM  all   91.88    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00    7.62
06:55:19 PM  all   95.38    0.00    0.62    0.00    0.00    0.00    0.00    0.00    0.00    4.00
06:55:20 PM  all   94.60    0.00    1.01    0.00    0.00    0.00    0.00    0.00    0.00    4.40
06:55:21 PM  all   94.94    0.00    0.63    0.00    0.00    0.00    0.00    0.00    0.00    4.43
06:55:22 PM  all   95.68    0.00    0.89    0.00    0.00    0.00    0.00    0.00    0.00    3.43
06:55:23 PM  all   92.89    0.00    0.63    0.00    0.00    0.00    0.00    0.00    0.00    6.47
06:55:24 PM  all   94.95    0.00    0.63    0.00    0.00    0.00    0.00    0.00    0.00    4.42
06:55:25 PM  all   95.11    0.00    0.50    0.00    0.00    0.00    0.00    0.00    0.00    4.39
06:55:26 PM  all   96.62    0.00    0.62    0.00    0.00    0.00    0.12    0.00    0.00    2.62
06:55:27 PM  all   93.24    0.00    0.88    0.00    0.00    0.00    0.00    0.00    0.00    5.88
06:55:28 PM  all   94.97    0.00    0.75    0.00    0.00    0.00    0.00    0.00    0.00    4.27
06:55:29 PM  all   93.07    0.00    0.63    0.00    0.00    0.00    0.00    0.00    0.00    6.30
06:55:30 PM  all   94.59    0.00    0.75    0.00    0.00    0.00    0.00    0.00    0.00    4.65
06:55:31 PM  all   94.82    0.00    0.51    0.00    0.00    0.00    0.00    0.00    0.00    4.68
06:55:32 PM  all   94.37    0.00    0.75    0.00    0.00    0.00    0.00    0.00    0.00    4.88
06:55:33 PM  all   95.96    0.00    0.76    0.00    0.00    0.00    0.00    0.00    0.00    3.28
06:55:34 PM  all   93.72    0.00    0.63    0.00    0.00    0.00    0.00    0.00    0.00    5.65
06:55:35 PM  all   92.93    0.00    0.63    0.00    0.00    0.00    0.00    0.00    0.00    6.44
06:55:36 PM  all   94.24    0.00    1.25    0.00    0.00    0.00    0.00    0.00    0.00    4.51
06:55:37 PM  all   95.08    0.00    0.63    0.00    0.00    0.00    0.00    0.00    0.00    4.29
06:55:38 PM  all   92.99    0.00    0.75    0.00    0.00    0.00    0.00    0.00    0.00    6.26
06:55:39 PM  all   94.62    0.00    0.63    0.00    0.00    0.00    0.00    0.00    0.00    4.76
06:55:40 PM  all   94.93    0.00    0.25    0.00    0.00    0.00    0.00    0.00    0.00    4.82
06:55:41 PM  all   93.46    0.00    1.01    0.00    0.00    0.00    0.00    0.00    0.00    5.53
06:55:42 PM  all   92.29    0.00    0.88    0.00    0.00    0.00    0.00    0.00    0.00    6.83
06:55:43 PM  all   94.00    0.00    0.88    0.00    0.00    0.00    0.12    0.00    0.00    5.00
06:55:44 PM  all   54.76    0.00    2.76    0.00    0.00    0.00    0.00    0.00    0.00   42.48
06:55:45 PM  all    1.64    0.00    1.51    0.13    0.00    0.00    0.00    0.00    0.00   96.73

szilard commented 5 years ago

lightgbm 1M again:

15.48 0.7660041

Screen Shot 2019-05-12 at 7 56 16 AM

nthread = 1

21.901 0.7661341

Screen Shot 2019-05-12 at 7 59 02 AM

szilard commented 5 years ago

Force matrix construct before/outside training with:

lgb.Dataset.construct(dlgb_train)

though this crashes, unless this hack https://github.com/szilard/GBM-perf/issues/12#issuecomment-491610532

dlgb_train <- lgb.Dataset( ... , device = "gpu")

szilard commented 5 years ago

default threads (all cores):

construct: user system elapsed 1.904 0.008 0.472

16.11 0.7660714

nthread = 1

construct: user system elapsed 1.956 0.004 0.486

20.178 0.7661254

szilard / GBM-perf

GPU utilization patterns #11