Add tensorflow profiler

[djshen]

If we pass --profile options to train.py, the profiler is enabled and output the statistics for each operation.

The StatSummarizer class is originally used in the TensorFlow Model Benchmark Tool, but it is not exposed in any public python API. I'm not sure whether it would be removed in the future.

============================== Run Order ==============================
                     [node type]                  [start]         [first]        [avg ms]            [%]          [cdf%]          [mem KB]      [times called]  [Name]
                            NoOp                   -0.161           0.012           0.021         0.018%          0.018%             0.000              1       _SOURCE
                     VarHandleOp                   -0.051           0.089           0.044         0.038%          0.057%             0.000              1       fully_connected/fully_connected/bias
                     VarHandleOp                   -0.068           0.100           0.035         0.030%          0.087%             0.000              1       conv3/conv2d_2/kernel
                     VarHandleOp                   -0.018           0.032           0.040         0.035%          0.122%             0.000              1       conv1/conv2d/kernel/Momentum
                     VarHandleOp                   -0.029           0.043           0.040         0.035%          0.157%             0.000              1       conv1/conv2d/bias/Momentum
                     VarHandleOp                    0.026           0.013           0.032         0.028%          0.184%             0.000              1       conv2/conv2d_1/bias/Momentum
                     VarHandleOp                    0.006           0.038           0.030         0.026%          0.210%             0.000              1       conv2/conv2d_1/kernel
                     VarHandleOp                    0.044           0.006           0.025         0.022%          0.232%             0.000              1       conv3/conv2d_2/bias/Momentum
                  ReadVariableOp                    0.020           0.048           0.031         0.027%          0.259%             0.000              1       fully_connected/fully_connected/BiasAdd/ReadVariableOp
                     VarHandleOp                    0.017           0.035           0.032         0.028%          0.287%             0.000              1       conv2/conv2d_1/bias
                  ReadVariableOp                    0.026           0.054           0.048         0.041%          0.328%             0.000              1       Pow_8/ReadVariableOp
                  ReadVariableOp                    0.064           0.014           0.033         0.029%          0.357%             0.000              1       Pow_3/ReadVariableOp
                     VarHandleOp                    0.052           0.007           0.028         0.024%          0.381%             0.000              1       fully_connected/fully_connected/kernel
                     VarHandleOp                   -0.006           0.080           0.042         0.036%          0.417%             0.000              1       fully_connected/fully_connected/bias/Momentum
                     VarHandleOp                    0.001           0.078           0.031         0.027%          0.444%             0.000              1       conv1/conv2d/kernel
                  ReadVariableOp                    0.004           0.049           0.045         0.039%          0.483%             0.000              1       Pow_5/ReadVariableOp
                     VarHandleOp                    0.031           0.037           0.029         0.025%          0.508%             0.000              1       conv2/conv2d_1/kernel/Momentum
                  ReadVariableOp                    0.064           0.017           0.024         0.021%          0.529%             0.000              1       conv2/conv2d_1/Conv2D/ReadVariableOp
                     VarHandleOp                    0.038           0.048           0.032         0.028%          0.557%             0.000              1       global_step
                  ReadVariableOp                    0.073           0.015           0.024         0.021%          0.577%             0.000              1       conv2/conv2d_1/BiasAdd/ReadVariableOp
                          Square                    0.086           0.030           0.028         0.024%          0.601%             0.040              1       Pow_8
                  ReadVariableOp                    0.060           0.015           0.030         0.026%          0.627%             0.000              1       Pow_1/ReadVariableOp
                  ReadVariableOp                    0.072           0.039           0.029         0.025%          0.652%             0.000              1       Pow_4/ReadVariableOp
                  ReadVariableOp                    0.061           0.017           0.028         0.025%          0.677%             0.000              1       conv1/conv2d/Conv2D/ReadVariableOp
                  ReadVariableOp                    0.089           0.013           0.014         0.012%          0.689%             0.000              1       global_step/Read/ReadVariableOp
                     VarHandleOp                   -0.039           0.152           0.047         0.041%          0.730%             0.000              1       fully_connected/fully_connected/kernel/Momentum
                         _Retval                    0.113           0.004           0.011         0.009%          0.740%             0.000              1       _retval_global_step/Read/ReadVariableOp_0_1
                     VarHandleOp                    0.027           0.105           0.038         0.033%          0.772%             0.000              1       conv3/conv2d_2/bias
                  ReadVariableOp                    0.101           0.055           0.025         0.021%          0.794%             0.000              1       Pow_7/ReadVariableOp
                          Square                    0.107           0.031           0.055         0.048%          0.842%             3.200              1       Pow_1
                  ReadVariableOp                   -0.006           0.123           0.037         0.032%          0.874%             0.000              1       conv3/conv2d_2/Conv2D/ReadVariableOp
                  ReadVariableOp                    0.084           0.010           0.025         0.022%          0.895%             0.000              1       Pow_6/ReadVariableOp
                           Const                    0.032           0.092           0.017         0.015%          0.910%             0.000              1       mul/x
                            _Arg                    0.065           0.014           0.021         0.018%          0.928%             0.000              1       _arg_main_params/input_of_net_0_1
                  ReadVariableOp                    0.105           0.097           0.024         0.021%          0.949%             0.000              1       fully_connected/fully_connected/MatMul/ReadVariableOp
                          Square                    0.120           0.027           0.022         0.019%          0.968%             0.256              1       Pow_6
                  ReadVariableOp                    0.084           0.016           0.018         0.016%          0.984%             0.000              1       conv3/conv2d_2/BiasAdd/ReadVariableOp
                            _Arg                    0.098           0.006           0.011         0.010%          0.993%             0.000              1       _arg_main_params/labels_0_2
                          Square                    0.111           0.074           0.023         0.020%          1.013%             0.256              1       Pow_4
                          Square                    0.136           0.053           0.062         0.054%          1.067%            23.040              1       Pow_7
                           Const                    0.119           0.006           0.009         0.008%          1.075%             0.000              1       gradients/Pow_2_grad/ones_like/Const
                           Const                    0.132           0.005           0.010         0.009%          1.084%             0.000              1       Pow_1/y
                          Square                    0.118           0.143           0.081         0.070%          1.154%            73.728              1       Pow_3
                           Const                    0.149           0.005           0.007         0.006%          1.160%             0.000              1       gradients/Pow_7_grad/Sum/reduction_indices
                     VarHandleOp                   -0.007           0.241           0.038         0.033%          1.193%             0.000              1       conv1/conv2d/bias
                           Const                    0.162           0.003           0.006         0.005%          1.199%             0.000              1       gradients/Sum_grad/Reshape/shape
                     VarHandleOp                    0.014           0.242           0.026         0.022%          1.221%             0.000              1       conv3/conv2d_2/kernel/Momentum
                           Const                    0.172           0.005           0.007         0.006%          1.227%             0.000              1       gradients/Pow_2_grad/Sum/reduction_indices
                  ReadVariableOp                    0.060           0.007           0.027         0.023%          1.250%             0.000              1       Pow_2/ReadVariableOp
                  ReadVariableOp                    0.059           0.012           0.032         0.028%          1.278%             0.000              1       conv1/conv2d/BiasAdd/ReadVariableOp
                          Square                    0.106           0.015           0.023         0.020%          1.298%             0.128              1       Pow_2
                           Const                    0.200           0.006           0.005         0.004%          1.303%             0.000              1       gradients/Pow_grad/Shape_1
                           Const                    0.209           0.005           0.006         0.005%          1.307%             0.000              1       strided_slice/stack_1
                           Const                    0.218           0.006           0.005         0.005%          1.312%             0.000              1       gradients/Pow_2_grad/Greater/y
                           Const                    0.228           0.006           0.005         0.004%          1.316%             0.000              1       fully_connected/Reshape/shape
                          Square                    0.063           0.221           0.147         0.128%          1.444%           147.456              1       Pow_5
                           Const                    0.237           0.006           0.004         0.004%          1.448%             0.000              1       gradients/Pow_1_grad/Sum/reduction_indices
                             Sum                    0.207           0.108           0.026         0.022%          1.470%             0.004              1       Sum_8
                             Sum                    0.199           0.119           0.025         0.022%          1.492%             0.004              1       Sum_2
                             Sum                    0.199           0.143           0.027         0.024%          1.516%             0.004              1       Sum_4
                             Sum                    0.204           0.130           0.027         0.024%          1.540%             0.004              1       Sum_6
                             Sum                    0.220           0.177           0.025         0.022%          1.561%             0.004              1       Sum_7
                            _Arg                    0.259           0.007           0.006         0.005%          1.567%             0.000              1       _arg_learning_rate_0_0
                             Sum                    0.313           0.030           0.022         0.019%          1.585%             0.004              1       Sum_1
                             Sum                    0.299           0.026           1.186         1.031%          2.617%             0.004              1       Sum_5
                           Shape                    0.270           0.023           0.014         0.012%          2.629%             0.016              1       Shape
                             Sum                    0.324           0.053           1.101         0.957%          3.586%             0.004              1       Sum_3
                           Const                    0.217           0.005           0.013         0.011%          3.597%             0.000              1       ConstantFolding/gradients/conv1/conv2d/Conv2D_grad/ShapeN-matshapes-1
                           Shape                    0.314           0.006           0.021         0.019%          3.616%             0.008              1       gradients/sub_grad/Shape_1
                            AddN                    2.574           0.027           0.033         0.029%          3.644%             0.000              1       ArithmeticOptimizer/AddOpsRewrite_add_7
                           Const                    2.635           0.006           0.007         0.006%          3.651%             0.000              1       gradients/Pow_7_grad/mul
                             Mul                    2.671           0.015           0.024         0.021%          3.671%             0.000              1       mul
                           Const                    2.669           0.008           0.005         0.004%          3.676%             0.000              1       gradients/Pow_2_grad/mul
                           Const                    2.694           0.005           0.006         0.006%          3.681%             0.000              1       gradients/Pow_1_grad/mul
                    StridedSlice                    0.307           0.065           0.032         0.028%          3.709%             0.004              1       strided_slice
                             Mul                    2.735           0.013           0.026         0.022%          3.732%             0.128              1       gradients/Pow_2_grad/mul_1
                           Const                    2.728           0.005           0.006         0.005%          3.737%             0.000              1       gradients/Pow_4_grad/mul
                             Mul                    2.760           0.016           0.032         0.027%          3.764%             3.200              1       gradients/Pow_1_grad/mul_1
                            Cast                    0.347           0.010           0.013         0.012%          3.776%             0.004              1       Cast
                             Mul                    2.794           0.011           0.018         0.015%          3.791%             0.256              1       gradients/Pow_4_grad/mul_1
                           Const                    2.764           0.005           0.008         0.007%          3.798%             0.000              1       gradients/Pow_3_grad/mul
                             Mul                    2.699           0.056           0.065         0.057%          3.855%            23.040              1       gradients/Pow_7_grad/mul_1
                      Reciprocal                    0.367           0.015           0.012         0.010%          3.865%             0.004              1       gradients/truediv_grad/RealDiv
                           Const                    2.799           0.005           0.007         0.006%          3.871%             0.000              1       gradients/Pow_5_grad/mul
                           Const                    2.836           0.004           0.006         0.005%          3.876%             0.000              1       gradients/Pow_8_grad/mul
                             Mul                    2.808           0.031           0.019         0.016%          3.892%             0.256              1       gradients/Pow_6_grad/mul_1
                             Mul                    2.832           0.031           0.115         0.100%          3.992%            73.728              1       gradients/Pow_3_grad/mul_1
                             Mul                    2.846           0.019           0.022         0.019%          4.012%             0.040              1       gradients/Pow_8_grad/mul_1
                             Mul                    2.825           0.198           0.263         0.229%          4.240%           147.456              1       gradients/Pow_5_grad/mul_1
                    _FusedConv2D                    0.147          39.243           6.863         5.966%         10.206%         25690.112              1       conv1/conv2d/Relu
                         MaxPool                    7.028           2.236           2.548         2.215%         12.421%          6422.528              1       conv1/max_pooling2d/MaxPool
                          ShapeN                    9.595           0.033           0.020         0.018%         12.439%             0.032              1       gradients/conv2/conv2d_1/Conv2D_grad/ShapeN
                           Const                    9.619           0.007           0.005         0.005%         12.444%             0.000              1       ConstantFolding/gradients/conv2/conv2d_1/Conv2D_grad/ShapeN-matshapes-1
                    _FusedConv2D                    9.613           4.598           5.308         4.614%         17.058%         12845.056              1       conv2/conv2d_1/Relu
                         MaxPool                   14.937           0.943           1.291         1.123%         18.180%          3211.264              1       conv2/max_pooling2d_1/MaxPool
                          ShapeN                   16.243           0.011           0.016         0.014%         18.194%             0.032              1       gradients/conv3/conv2d_2/Conv2D_grad/ShapeN
                           Const                   16.262           0.003           0.004         0.003%         18.197%             0.000              1       ConstantFolding/gradients/conv3/conv2d_2/Conv2D_grad/ShapeN-matshapes-1
                    _FusedConv2D                   16.255           2.034           2.407         2.092%         20.289%          3211.264              1       conv3/conv2d_2/Relu
                         MaxPool                   18.673           0.273           0.312         0.271%         20.560%           589.824              1       conv3/max_pooling2d_2/MaxPool
                           Shape                   18.994           0.022           0.008         0.007%         20.568%             0.016              1       gradients/fully_connected/Reshape_grad/Shape
                         Reshape                   19.006           0.011           0.008         0.007%         20.575%             0.000              1       fully_connected/Reshape
                    _FusedMatMul                   19.017           0.194           0.150         0.130%         20.705%            10.240              1       fully_connected/fully_connected/BiasAdd
                           Shape                   19.177           0.008           0.007         0.006%         20.711%             0.008              1       gradients/sub_grad/Shape
           BroadcastGradientArgs                   19.188           0.013           0.009         0.008%         20.719%             0.000              1       gradients/sub_grad/BroadcastGradientArgs
                             Sub                   19.190           0.013           0.018         0.016%         20.735%             0.000              1       sub
                           Shape                   19.224           0.006           0.008         0.007%         20.742%             0.008              1       gradients/Pow_grad/ones_like/Shape
           BroadcastGradientArgs                   19.236           0.008           0.010         0.009%         20.751%             0.008              1       gradients/Pow_grad/BroadcastGradientArgs
                          Square                   19.238           0.043           0.036         0.031%         20.782%            10.240              1       Pow
                           Shape                   19.286           0.009           0.009         0.008%         20.790%             0.008              1       gradients/Sum_grad/Shape
                             Sum                   19.289           0.019           0.025         0.022%         20.812%             0.004              1       Sum
                         Reshape                   19.323           0.005           0.006         0.005%         20.817%             0.000              1       gradients/Sum_grad/Reshape
                             Pow                   19.218           0.127           0.082         0.071%         20.888%            10.240              1       gradients/Pow_grad/Pow
                         RealDiv                   19.325           0.027           0.014         0.012%         20.900%             0.000              1       truediv
                            Tile                   19.334           0.024           0.028         0.025%         20.925%            10.240              1       gradients/Sum_grad/Tile
                             Mul                   19.366           0.008           0.009         0.008%         20.932%             0.000              1       gradients/Pow_grad/mul
                             Mul                   19.378           0.007           0.007         0.006%         20.939%             0.000              1       gradients/Pow_grad/mul_1
                           AddV2                   19.342           0.021           0.007         0.006%         20.945%             0.000              1       add_8
                             Sum                   19.388           0.004           0.008         0.007%         20.952%             0.000              1       gradients/Pow_grad/Sum
                         Reshape                   19.399           0.003           0.003         0.003%         20.955%             0.000              1       gradients/Pow_grad/Reshape
                         _Retval                   19.353           0.004           0.003         0.002%         20.957%             0.000              1       _retval_add_8_0_0
                             Sum                   19.405           0.003           0.004         0.003%         20.960%             0.000              1       gradients/sub_grad/Sum
                         Reshape                   19.411           0.002           0.003         0.003%         20.963%             0.000              1       gradients/sub_grad/Reshape
                     BiasAddGrad                   19.418           0.037           0.016         0.014%         20.977%             0.040              1       gradients/fully_connected/fully_connected/BiasAdd_grad/BiasAddGrad
                            AddN                   19.449           0.021           0.012         0.011%         20.988%             0.000              1       gradients/AddN
           ResourceApplyMomentum                   19.466           0.111           0.033         0.028%         21.016%             0.000              1       Momentum/update_fully_connected/fully_connected/bias/ResourceApplyMomentum
                          MatMul                   19.446           0.315           0.470         0.408%         21.425%            23.040              1       gradients/fully_connected/fully_connected/MatMul_grad/MatMul_1
                          MatMul                   19.458           0.402           0.334         0.290%         21.715%           589.824              1       gradients/fully_connected/fully_connected/MatMul_grad/MatMul
                         Reshape                   20.018           0.007           0.007         0.006%         21.721%             0.000              1       gradients/fully_connected/Reshape_grad/Reshape
                            AddN                   20.027           0.017           0.019         0.017%         21.737%             0.000              1       gradients/AddN_1
           ResourceApplyMomentum                   20.055           0.050           0.131         0.113%         21.851%             0.000              1       Momentum/update_fully_connected/fully_connected/kernel/ResourceApplyMomentum
                     MaxPoolGrad                   20.029           0.806           1.165         1.013%         22.863%          4390.912              1       gradients/conv3/max_pooling2d_2/MaxPool_grad/MaxPoolGrad
                        ReluGrad                   21.205           0.350           0.428         0.372%         23.236%             0.000              1       gradients/conv3/conv2d_2/Relu_grad/ReluGrad
                     BiasAddGrad                   21.638           0.228           0.285         0.248%         23.484%             0.256              1       gradients/conv3/conv2d_2/BiasAdd_grad/BiasAddGrad
                            AddN                   21.939           0.046           0.015         0.013%         23.497%             0.000              1       gradients/AddN_2
           ResourceApplyMomentum                   21.959           0.025           0.028         0.024%         23.521%             0.000              1       Momentum/update_conv3/conv2d_2/bias/ResourceApplyMomentum
             Conv2DBackpropInput                   21.937           6.871           8.605         7.481%         31.002%         16307.200              1       gradients/conv3/conv2d_2/Conv2D_grad/Conv2DBackpropInput
            Conv2DBackpropFilter                   21.945          12.851          13.035        11.332%         42.333%         13243.392              1       gradients/conv3/conv2d_2/Conv2D_grad/Conv2DBackpropFilter
                            AddN                   35.051           0.052           0.054         0.047%         42.380%             0.000              1       gradients/AddN_3
           ResourceApplyMomentum                   35.110           0.564           0.720         0.626%         43.007%             0.000              1       Momentum/update_conv3/conv2d_2/kernel/ResourceApplyMomentum
                     MaxPoolGrad                   35.071           3.223           3.666         3.187%         46.193%         19267.584              1       gradients/conv2/max_pooling2d_1/MaxPool_grad/MaxPoolGrad
                        ReluGrad                   38.758           1.798           1.801         1.566%         47.759%             0.000              1       gradients/conv2/conv2d_1/Relu_grad/ReluGrad
                     BiasAddGrad                   40.565           0.980           1.006         0.874%         48.633%             0.256              1       gradients/conv2/conv2d_1/BiasAdd_grad/BiasAddGrad
                            AddN                   41.601           0.020           0.041         0.035%         48.668%             0.000              1       gradients/AddN_4
           ResourceApplyMomentum                   41.650           0.042           0.081         0.071%         48.739%             0.000              1       Momentum/update_conv2/conv2d_1/bias/ResourceApplyMomentum
             Conv2DBackpropInput                   41.589          13.151          15.822        13.755%         62.494%         26743.808              1       gradients/conv2/conv2d_1/Conv2D_grad/Conv2DBackpropInput
            Conv2DBackpropFilter                   41.612          23.332          22.638        19.680%         82.174%         20395.008              1       gradients/conv2/conv2d_1/Conv2D_grad/Conv2DBackpropFilter
                            AddN                   64.289           0.036           0.041         0.036%         82.210%             0.000              1       gradients/AddN_5
           ResourceApplyMomentum                   64.335           0.086           0.243         0.211%         82.421%             0.000              1       Momentum/update_conv2/conv2d_1/kernel/ResourceApplyMomentum
                     MaxPoolGrad                   64.333           6.798           6.705         5.829%         88.250%         38535.168              1       gradients/conv1/max_pooling2d/MaxPool_grad/MaxPoolGrad
                        ReluGrad                   71.076           6.099           4.421         3.843%         92.093%             0.000              1       gradients/conv1/conv2d/Relu_grad/ReluGrad
                     BiasAddGrad                   75.530           2.033           2.377         2.067%         94.160%             0.128              1       gradients/conv1/conv2d/BiasAdd_grad/BiasAddGrad
                            AddN                   77.954           0.020           0.029         0.025%         94.185%             0.000              1       gradients/AddN_6
           ResourceApplyMomentum                   77.990           0.038           0.036         0.032%         94.217%             0.000              1       Momentum/update_conv1/conv2d/bias/ResourceApplyMomentum
            Conv2DBackpropFilter                   77.924           7.731           6.572         5.713%         99.930%         13566.400              1       gradients/conv1/conv2d/Conv2D_grad/Conv2DBackpropFilter
                            AddN                   84.522           0.016           0.021         0.018%         99.948%             0.000              1       gradients/AddN_7
           ResourceApplyMomentum                   84.546           0.035           0.033         0.028%         99.976%             0.000              1       Momentum/update_conv1/conv2d/kernel/ResourceApplyMomentum
                           Const                   84.583           0.003           0.004         0.004%         99.980%             0.000              1       Momentum/Const
             AssignAddVariableOp                   84.589           0.014           0.023         0.020%        100.000%             0.004              1       Momentum

============================== Top by Computation Time ==============================
                     [node type]                  [start]         [first]        [avg ms]            [%]          [cdf%]          [mem KB]      [times called]  [Name]
            Conv2DBackpropFilter                   41.612          23.332          22.638        19.680%         19.680%         20395.008              1       gradients/conv2/conv2d_1/Conv2D_grad/Conv2DBackpropFilter
             Conv2DBackpropInput                   41.589          13.151          15.822        13.755%         33.435%         26743.808              1       gradients/conv2/conv2d_1/Conv2D_grad/Conv2DBackpropInput
            Conv2DBackpropFilter                   21.945          12.851          13.035        11.332%         44.767%         13243.392              1       gradients/conv3/conv2d_2/Conv2D_grad/Conv2DBackpropFilter
             Conv2DBackpropInput                   21.937           6.871           8.605         7.481%         52.247%         16307.200              1       gradients/conv3/conv2d_2/Conv2D_grad/Conv2DBackpropInput
                    _FusedConv2D                    0.147          39.243           6.863         5.966%         58.213%         25690.112              1       conv1/conv2d/Relu
                     MaxPoolGrad                   64.333           6.798           6.705         5.829%         64.042%         38535.168              1       gradients/conv1/max_pooling2d/MaxPool_grad/MaxPoolGrad
            Conv2DBackpropFilter                   77.924           7.731           6.572         5.713%         69.755%         13566.400              1       gradients/conv1/conv2d/Conv2D_grad/Conv2DBackpropFilter
                    _FusedConv2D                    9.613           4.598           5.308         4.614%         74.370%         12845.056              1       conv2/conv2d_1/Relu
                        ReluGrad                   71.076           6.099           4.421         3.843%         78.213%             0.000              1       gradients/conv1/conv2d/Relu_grad/ReluGrad
                     MaxPoolGrad                   35.071           3.223           3.666         3.187%         81.399%         19267.584              1       gradients/conv2/max_pooling2d_1/MaxPool_grad/MaxPoolGrad

============================== Top by Memory Use ==============================
                     [node type]                  [start]         [first]        [avg ms]            [%]          [cdf%]          [mem KB]      [times called]  [Name]
                     MaxPoolGrad                   64.333           6.798           6.705         5.829%          5.829%         38535.168              1       gradients/conv1/max_pooling2d/MaxPool_grad/MaxPoolGrad
             Conv2DBackpropInput                   41.589          13.151          15.822        13.755%         19.584%         26743.808              1       gradients/conv2/conv2d_1/Conv2D_grad/Conv2DBackpropInput
                    _FusedConv2D                    0.147          39.243           6.863         5.966%         25.550%         25690.112              1       conv1/conv2d/Relu
            Conv2DBackpropFilter                   41.612          23.332          22.638        19.680%         45.230%         20395.008              1       gradients/conv2/conv2d_1/Conv2D_grad/Conv2DBackpropFilter
                     MaxPoolGrad                   35.071           3.223           3.666         3.187%         48.417%         19267.584              1       gradients/conv2/max_pooling2d_1/MaxPool_grad/MaxPoolGrad
             Conv2DBackpropInput                   21.937           6.871           8.605         7.481%         55.897%         16307.200              1       gradients/conv3/conv2d_2/Conv2D_grad/Conv2DBackpropInput
            Conv2DBackpropFilter                   77.924           7.731           6.572         5.713%         61.610%         13566.400              1       gradients/conv1/conv2d/Conv2D_grad/Conv2DBackpropFilter
            Conv2DBackpropFilter                   21.945          12.851          13.035        11.332%         72.942%         13243.392              1       gradients/conv3/conv2d_2/Conv2D_grad/Conv2DBackpropFilter
                    _FusedConv2D                    9.613           4.598           5.308         4.614%         77.556%         12845.056              1       conv2/conv2d_1/Relu
                         MaxPool                    7.028           2.236           2.548         2.215%         79.771%          6422.528              1       conv1/max_pooling2d/MaxPool

Number of nodes executed: 158
============================== Summary by node type ==============================
                     [Node type]          [count]         [avg ms]          [avg %]         [cdf %]       [mem KB]      [times called]
            Conv2DBackpropFilter                3           42.242          36.750%         36.750%      47204.801              3
             Conv2DBackpropInput                2           24.426          21.250%         58.000%      43051.008              2
                    _FusedConv2D                3           14.575          12.680%         70.681%      41746.434              3
                     MaxPoolGrad                3           11.534          10.034%         80.715%      62193.664              3
                        ReluGrad                3            6.648           5.784%         86.499%          0.000              3
                         MaxPool                3            4.149           3.610%         90.108%      10223.616              3
                     BiasAddGrad                4            3.682           3.203%         93.312%          0.680              4
                             Sum               11            2.471           2.150%         95.461%          0.036             11
           ResourceApplyMomentum                8            1.300           1.131%         96.592%          0.000              8
                          MatMul                2            0.802           0.698%         97.290%        612.864              2
                             Mul               11            0.594           0.517%         97.807%        248.104             11
                     VarHandleOp               17            0.578           0.503%         98.310%          0.000             17
                  ReadVariableOp               17            0.481           0.418%         98.728%          0.000             17
                          Square                9            0.472           0.411%         99.139%        258.344              9
                            AddN                9            0.259           0.225%         99.364%          0.000              9
                    _FusedMatMul                1            0.149           0.130%         99.494%         10.240              1
                           Const               22            0.141           0.123%         99.616%          0.000             22
                             Pow                1            0.082           0.071%         99.688%         10.240              1
                           Shape                6            0.065           0.057%         99.744%          0.064              6
                            _Arg                3            0.037           0.032%         99.776%          0.000              3
                          ShapeN                2            0.035           0.030%         99.807%          0.064              2
                    StridedSlice                1            0.032           0.028%         99.835%          0.004              1
                            Tile                1            0.028           0.024%         99.859%         10.240              1
                         Reshape                5            0.025           0.022%         99.881%          0.000              5
             AssignAddVariableOp                1            0.023           0.020%         99.901%          0.004              1
                            NoOp                1            0.021           0.018%         99.919%          0.000              1
           BroadcastGradientArgs                2            0.019           0.017%         99.936%          0.008              2
                             Sub                1            0.018           0.016%         99.951%          0.000              1
                         RealDiv                1            0.013           0.011%         99.963%          0.000              1
                            Cast                1            0.013           0.011%         99.974%          0.004              1
                         _Retval                2            0.012           0.010%         99.984%          0.000              2
                      Reciprocal                1            0.011           0.010%         99.994%          0.004              1
                           AddV2                1            0.007           0.006%        100.000%          0.000              1

Timings (microseconds): count=80 first=142264 curr=110160 min=98340 max=171880 avg=115030 std=14609
Memory (bytes): count=80 first=206066628 curr=205570420 min=205570420 max=206066628 avg=2.05577e+08 std=55129
158 nodes observed