About the tensorflow benchmark in half-percision

[foreverYoungGitHub]

Hi, @u39kun. Thank you for your work!

When I check the way you implement the tensorflow here, I found that there is a notation in the Func get_variable() like following:

  def get_variable(self, name, shape, dtype, cast_dtype, *args, **kwargs):
    # TODO(reedwm): Currently variables and gradients are transferred to other
    # devices and machines as type `dtype`, not `cast_dtype`. In particular,
    # this means in fp16 mode, variables are transferred as fp32 values, not
    # fp16 values, which uses extra bandwidth.

Do you mean that currently, if the model is trained as float32, when the model'll be loaded as float32, but it will compute as float16? Only the bandwidth in the GPU will be effected, but the speed will keep same?

[u39kun]

Hi @foreverYoungGitHub I borrowed that piece of code from https://github.com/tensorflow/benchmarks/blob/master/scripts/tf_cnn_benchmarks/convnet_builder.py#L112-L118 Looks like this came from https://github.com/tensorflow/benchmarks/pull/63

You may want to ask the TensorFlow community on that.

[foreverYoungGitHub]

Oh, Thanks for that!

Actually, I just did the time benchmark for that. And the filter/kernel is actually computed as float16, which is much faster.

But when I run your code and the my own time benchmark code. I found that the speed of your code is much faster than my. Except the warm-up part in your code, what causes this different?

I attach my code as following.

import tensorflow as tf
import time
from datetime import datetime
import math
import argparse
import sys

import numpy as np

slim = tf.contrib.slim

import os
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"   # see issue #152
os.environ["CUDA_VISIBLE_DEVICES"]="0"

def time_tensorflow_run_placeholder(session, target, feed_dict, info_string):
  num_steps_burn_in = 10
  total_duration = 0.0
  total_duration_squared = 0.0

  for i in range(FLAGS.num_batches + num_steps_burn_in):
    start_time = time.time()
    _ = session.run(target,feed_dict=feed_dict)
    duration = time.time() - start_time
    if i >= num_steps_burn_in:
      if not i % 10:
        print('%s: step %d, duration = %.3f' % (datetime.now(), i - num_steps_burn_in, duration))
      total_duration += duration
      total_duration_squared += duration * duration

  mn = total_duration / FLAGS.num_batches
  vr = total_duration_squared / FLAGS.num_batches - mn * mn
  sd = math.sqrt(vr)
  print('%s: %s across %d steps, %.3f +/- %.3f sec / batch' % (datetime.now(), info_string, FLAGS.num_batches, mn, sd))

def run_benchmark():
  graph_filename = FLAGS.graph_dir + "-{DATA_FORMAT}-{PRECISION}/frozen_graph.pb".format(DATA_FORMAT=FLAGS.data_format, PRECISION=FLAGS.precision)
  # Create a graph def object to read the graph
  with tf.gfile.GFile(graph_filename, "rb") as f:
    graph_def = tf.GraphDef()
    graph_def.ParseFromString(f.read())

  with tf.Graph().as_default() as graph:
    with tf.device('/'+FLAGS.mode+':0'):
      if FLAGS.data_format == 'NCHW':
        inputs = np.random.random((FLAGS.batch_size, 3, FLAGS.input_width, FLAGS.input_height))
      elif FLAGS.data_format == 'NHWC':
        inputs = np.random.random((FLAGS.batch_size, FLAGS.input_width, FLAGS.input_height, 3))
      if precision == 'fp16':
        inputs = inputs.astype(np.float16)
      tf.import_graph_def(graph_def)
      config = tf.ConfigProto()
      config.gpu_options.allocator_type = 'BFC'
      sess = tf.Session(config=config)
      # We define the input and output node we will feed in
      input_node = graph.get_tensor_by_name('import/input:0')
      output_node = graph.get_tensor_by_name('import/predictions/Reshape_1:0')
      time_tensorflow_run_placeholder(sess, output_node, {input_node: inputs}, "Forward")

def main(_):
  run_benchmark()

[u39kun]

@foreverYoungGitHub Off the top of my head, things that can definitely affect performance is the data format. NCHW is much faster than NHWC. Also, I noticed with TensorFlow is that it takes many warmup runs to get to the optimal speed. That's why in my code the number of warmup runs is set to 20... This was to specifically accommodate TensorFlow (PyTorch, for example, would "warmup" in just one run.)