the predict is not with the same result of evaluate

[WoNiuHu]

when I predict the model, the result is not the same with evaluate, and the result has much difference. I fetch the fc_weight , save to model and restore from model of the weight is not the same. Maybe the model has something wrong.

[wenxinxu]

Do you mind sharing your code for testing? Did you try loading the saved checkpoint directly?

[WoNiuHu]

def predict_tfrecords_by_checkpoint_image(checkpoint_dir, in_file_pattern, to_file):

config = ConfigParser.ConfigParser()
config.read(flags.FLAGS.predict_meta_file)
input_shape = config.getint("Input", "INPUT_SHAPE")
frozen_graph_path = os.path.join(checkpoint_dir, "model.pb")
prefix = "prefix"
graph = _load_graph(frozen_graph_path, prefix)
with tf.Session(graph=graph, config=_build_config_proto()) as sess:
    # First let's load meta graph and restore weights.
    # checkpoint = tf.train.latest_checkpoint(checkpoint_dir)
    # saver = tf.train.import_meta_graph("%s.meta" % checkpoint)
    # saver.restore(sess, checkpoint)
    #
    # # Get the input & output node of the graph.
    # graph = tf.get_default_graph()

    image_raw = graph.get_tensor_by_name('%s/image_raw:0' % prefix)
    y = graph.get_tensor_by_name('%s/y:0' % prefix)
    fc_weights = graph.get_tensor_by_name('%s/fc/fc_weights:0' %prefix)

    ys = []
    pred_ys = []
    for batch in tqdm(_batch_iter_in_tfrecord_files_image(in_file_pattern, batch_size=1)):
        label_batch, image_raw_batch = list(zip(*batch))

        y_out, _fc_weights = sess.run([y,fc_weights] , feed_dict={
            image_raw: image_raw_batch
        })
        with open("fc_weight6", 'w') as file:
            file.write(','.join(str(i) for i in _fc_weights))
        if isinstance(y_out, numpy.ndarray):
            pred_ys += list(y_out)
            ys += label_batch
        else:
            pred_ys += [y_out]
            ys += label_batch

    with open(to_file, "w") as fout:
        for y, pred_y in zip(ys, pred_ys):
            pred_y = ','.join(str(item) for item in pred_y)
            fout.write("%s\t%s\n" % (y, pred_y))

    print "==> predict in %s" % to_file`

[WoNiuHu]

`def record_image_placeholder_input_fn(meta_file, default_batch_size=None): """Build the serving inputs.

Args:
  default_batch_size (int): Batch size for the tf.placeholder shape
"""
input_meta = _parse_record_meta(meta_file)
LABELS = input_meta["LABELS"]
INPUT_SHAPE = input_meta["INPUT_SHAPE"]   # here is 3072

inputs = {
    'image_raw': tf.placeholder(dtype=tf.int64, shape=[None, INPUT_SHAPE], name="image_raw")
}

features = {
    'image_raw': inputs['image_raw']
}

return features, inputs`

[WoNiuHu]

Actually , I rewrite the model, in the training process, it can run normally and also evaluate the model , in 20k train steps, it has the accuracy about 0.7, but when the model saved, I predict it again, it's accuracy is about 0.1, when I debug this , I found the last weight saved in model, and then I restored the model, but the same weight did not the same.