search

VisualComputingInstitute / triplet-reid

Code for reproducing the results of our "In Defense of the Triplet Loss for Person Re-Identification" paper.
https://arxiv.org/abs/1703.07737
MIT License
764 stars 215 forks source link

Tensorflow loading latest checkpoint doesn't work #85

Closed mazatov closed 4 years ago

mazatov commented 4 years ago

I run into an error when trying to run embed.py. Looks like checkpoint =tf.train.latest_checkpoint(args.experiment_root) is returning None. Currently, args.experiment_root is set up to show the folder where I have downloaded checkpoint-25000.data-00000-of-00001. Should I be providing something else in args.experiment_root? 

Restoring from checkpoint: None
Traceback (most recent call last):
  File "embed.py", line 250, in <module>
    main()
  File "embed.py", line 211, in main
    tf.train.Saver().restore(sess, checkpoint)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\saver.py", line 1264, in restore
    raise ValueError("Can't load save_path when it is None.")
ValueError: Can't load save_path when it is None.
mazatov commented 4 years ago

Trying passing an absolute path of the experiments folder as suggested in some other threads, but I get the same result.

Edit: I tried the following and it is not giving me an error: 

saver = tf.train.import_meta_graph('experiments\my_experiment\checkpoint-25000.meta')
saver.restore(sess, 'experiments\my_experiment\checkpoint-25000')

I'm not really sure where to go from here! How's one giving me an error and another is fine? 😕 Is there some naming convention I'm not aware of? Or there's a conflict with TF versions? I'm using 1.13.1

Update: When I tried to get the embeddings for an image using this loading method I get the following error:

My code

with tf.Session() as sess:
    saver = tf.train.import_meta_graph(os.path.join(config['experiment_root'],'checkpoint-25000.meta'))
    saver.restore(sess,os.path.join(config['experiment_root'],'checkpoint-25000') )

    raw_img = cv2.imread(os.path.join(config['image_root'],'query', '0001_c1s1_001051_00.jpg'))
    raw_img = cv2.resize(raw_img, net_input_size)
    raw_img = np.swapaxes(raw_img, 0, 1)
    raw_img = np.expand_dims(raw_img, axis=0)

    emb = sess.run(endpoints['emb'],  feed_dict={img: raw_img})[0]

Error

FailedPreconditionError: Attempting to use uninitialized value resnet_v1_50/block4/unit_3/bottleneck_v1/conv3/BatchNorm/gamma
     [[node resnet_v1_50/block4/unit_3/bottleneck_v1/conv3/BatchNorm/gamma/read (defined at C:\Users\mazat\Documents\Python\trinet\nets\resnet_v1.py:118) ]]
     [[node head/emb/BiasAdd (defined at C:\Users\mazat\Documents\Python\trinet\heads\fc1024.py:17) ]]

Caused by op 'resnet_v1_50/block4/unit_3/bottleneck_v1/conv3/BatchNorm/gamma/read', defined at:
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\spyder_kernels\console\__main__.py", line 11, in <module>
    start.main()
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\spyder_kernels\console\start.py", line 318, in main
    kernel.start()
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\ipykernel\kernelapp.py", line 563, in start
    self.io_loop.start()
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tornado\platform\asyncio.py", line 148, in start
    self.asyncio_loop.run_forever()
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\asyncio\base_events.py", line 438, in run_forever
    self._run_once()
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\asyncio\base_events.py", line 1451, in _run_once
    handle._run()
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\asyncio\events.py", line 145, in _run
    self._callback(*self._args)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tornado\ioloop.py", line 690, in <lambda>
    lambda f: self._run_callback(functools.partial(callback, future))
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tornado\ioloop.py", line 743, in _run_callback
    ret = callback()
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tornado\gen.py", line 787, in inner
    self.run()
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tornado\gen.py", line 748, in run
    yielded = self.gen.send(value)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\ipykernel\kernelbase.py", line 365, in process_one
    yield gen.maybe_future(dispatch(*args))
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tornado\gen.py", line 209, in wrapper
    yielded = next(result)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\ipykernel\kernelbase.py", line 272, in dispatch_shell
    yield gen.maybe_future(handler(stream, idents, msg))
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tornado\gen.py", line 209, in wrapper
    yielded = next(result)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\ipykernel\kernelbase.py", line 542, in execute_request
    user_expressions, allow_stdin,
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tornado\gen.py", line 209, in wrapper
    yielded = next(result)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\ipykernel\ipkernel.py", line 294, in do_execute
    res = shell.run_cell(code, store_history=store_history, silent=silent)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\ipykernel\zmqshell.py", line 536, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\IPython\core\interactiveshell.py", line 2855, in run_cell
    raw_cell, store_history, silent, shell_futures)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\IPython\core\interactiveshell.py", line 2881, in _run_cell
    return runner(coro)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\IPython\core\async_helpers.py", line 68, in _pseudo_sync_runner
    coro.send(None)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\IPython\core\interactiveshell.py", line 3058, in run_cell_async
    interactivity=interactivity, compiler=compiler, result=result)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\IPython\core\interactiveshell.py", line 3249, in run_ast_nodes
    if (await self.run_code(code, result,  async_=asy)):
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\IPython\core\interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-1-c077a3858941>", line 1, in <module>
    runfile('C:/Users/mazat/Documents/Python/trinet/g1.py', wdir='C:/Users/mazat/Documents/Python/trinet')
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 827, in runfile
    execfile(filename, namespace)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 110, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)
  File "C:/Users/mazat/Documents/Python/trinet/g1.py", line 25, in <module>
    endpoints, _ = model.endpoints(img, is_training=False)
  File "C:\Users\mazat\Documents\Python\trinet\nets\resnet_v1_50.py", line 14, in endpoints
    _, endpoints = resnet_v1_50(image, num_classes=None, is_training=is_training, global_pool=True)
  File "C:\Users\mazat\Documents\Python\trinet\nets\resnet_v1.py", line 276, in resnet_v1_50
    reuse=reuse, scope=scope)
  File "C:\Users\mazat\Documents\Python\trinet\nets\resnet_v1.py", line 216, in resnet_v1
    net = resnet_utils.stack_blocks_dense(net, blocks, output_stride)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 182, in func_with_args
    return func(*args, **current_args)
  File "C:\Users\mazat\Documents\Python\trinet\nets\resnet_utils.py", line 189, in stack_blocks_dense
    net = block.unit_fn(net, rate=1, **unit)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 182, in func_with_args
    return func(*args, **current_args)
  File "C:\Users\mazat\Documents\Python\trinet\nets\resnet_v1.py", line 118, in bottleneck
    activation_fn=None, scope='conv3')
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 182, in func_with_args
    return func(*args, **current_args)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 1155, in convolution2d
    conv_dims=2)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 182, in func_with_args
    return func(*args, **current_args)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 1067, in convolution
    outputs = normalizer_fn(outputs, **normalizer_params)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 182, in func_with_args
    return func(*args, **current_args)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 651, in batch_norm
    outputs = layer.apply(inputs, training=is_training)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 1227, in apply
    return self.__call__(inputs, *args, **kwargs)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\layers\base.py", line 530, in __call__
    outputs = super(Layer, self).__call__(inputs, *args, **kwargs)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 538, in __call__
    self._maybe_build(inputs)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 1603, in _maybe_build
    self.build(input_shapes)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\layers\normalization.py", line 330, in build
    trainable=True)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\layers\base.py", line 435, in add_weight
    getter=vs.get_variable)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 349, in add_weight
    aggregation=aggregation)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\checkpointable\base.py", line 607, in _add_variable_with_custom_getter
    **kwargs_for_getter)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 1479, in get_variable
    aggregation=aggregation)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 1220, in get_variable
    aggregation=aggregation)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 530, in get_variable
    return custom_getter(**custom_getter_kwargs)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 1938, in wrapped_custom_getter
    *args, **kwargs)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 1750, in layer_variable_getter
    return _model_variable_getter(getter, *args, **kwargs)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 1741, in _model_variable_getter
    aggregation=aggregation)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 182, in func_with_args
    return func(*args, **current_args)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\contrib\framework\python\ops\variables.py", line 350, in model_variable
    aggregation=aggregation)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 182, in func_with_args
    return func(*args, **current_args)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\contrib\framework\python\ops\variables.py", line 277, in variable
    aggregation=aggregation)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 1750, in layer_variable_getter
    return _model_variable_getter(getter, *args, **kwargs)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 1741, in _model_variable_getter
    aggregation=aggregation)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 182, in func_with_args
    return func(*args, **current_args)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\contrib\framework\python\ops\variables.py", line 350, in model_variable
    aggregation=aggregation)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 182, in func_with_args
    return func(*args, **current_args)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\contrib\framework\python\ops\variables.py", line 277, in variable
    aggregation=aggregation)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 499, in _true_getter
    aggregation=aggregation)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 911, in _get_single_variable
    aggregation=aggregation)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\variables.py", line 213, in __call__
    return cls._variable_v1_call(*args, **kwargs)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\variables.py", line 176, in _variable_v1_call
    aggregation=aggregation)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\variables.py", line 155, in <lambda>
    previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 2495, in default_variable_creator
    expected_shape=expected_shape, import_scope=import_scope)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\variables.py", line 217, in __call__
    return super(VariableMetaclass, cls).__call__(*args, **kwargs)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\variables.py", line 1395, in __init__
    constraint=constraint)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\variables.py", line 1557, in _init_from_args
    self._snapshot = array_ops.identity(self._variable, name="read")
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\util\dispatch.py", line 180, in wrapper
    return target(*args, **kwargs)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\array_ops.py", line 81, in identity
    ret = gen_array_ops.identity(input, name=name)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\gen_array_ops.py", line 3890, in identity
    "Identity", input=input, name=name)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\framework\ops.py", line 3300, in create_op
    op_def=op_def)
  File "C:\Users\mazat\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\framework\ops.py", line 1801, in __init__
    self._traceback = tf_stack.extract_stack()

FailedPreconditionError (see above for traceback): Attempting to use uninitialized value resnet_v1_50/block4/unit_3/bottleneck_v1/conv3/BatchNorm/gamma
     [[node resnet_v1_50/block4/unit_3/bottleneck_v1/conv3/BatchNorm/gamma/read (defined at C:\Users\mazat\Documents\Python\trinet\nets\resnet_v1.py:118) ]]
     [[node head/emb/BiasAdd (defined at C:\Users\mazat\Documents\Python\trinet\heads\fc1024.py:17) ]]
mazatov commented 4 years ago

Update

I was able to get embeddings by loading the model as 

tf.train.Saver().restore(sess, os.path.join(config['experiment_root'],'checkpoint-25000') )

Not sure why the other way it doesn't find the checkpoint but it seems to work this way. Also when running embed.py, I ran it as 

python embed.py --experiment_root experiments/my_experiment --dataset data/market1501_query.csv --filename market1501_query_embeddings.h5 --checkpoint checkpoint-25000

Specifying the checkpoint name seems to do the trick.