search

Thinklab-SJTU / R3Det_Tensorflow

Code for AAAI 2021 paper: R3Det: Refined Single-Stage Detector with Feature Refinement for Rotating Object
Apache License 2.0
546 stars 122 forks source link

ValueError: Tried to convert 'input' to a tensor and failed. Error: None values not supported. #104

Closed RomStriker closed 4 years ago

RomStriker commented 4 years ago

I am trying to train the model on a custom dataset. The dataset is ~650 images of books with around ~15500 mask and bbox annotations. I followed all the steps except using data_crop.py for my dataset and I have a tfrecord file of size 23.6GB. I also converted my dataset using the converter in tensorflow object detection api, and the size of that is 760.5MB. When I run multi_gpu_train_r3det.py, I get the following error in the output. Can you help me out in what could be wrong?

/home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:529: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:530: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:535: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
++--++--++--++--++--++--++--++--++--++--++--++--++--++--++--++--++--++--++--++--
/home/test/R3Det_Tensorflow
WARNING:tensorflow:From multi_gpu_train_r3det.py:128: get_or_create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.get_or_create_global_step
WARNING:tensorflow:From /home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
tfrecord path is --> /home/test/R3Det_Tensorflow/data/tfrecord/book-spine_train*
WARNING:tensorflow:From ../data/io/read_tfrecord_multi_gpu.py:115: string_input_producer (from tensorflow.python.training.input) is deprecated and will be removed in a future version.
Instructions for updating:
Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.from_tensor_slices(string_tensor).shuffle(tf.shape(input_tensor, out_type=tf.int64)[0]).repeat(num_epochs)`. If `shuffle=False`, omit the `.shuffle(...)`.
WARNING:tensorflow:From /home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/training/input.py:278: input_producer (from tensorflow.python.training.input) is deprecated and will be removed in a future version.
Instructions for updating:
Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.from_tensor_slices(input_tensor).shuffle(tf.shape(input_tensor, out_type=tf.int64)[0]).repeat(num_epochs)`. If `shuffle=False`, omit the `.shuffle(...)`.
WARNING:tensorflow:From /home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/training/input.py:190: limit_epochs (from tensorflow.python.training.input) is deprecated and will be removed in a future version.
Instructions for updating:
Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.from_tensors(tensor).repeat(num_epochs)`.
WARNING:tensorflow:From /home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/training/input.py:199: QueueRunner.__init__ (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
WARNING:tensorflow:From /home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/training/input.py:199: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
WARNING:tensorflow:From /home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/training/input.py:202: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From ../data/io/read_tfrecord_multi_gpu.py:22: TFRecordReader.__init__ (from tensorflow.python.ops.io_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.TFRecordDataset`.
WARNING:tensorflow:From ../data/io/image_preprocess_multi_gpu.py:175: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, use
    tf.py_function, which takes a python function which manipulates tf eager
    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
    an ndarray (just call tensor.numpy()) but having access to eager tensors
    means `tf.py_function`s can use accelerators such as GPUs as well as
    being differentiable using a gradient tape.

WARNING:tensorflow:From ../data/io/read_tfrecord_multi_gpu.py:125: batch (from tensorflow.python.training.input) is deprecated and will be removed in a future version.
Instructions for updating:
Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.batch(batch_size)` (or `padded_batch(...)` if `dynamic_pad=True`).
WARNING:tensorflow:From /home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
/home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py:110: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
  "Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
Traceback (most recent call last):
  File "/home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 511, in _apply_op_helper
    preferred_dtype=default_dtype)
  File "/home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1175, in internal_convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 304, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)
  File "/home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 245, in constant
    allow_broadcast=True)
  File "/home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 283, in _constant_impl
    allow_broadcast=allow_broadcast))
  File "/home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/framework/tensor_util.py", line 454, in make_tensor_proto
    raise ValueError("None values not supported.")
ValueError: None values not supported.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 525, in _apply_op_helper
    values, as_ref=input_arg.is_ref).dtype.name
  File "/home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1175, in internal_convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 304, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)
  File "/home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 245, in constant
    allow_broadcast=True)
  File "/home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 283, in _constant_impl
    allow_broadcast=allow_broadcast))
  File "/home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/framework/tensor_util.py", line 454, in make_tensor_proto
    raise ValueError("None values not supported.")
ValueError: None values not supported.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "multi_gpu_train_r3det.py", line 375, in <module>
    train()
  File "multi_gpu_train_r3det.py", line 271, in train
    grads = sum_gradients(tower_grads)
  File "multi_gpu_train_r3det.py", line 81, in sum_gradients
    expanded_g = tf.expand_dims(g, 0)
  File "/home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/util/dispatch.py", line 180, in wrapper
    return target(*args, **kwargs)
  File "/home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 144, in expand_dims
    return expand_dims_v2(input, axis, name)
  File "/home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/util/dispatch.py", line 180, in wrapper
    return target(*args, **kwargs)
  File "/home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 194, in expand_dims_v2
    return gen_array_ops.expand_dims(input, axis, name)
  File "/home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 2285, in expand_dims
    "ExpandDims", input=input, dim=axis, name=name)
  File "/home/test/anaconda3/envs/tf-gpu-r3det/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 529, in _apply_op_helper
    (input_name, err))
ValueError: Tried to convert 'input' to a tensor and failed. Error: None values not supported.
RomStriker commented 4 years ago

Closing this issue as properly converting the dataset fixed the issue.