search

alexklwong / unsupervised-depth-completion-visual-inertial-odometry

Tensorflow and PyTorch implementation of Unsupervised Depth Completion from Visual Inertial Odometry (in RA-L January 2020 & ICRA 2020)
Other
192 stars 24 forks source link

Invalid argument: Cannot add tensor to the batch: number of elements does not match. #3

Closed Li-goudan closed 2 years ago

Li-goudan commented 2 years ago

Sorry to bother you, but when I was training VOICED on the KITTI dataset, I encountered the following error and I didn't know how to solve it.

Traceback (most recent call last):
  File "src/train_voiced.py", line 136, in <module>
    n_thread=args.n_thread)
  File "/home/udcvio/src/voiced_main.py", line 186, in train
    _, loss_value, summary = session.run([gradients, loss, model_summary])
  File "/root/anaconda3/envs/udcvio/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/root/anaconda3/envs/udcvio/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)
  File "/root/anaconda3/envs/udcvio/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
    run_metadata)
  File "/root/anaconda3/envs/udcvio/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument: Cannot add tensor to the batch: number of elements does not match. Shapes are: [tensor]: [178,768,2], [batch]: [320,768,2]
         [[node dataloader/IteratorGetNext (defined at /root/anaconda3/envs/udcvio/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
         [[dataloader/IteratorGetNext/_15]]
  (1) Invalid argument: Cannot add tensor to the batch: number of elements does not match. Shapes are: [tensor]: [178,768,2], [batch]: [320,768,2]
         [[node dataloader/IteratorGetNext (defined at /root/anaconda3/envs/udcvio/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'dataloader/IteratorGetNext':
  File "src/train_voiced.py", line 136, in <module>
    n_thread=args.n_thread)
  File "/home/udcvio/src/voiced_main.py", line 91, in train
    prefetch_size=2*n_thread)
  File "/home/udcvio/src/dataloader.py", line 73, in __init__
    self.next_element = self.iterator.get_next()
  File "/root/anaconda3/envs/udcvio/lib/python3.7/site-packages/tensorflow_core/python/data/ops/iterator_ops.py", line 426, in get_next
    name=name)
  File "/root/anaconda3/envs/udcvio/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_dataset_ops.py", line 2518, in iterator_get_next
    output_shapes=output_shapes, name=name)
  File "/root/anaconda3/envs/udcvio/lib/python3.7/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "/root/anaconda3/envs/udcvio/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/root/anaconda3/envs/udcvio/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "/root/anaconda3/envs/udcvio/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "/root/anaconda3/envs/udcvio/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()
Li-goudan commented 2 years ago

The above problem may be caused by incomplete preprocessing of the KITTI data set. The KITTI data set is too large. So I tried to train on the VOID data set and succeeded. So, thank you!

alexklwong commented 2 years ago

That's strange, were you executing the setup code from the root of the repository?

unsupervised-depth-completion-visual-inertial-odometry <---- execute here |----- data |----- setup |----- src

Li-goudan commented 2 years ago

Yes, I was executing from the root of the repository. In fact, I did not check the integrity of the KITTI data set after it was transferred from my laptop to the server. I think some picture information may have been lost during the transfer, which caused this problem. Anyway, I've reproduced your work using the VOID dataset. Cool project, by the way!

alexklwong commented 2 years ago

Ah I see, you can actually execute this command from the server: bash bash/setup_dataset_kitti.sh

It runs the download script, so as long as your server has internet access it will download from AWS: https://github.com/alexklwong/unsupervised-depth-completion-visual-inertial-odometry/blob/master/bash/setup_dataset_kitti.sh

Also in case you are interested, this is a much better improvement over VOICED and shares a lot of the common themes in the work from densification to lifting and backprojecting the point cloud: https://github.com/alexklwong/calibrated-backprojection-network

Improvements of calibrated backprojection network is around 14% outdoor and 51% indoor over VOICED.

Li-goudan commented 2 years ago

Thank you for your suggestion. And I have followed your new project. Compared with VOICED, its performance improvement is very impressive.