Config file needs modification- not starting training

(math) kd636624@login-g:~/demo/math/VoxelNet-tensorflow[1053]$ python3 train.py --max-epoch 20 2021-03-25 14:21:03.448203: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/intel/impi/2018.4.274/compilers_and_libraries/linux/mpi/lib64:/opt/intel/impi/2018.4.274/compilers_and_libraries/linux/mpi/lib:/opt/intel/Compiler/19.0/1.144/rwthlnk/daal/lib/intel64_lin:/opt/intel/Compiler/19.0/1.144/rwthlnk/daal/lib/ia32_lin:/opt/intel/Compiler/19.0/1.144/rwthlnk/ipp/lib/intel64_lin:/opt/intel/Compiler/19.0/1.144/rwthlnk/ipp/lib/ia32_lin:/opt/intel/Compiler/19.0/1.144/rwthlnk/mkl/lib/intel64_lin:/opt/intel/Compiler/19.0/1.144/rwthlnk/mkl/lib/ia32_lin:/opt/intel/Compiler/19.0/1.144/rwthlnk/tbb/lib/intel64_lin/gcc4.7:/opt/intel/Compiler/19.0/1.144/rwthlnk/tbb/lib/ia32_lin/gcc4.7:/opt/intel/Compiler/19.0/1.144/rwthlnk/compiler/lib/intel64_lin:/opt/intel/Compiler/19.0/1.144/rwthlnk/compiler/lib/ia32_lin 2021-03-25 14:21:03.448243: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. GPU Available from Config 0,1,2,3 GPU count from Config 4 split: object dir: ./data/object/training data_tag: 7481 f_rgb: 7481 f_lidar: 7481 Dataset total length: 7481 split: object dir: ./data/object/testing data_tag: 7518 f_rgb: 7518 f_lidar: 7518 Dataset total length: 7518 Process Process-9: Process Process-10: Process Process-11: Process Process-12: Process Process-13: Process Process-14: Process Process-15: 2021-03-25 14:21:19.517590: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2021-03-25 14:21:19.519417: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set 2021-03-25 14:21:19.520750: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1 Process Process-16: Traceback (most recent call last): File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run self._target(*self._args, self._kwargs) File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 241, in loader_worker_main self.fill_queue(batch_size) File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 167, in fill_queue voxel[idx single_batch_size:(idx + 1) single_batch_size]) File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 274, in build_input feature = np.concatenate(feature_list) File "<__array_function__ internals>", line 6, in concatenate ValueError: need at least one array to concatenate Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run self._target(*self._args, *self._kwargs) File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run self._target(self._args, self._kwargs) File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 241, in loader_worker_main self.fill_queue(batch_size) File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 241, in loader_worker_main self.fill_queue(batch_size) File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run self._target(*self._args, self._kwargs) File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 167, in fill_queue voxel[idx single_batch_size:(idx + 1) single_batch_size]) File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run self._target(*self._args, *self._kwargs) File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run self._target(self._args, self._kwargs) File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 167, in fill_queue voxel[idx single_batch_size:(idx + 1) single_batch_size]) File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 241, in loader_worker_main self.fill_queue(batch_size) File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 274, in build_input feature = np.concatenate(feature_list) File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 241, in loader_worker_main self.fill_queue(batch_size) File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 241, in loader_worker_main self.fill_queue(batch_size) File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 274, in build_input feature = np.concatenate(feature_list) File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 167, in fill_queue voxel[idx single_batch_size:(idx + 1) single_batch_size]) File "<__array_function__ internals>", line 6, in concatenate File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 167, in fill_queue voxel[idx single_batch_size:(idx + 1) single_batch_size]) File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 167, in fill_queue voxel[idx single_batch_size:(idx + 1) single_batch_size]) File "<__array_function__ internals>", line 6, in concatenate File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 274, in build_input feature = np.concatenate(feature_list) File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 274, in build_input feature = np.concatenate(feature_list) File "<__array_function__ internals>", line 6, in concatenate File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 274, in build_input feature = np.concatenate(feature_list) File "<__array_function__ internals>", line 6, in concatenate ValueError: need at least one array to concatenate File "<__array_function__ internals>", line 6, in concatenate ValueError: need at least one array to concatenate ValueError: need at least one array to concatenate ValueError: need at least one array to concatenate ValueError: need at least one array to concatenate Traceback (most recent call last): Traceback (most recent call last): File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run self._target(*self._args, *self._kwargs) File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 241, in loader_worker_main self.fill_queue(batch_size) File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 167, in fill_queue voxel[idx single_batch_size:(idx + 1) single_batch_size]) File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 274, in build_input feature = np.concatenate(feature_list) File "<__array_function__ internals>", line 6, in concatenate File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap self.run() ValueError: need at least one array to concatenate File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run self._target(self._args, *self._kwargs) File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 241, in loader_worker_main self.fill_queue(batch_size) File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 167, in fill_queue voxel[idx single_batch_size:(idx + 1) * single_batch_size]) File "/rwthfs/rz/cluster/home/kd636624/demo/math/VoxelNet-tensorflow/utils/kitti_loader.py", line 274, in build_input feature = np.concatenate(feature_list) File "<__array_function__ internals>", line 6, in concatenate ValueError: need at least one array to concatenate 2021-03-25 14:21:20.790883: E tensorflow/core/common_runtime/session.cc:91] Failed to create session: Invalid argument: 'visible_device_list' listed an invalid GPU id '2' but visible device count is 2 2021-03-25 14:21:20.790922: E tensorflow/c/c_api.cc:2184] Invalid argument: 'visible_device_list' listed an invalid GPU id '2' but visible device count is 2 Traceback (most recent call last): File "train.py", line 142, in tf.app.run(main) File "/rwthfs/rz/cluster/home/kd636624/demo/math/lib64/python3.6/site-packages/tensorflow/python/platform/app.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "/rwthfs/rz/cluster/home/kd636624/demo/math/lib64/python3.6/site-packages/absl/app.py", line 303, in run _run_main(main, args) File "/rwthfs/rz/cluster/home/kd636624/demo/math/lib64/python3.6/site-packages/absl/app.py", line 251, in _run_main sys.exit(main(argv)) File "train.py", line 60, in main with tf.Session(config=config) as sess: File "/rwthfs/rz/cluster/home/kd636624/demo/math/lib64/python3.6/site-packages/tensorflow/python/client/session.py", line 1596, in init super(Session, self).init(target, graph, config=config) File "/rwthfs/rz/cluster/home/kd636624/demo/math/lib64/python3.6/site-packages/tensorflow/python/client/session.py", line 711, in init self._session = tf_session.TF_NewSessionRef(self._graph._c_graph, opts) tensorflow.python.framework.errors_impl.InvalidArgumentError: 'visible_device_list' listed an invalid GPU id '2' but visible device count is 2

Problems: Config file doesn't get imported in Kitti_loader, Kitti_loader fails to load complete dataset and training is not starting at all. Any help would be greatly appreciated. Thanks

tsinghua-rll / VoxelNet-tensorflow

Config file needs modification- not starting training #74