search

JOP-Lee / READ

AAAI2023，implementation of "READ: Large-Scale Neural Scene Rendering for Autonomous Driving", the experimental results are significantly better than Nerf-based methods
https://github.com/JOP-Lee/READ-Large-Scale-Neural-Scene-Rendering-for-Autonomous-Driving
GNU General Public License v2.0
447 stars 55 forks source link

train question #59

Open Wangcongyu620 opened 1 year ago

Wangcongyu620 commented 1 year ago

When I run train.py in kitti6 dataset, I always encounter the following bugs, and I can't find the reason. Can you help me take a look? ds_train: 3000 ds_val: 82 LOAD NET CHECKPOINT /home/wenbo/code_cy/READ-main/data/logs_offical/path/UNet_stage_0_epoch_net.pth

EPOCH 0

> TRAIN EVAL MODE IN TRAIN model parameters: 30193988 running on datasets [0] model3d {'rgb': array([[0., 0., 0.], [0., 0., 0.], [0., 0., 0.], ..., [0., 0., 0.], [0., 0., 0.], [0., 0., 0.]], dtype=float32), 'normals': None, 'uv2d': array([[0., 0.], [0., 0.], [0., 0.], ..., [0., 0.], [0., 0.], [0., 0.]], dtype=float32), 'faces': array([0, 1, 2], dtype=uint32), 'xyz': TrackedArray([[ -21.45231247, 13.24265003, -207.19848633], [ -21.45231247, 13.24265003, -207.19848633], [ -21.84580994, 14.13696384, -209.91720581], ..., [ -30.93203926, 15.21014023, 684.93322754], [ -30.93203926, 15.21014023, 684.93322754], [ -30.93203926, 15.21014023, 684.93322754]]), 'xyz_c': TrackedArray([[0.41255814, 0.4299459 , 0.02919084], [0.41255814, 0.4299459 , 0.02919084], [0.41 , 0.44497895, 0.02623283], ..., [0.3509302 , 0.46301866, 0.9998443 ], [0.3509302 , 0.46301866, 0.9998443 ], [0.3509302 , 0.46301866, 0.9998443 ]], dtype=float32), 'uv1d': array([ 0, 1, 2, ..., 2547319, 2547320, 2547321])} proj_matrix was not set total parameters: 50572564[w] b'GLX: No GLXFBConfigs returned' [w] b'GLX: Failed to find a suitable GLXFBConfig' [x] Window creation failed [w] b'GLX: No GLXFBConfigs returned' [w] b'GLX: Failed to find a suitable GLXFBConfig' [x] Window creation failed [w] b'GLX: No GLXFBConfigs returned' [w] b'GLX: Failed to find a suitable GLXFBConfig' [x] Window creation failed Traceback (most recent call last): File "/home/wenbo/anaconda3/envs/READ/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 990, in _try_get_data data = self._data_queue.get(timeout=timeout) File "/home/wenbo/anaconda3/envs/READ/lib/python3.9/multiprocessing/queues.py", line 114, in get raise Empty _queue.Empty

The above exception was the direct cause of the following exception:Traceback (most recent call last): File "/home/wenbo/code_cy/READ-main/train.py", line 546, in <module> train_loss = run_train(epoch, pipeline, args, iter_cb) File "/home/wenbo/code_cy/READ-main/train.py", line 281, in run_train return run_epoch(pipeline, 'train', epoch, args, iter_cb=iter_cb) File "/home/wenbo/code_cy/READ-main/train.py", line 256, in run_epoch run_sub(dl, extra_optimizer) File "/home/wenbo/code_cy/READ-main/train.py", line 146, in run_sub for it, data in enumerate(dl): File "/home/wenbo/anaconda3/envs/READ/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 521, in next data = self._next_data() File "/home/wenbo/anaconda3/envs/READ/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1186, in _next_data idx, data = self._get_data() File "/home/wenbo/anaconda3/envs/READ/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1152, in _get_data success, data = self._try_get_data() File "/home/wenbo/anaconda3/envs/READ/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1003, in _try_get_data raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e RuntimeError: DataLoader worker (pid(s) 8638, 8658, 8678) exited unexpectedly deleting buffers...

Process finished with exit code 1 @JOP-Lee Looking forward to your reply！

VerseWei commented 7 months ago

Hi，I encounter the same dilemma now, could you share your method for this problem with me? Many thanks for you!