Open liangchunlan opened 2 years ago
chenxuluo
commented
2 years ago I haven't tried using mini. I think you can directly use the original data preprocessing code by changing the version to v1.0-mini.
The training takes about 15 hours on 8 TITAN RTX,
As for the checkpoint, I used pytorch1.6. It is due to the _use_new_zipfile_serialization when saving the model. I think the simplest way is to upgrade your pytorch to 1.6+
liangchunlan
commented
2 years ago I can successfully train and test with v1.0-trainval on pytorch = 1.4.0 and torchvision = 0.5.0. Since I only have one RTX now, I only trained once, and the results are not good as expected. When I want to use the model provided by you, I upgrade pytorch to 1.6.0, but I run as follows. I don't know whether this is correct.
(base) ubuntu@ubuntu-System-Product-Name:/media/ubuntu/nuscenes/code/simtrack$ conda install pytorch=1.6.0 torchvision=0.7.0 torchaudio cudatoolkit=10.2 -c pytorch
Collecting package metadata (current_repodata.json): done
Solving environment: done
# All requested packages already installed.
(base) ubuntu@ubuntu-System-Product-Name:/media/ubuntu/nuscenes/code/simtrack$ cd det3d/ops/iou3d_nms/
(base) ubuntu@ubuntu-System-Product-Name:/media/ubuntu/nuscenes/code/simtrack/det3d/ops/iou3d_nms$ python setup.py build_ext --inplace
running build_ext
building 'iou3d_nms_cuda' extension
Emitting ninja build file /media/ubuntu/nuscenes/code/simtrack/det3d/ops/iou3d_nms/build/temp.linux-x86_64-3.8/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
g++ -pthread -shared -B /home/ubuntu/anaconda3/compiler_compat -L/home/ubuntu/anaconda3/lib -Wl,-rpath=/home/ubuntu/anaconda3/lib -Wl,--no-as-needed -Wl,--sysroot=/ /media/ubuntu/nuscenes/code/simtrack/det3d/ops/iou3d_nms/build/temp.linux-x86_64-3.8/src/iou3d_cpu.o /media/ubuntu/nuscenes/code/simtrack/det3d/ops/iou3d_nms/build/temp.linux-x86_64-3.8/src/iou3d_nms_api.o /media/ubuntu/nuscenes/code/simtrack/det3d/ops/iou3d_nms/build/temp.linux-x86_64-3.8/src/iou3d_nms.o /media/ubuntu/nuscenes/code/simtrack/det3d/ops/iou3d_nms/build/temp.linux-x86_64-3.8/src/iou3d_nms_kernel.o -L/home/ubuntu/anaconda3/lib/python3.8/site-packages/torch/lib -L/usr/local/cuda-10.0/lib64 -lc10 -ltorch -ltorch_cpu -ltorch_python -lcudart -lc10_cuda -ltorch_cuda -o build/lib.linux-x86_64-3.8/iou3d_nms_cuda.cpython-38-x86_64-linux-gnu.so
copying build/lib.linux-x86_64-3.8/iou3d_nms_cuda.cpython-38-x86_64-linux-gnu.so ->
(base) ubuntu@ubuntu-System-Product-Name:/media/ubuntu/nuscenes/code/simtrack/det3d/ops/iou3d_nms$ cd ../../../
(base) ubuntu@ubuntu-System-Product-Name:/media/ubuntu/nuscenes/code/simtrack$ ls
apex examples README.md
apex-6af09dd938adf11baedabd89da6cd092d3959081 INSTALL.md requirements.txt
det3d LICENSE.md save_dir
example.gif model_zoo tools
(base) ubuntu@ubuntu-System-Product-Name:/media/ubuntu/nuscenes/code/simtrack$ python ./tools/val_nusc_tracking.py examples/point_pillars/configs/nusc_all_pp_centernet_tracking.py --checkpoint /media/ubuntu/nuscenes/code/simtrack/model_zoo/simtrack_pillar.pth --work_dir /media/ubuntu/nuscenes/code/simtrack/save_dir/simtrack_test --local_rank 0
1.6.0
Use HM Bias: -2.19
======
Loading NuScenes tables for version v1.0-trainval...
Loading nuScenes-lidarseg...
32 category,
8 attribute,
4 visibility,
64386 instance,
12 sensor,
10200 calibrated_sensor,
2631083 ego_pose,
68 log,
850 scene,
34149 sample,
2631083 sample_data,
1166187 sample_annotation,
4 map,
34149 lidarseg,
Done loading in 42.830 seconds.
======
Reverse indexing ...
Done reverse indexing in 5.9 seconds.
======
Error!
Error!
Error!
Error!
Error!
Error!
/media/ubuntu/nuscenes/code/simtrack/det3d/core/bbox/geometry.py:160: NumbaWarning:
Compilation is falling back to object mode WITH looplifting enabled because Function "points_in_convex_polygon_jit" failed type inference due to: No implementation of function Function(<built-in function getitem>) found for signature:
>>> getitem(array(float64, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))
There are 22 candidate implementations:
- Of which 20 did not match due to:
Overload of function 'getitem': File: <numerous>: Line N/A.
With argument(s): '(array(float64, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))':
No match.
- Of which 2 did not match due to:
Overload in function 'GetItemBuffer.generic': File: numba/core/typing/arraydecl.py: Line 162.
With argument(s): '(array(float64, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))':
Rejected as the implementation raised a specific error:
TypeError: unsupported array index type list(int64)<iv=None> in Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>)
raised from /home/ubuntu/anaconda3/lib/python3.8/site-packages/numba/core/typing/arraydecl.py:68
During: typing of intrinsic-call at /media/ubuntu/nuscenes/code/simtrack/det3d/core/bbox/geometry.py (177)
File "det3d/core/bbox/geometry.py", line 177:
def points_in_convex_polygon_jit(points, polygon, clockwise=True):
<source elided>
polygon
- polygon[
^
@numba.jit
/media/ubuntu/nuscenes/code/simtrack/det3d/core/bbox/geometry.py:160: NumbaWarning:
Compilation is falling back to object mode WITHOUT looplifting enabled because Function "points_in_convex_polygon_jit" failed type inference due to: cannot determine Numba type of <class 'numba.core.dispatcher.LiftedLoop'>
File "det3d/core/bbox/geometry.py", line 193:
def points_in_convex_polygon_jit(points, polygon, clockwise=True):
<source elided>
# vec1: [num_polygon, num_points_of_polygon, 2]
ret = np.zeros((num_points, num_polygons), dtype=np.bool_)
^
@numba.jit
/home/ubuntu/anaconda3/lib/python3.8/site-packages/numba/core/object_mode_passes.py:177: NumbaWarning: Function "points_in_convex_polygon_jit" was compiled in object mode without forceobj=True, but has lifted loops.
File "det3d/core/bbox/geometry.py", line 171:
def points_in_convex_polygon_jit(points, polygon, clockwise=True):
<source elided>
\ # first convert polygon to directed lines
num_points_of_polygon = polygon.shape[1]
^
warnings.warn(errors.NumbaWarning(warn_msg,
/home/ubuntu/anaconda3/lib/python3.8/site-packages/numba/core/object_mode_passes.py:187: NumbaDeprecationWarning:
Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.
For more information visit https://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit
File "det3d/core/bbox/geometry.py", line 171:
def points_in_convex_polygon_jit(points, polygon, clockwise=True):
<source elided>
# first convert polygon to directed lines
num_points_of_polygon = polygon.shape[1]
^
warnings.warn(errors.NumbaDeprecationWarning(msg,
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Error!
Thank you very much for your help the other day, but I can't thank you immediately because my machine is down. I look forward to your reply
liangchunlan
commented
2 years ago I solved this problem by reinstalling CUDA and cudnn.
yancie-yjr
commented
2 years ago I can successfully train and test with v1.0-trainval on pytorch = 1.4.0 and torchvision = 0.5.0. Since I only have one RTX now, I only trained once, and the results are not good as expected. When I want to use the model provided by you, I upgrade pytorch to 1.6.0, but I run as follows. I don't know whether this is correct.
(base) ubuntu@ubuntu-System-Product-Name:/media/ubuntu/nuscenes/code/simtrack$ conda install pytorch=1.6.0 torchvision=0.7.0 torchaudio cudatoolkit=10.2 -c pytorch Collecting package metadata (current_repodata.json): done Solving environment: done # All requested packages already installed. (base) ubuntu@ubuntu-System-Product-Name:/media/ubuntu/nuscenes/code/simtrack$ cd det3d/ops/iou3d_nms/ (base) ubuntu@ubuntu-System-Product-Name:/media/ubuntu/nuscenes/code/simtrack/det3d/ops/iou3d_nms$ python setup.py build_ext --inplace running build_ext building 'iou3d_nms_cuda' extension Emitting ninja build file /media/ubuntu/nuscenes/code/simtrack/det3d/ops/iou3d_nms/build/temp.linux-x86_64-3.8/build.ninja... Compiling objects... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) ninja: no work to do. g++ -pthread -shared -B /home/ubuntu/anaconda3/compiler_compat -L/home/ubuntu/anaconda3/lib -Wl,-rpath=/home/ubuntu/anaconda3/lib -Wl,--no-as-needed -Wl,--sysroot=/ /media/ubuntu/nuscenes/code/simtrack/det3d/ops/iou3d_nms/build/temp.linux-x86_64-3.8/src/iou3d_cpu.o /media/ubuntu/nuscenes/code/simtrack/det3d/ops/iou3d_nms/build/temp.linux-x86_64-3.8/src/iou3d_nms_api.o /media/ubuntu/nuscenes/code/simtrack/det3d/ops/iou3d_nms/build/temp.linux-x86_64-3.8/src/iou3d_nms.o /media/ubuntu/nuscenes/code/simtrack/det3d/ops/iou3d_nms/build/temp.linux-x86_64-3.8/src/iou3d_nms_kernel.o -L/home/ubuntu/anaconda3/lib/python3.8/site-packages/torch/lib -L/usr/local/cuda-10.0/lib64 -lc10 -ltorch -ltorch_cpu -ltorch_python -lcudart -lc10_cuda -ltorch_cuda -o build/lib.linux-x86_64-3.8/iou3d_nms_cuda.cpython-38-x86_64-linux-gnu.so copying build/lib.linux-x86_64-3.8/iou3d_nms_cuda.cpython-38-x86_64-linux-gnu.so -> (base) ubuntu@ubuntu-System-Product-Name:/media/ubuntu/nuscenes/code/simtrack/det3d/ops/iou3d_nms$ cd ../../../ (base) ubuntu@ubuntu-System-Product-Name:/media/ubuntu/nuscenes/code/simtrack$ ls apex examples README.md apex-6af09dd938adf11baedabd89da6cd092d3959081 INSTALL.md requirements.txt det3d LICENSE.md save_dir example.gif model_zoo tools (base) ubuntu@ubuntu-System-Product-Name:/media/ubuntu/nuscenes/code/simtrack$ python ./tools/val_nusc_tracking.py examples/point_pillars/configs/nusc_all_pp_centernet_tracking.py --checkpoint /media/ubuntu/nuscenes/code/simtrack/model_zoo/simtrack_pillar.pth --work_dir /media/ubuntu/nuscenes/code/simtrack/save_dir/simtrack_test --local_rank 0 1.6.0 Use HM Bias: -2.19 ====== Loading NuScenes tables for version v1.0-trainval... Loading nuScenes-lidarseg... 32 category, 8 attribute, 4 visibility, 64386 instance, 12 sensor, 10200 calibrated_sensor, 2631083 ego_pose, 68 log, 850 scene, 34149 sample, 2631083 sample_data, 1166187 sample_annotation, 4 map, 34149 lidarseg, Done loading in 42.830 seconds. ====== Reverse indexing ... Done reverse indexing in 5.9 seconds. ====== Error! Error! Error! Error! Error! Error! /media/ubuntu/nuscenes/code/simtrack/det3d/core/bbox/geometry.py:160: NumbaWarning: Compilation is falling back to object mode WITH looplifting enabled because Function "points_in_convex_polygon_jit" failed type inference due to: No implementation of function Function(<built-in function getitem>) found for signature: >>> getitem(array(float64, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>)) There are 22 candidate implementations: - Of which 20 did not match due to: Overload of function 'getitem': File: <numerous>: Line N/A. With argument(s): '(array(float64, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))': No match. - Of which 2 did not match due to: Overload in function 'GetItemBuffer.generic': File: numba/core/typing/arraydecl.py: Line 162. With argument(s): '(array(float64, 3d, C), Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>))': Rejected as the implementation raised a specific error: TypeError: unsupported array index type list(int64)<iv=None> in Tuple(slice<a:b>, list(int64)<iv=None>, slice<a:b>) raised from /home/ubuntu/anaconda3/lib/python3.8/site-packages/numba/core/typing/arraydecl.py:68 During: typing of intrinsic-call at /media/ubuntu/nuscenes/code/simtrack/det3d/core/bbox/geometry.py (177) File "det3d/core/bbox/geometry.py", line 177: def points_in_convex_polygon_jit(points, polygon, clockwise=True): <source elided> polygon - polygon[ ^ @numba.jit /media/ubuntu/nuscenes/code/simtrack/det3d/core/bbox/geometry.py:160: NumbaWarning: Compilation is falling back to object mode WITHOUT looplifting enabled because Function "points_in_convex_polygon_jit" failed type inference due to: cannot determine Numba type of <class 'numba.core.dispatcher.LiftedLoop'> File "det3d/core/bbox/geometry.py", line 193: def points_in_convex_polygon_jit(points, polygon, clockwise=True): <source elided> # vec1: [num_polygon, num_points_of_polygon, 2] ret = np.zeros((num_points, num_polygons), dtype=np.bool_) ^ @numba.jit /home/ubuntu/anaconda3/lib/python3.8/site-packages/numba/core/object_mode_passes.py:177: NumbaWarning: Function "points_in_convex_polygon_jit" was compiled in object mode without forceobj=True, but has lifted loops. File "det3d/core/bbox/geometry.py", line 171: def points_in_convex_polygon_jit(points, polygon, clockwise=True): <source elided> \ # first convert polygon to directed lines num_points_of_polygon = polygon.shape[1] ^ warnings.warn(errors.NumbaWarning(warn_msg, /home/ubuntu/anaconda3/lib/python3.8/site-packages/numba/core/object_mode_passes.py:187: NumbaDeprecationWarning: Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour. For more information visit https://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit File "det3d/core/bbox/geometry.py", line 171: def points_in_convex_polygon_jit(points, polygon, clockwise=True): <source elided> # first convert polygon to directed lines num_points_of_polygon = polygon.shape[1] ^ warnings.warn(errors.NumbaDeprecationWarning(msg, Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error! Error!
Thank you very much for your help the other day, but I can't thank you immediately because my machine is down. I look forward to your reply
I met the same problem, I still have no idea to solve it.
How to use v1.0-mini of nuscenes dataset? I can train v1.0-mini, but it is not supported during the test. I want to know if there is a solution. How long did you spend training nuscenes dataset? I only have one GPU. Should i set -- nproc per Set to 1? About ./model zoo/simtrack Pillar.pth, I get the following results: Super (_open_zipfile_reader, self) init (torch._C.PyTorchFileReader(name_orbuffer)) RuntimeError: version <= kMaxSupportedFileFormatVersion INTERNAL ASSERT FAILED at /opt/conda/conda-bld/pytorch 1579061855666/work/caffe2/serialize/inline container.cc:132, please report a bug to PyTorch. Attempted to read a PyTorch file with version 3, but the maximum supported version for reading is 2. Your PyTorch installation may be too old. (init at /opt/conda/conda-bld/pytorch_1579061855666/work/caffe2/serialize/inline_container.cc:132) frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x47 (0x7f24ac0a7627 in /home/ubuntu/anaconda3/lib/python3.8/site-packages/torch/lib/libc10.so) Can you provide a trained (pillar based) model with unlimited pytorch version and the specific environment version? Thank you very much for your help.