create_data.py - Githubissues

lmh0501 commented 1 year ago

你好，想问一下，运行python tools/create_data.py nuscenes --root-path ./data/nuscenes --out-dir ./data/nuscenes --extra-tag nuscenes时出现这个问题该怎么解决呢？ C2F42D82-D267-4b0c-B3AD-A78295534D01

yichen928 commented 1 year ago

感谢提问！我没有遇到这个问题，你可以检查一下numpy和numba的版本是否正确，我们使用的是numpy 1.20.0, numba 0.48.0

lmh0501 commented 1 year ago

是一样的。我现在是用nuscenes中的mini数据集，会跟这个有关系嘛？

yichen928 commented 1 year ago

你可以尝试一下把numpy版本降低到1.19.5

lmh0501 commented 1 year ago

好像不大行，我试了numpy和numba的不同版本，不同情况的报错，只有你给的这个版本能运行到上面这一步然后也是会报错

lmh0501 commented 1 year ago

好像不大行，我试了numpy和numba的不同版本，不同情况的报错，只有你给的这个版本能运行到上面这一步，然后也是会报错

我用的单张3090显卡，定位到报错的那行，然后把最后的dtype=np.bool改成dtype=np.int64，代码就能跑起来了，但是构建GT Database这步运行得特别慢，我看了一下大概需要七个小时，这还是mini数据集，是不是不大对啊？

yichen928 commented 1 year ago

我们在完整数据集上确实是需要六七个小时，但是mini数据集我不太确定，你们可以确认一下是不是确实只构建了mini数据集，还是其实是在构建完整数据集。

xtk8532704 commented 5 months ago

同样的问题。

Create GT Database of NuScenesDataset
[                                                  ] 0/28130, elapsed: 0s, ETA:Traceback (most recent call last):
  File "tools/create_data.py", line 225, in <module>
    max_sweeps=args.max_sweeps)
  File "tools/create_data.py", line 67, in nuscenes_data_prep
    f'{out_dir}/{info_prefix}_infos_train.pkl')
  File "/home/palmdong/mmfusion/SparseFusion/tools/data_converter/create_gt_database.py", line 254, in create_groundtruth_database
    point_indices = box_np_ops.points_in_rbbox(points, gt_boxes_3d)
  File "/home/palmdong/mmfusion/SparseFusion/mmdet3d/core/bbox/box_np_ops.py", line 433, in points_in_rbbox
    indices = points_in_convex_polygon_3d_jit(points[:, :3], surfaces)
  File "/home/palmdong/mmfusion/SparseFusion/mmdet3d/core/bbox/box_np_ops.py", line 763, in points_in_convex_polygon_3d_jit
    normal_vec, d, num_surfaces)
TypeError: expected dtype object, got 'numpy.dtype[bool_]'

我的版本：

(sparsefusion) palmdong@473370ef5807:~/mmfusion/SparseFusion$ pip list 
Package                       Version     Editable project location
----------------------------- ----------- ------------------------------------
absl-py                       2.1.0
addict                        2.4.0
anyio                         3.7.1
argon2-cffi                   23.1.0
argon2-cffi-bindings          21.2.0
attrs                         23.2.0
backcall                      0.2.0
beautifulsoup4                4.12.3
black                         23.3.0
bleach                        6.0.0
cachetools                    5.3.3
ccimport                      0.4.2
certifi                       2022.12.7
cffi                          1.15.1
charset-normalizer            3.3.2
click                         8.1.7
comm                          0.1.4
cumm-cu117                    0.4.11
cycler                        0.11.0
Cython                        0.29.33
debugpy                       1.7.0
decorator                     5.1.1
defusedxml                    0.7.1
descartes                     1.1.0
entrypoints                   0.4
exceptiongroup                1.2.1
fastjsonschema                2.19.1
fire                          0.6.0
flake8                        3.9.2
fonttools                     4.38.0
google-auth                   2.29.0
google-auth-oauthlib          0.4.6
grpcio                        1.62.2
idna                          3.7
imageio                       2.31.2
importlib-metadata            6.7.0
importlib-resources           5.12.0
iniconfig                     2.0.0
ipykernel                     6.16.2
ipython                       7.34.0
ipython-genutils              0.2.0
ipywidgets                    8.1.3
jedi                          0.19.1
Jinja2                        3.1.4
joblib                        1.3.2
jsonschema                    4.17.3
jupyter                       1.0.0
jupyter_client                7.4.9
jupyter-console               6.6.3
jupyter_core                  4.12.0
jupyter-server                1.24.0
jupyterlab-pygments           0.2.2
jupyterlab_widgets            3.0.11
kiwisolver                    1.4.5
lark                          1.1.9
llvmlite                      0.31.0
lyft-dataset-sdk              0.0.8
Markdown                      3.4.4
MarkupSafe                    2.1.5
matplotlib                    3.5.2
matplotlib-inline             0.1.6
mccabe                        0.6.1
mistune                       3.0.2
mmcv-full                     1.2.7
mmdet                         2.10.0
mmdet3d                       0.11.0      /home/palmdong/mmfusion/SparseFusion
mmpycocotools                 12.0.3
MultiScaleDeformableAttention 1.0
mypy-extensions               1.0.0
nbclassic                     1.1.0
nbclient                      0.7.4
nbconvert                     7.6.0
nbformat                      5.8.0
nest-asyncio                  1.6.0
networkx                      2.2
ninja                         1.11.1.1
notebook                      6.5.7
notebook_shim                 0.2.4
numba                         0.48.0
numpy                         1.20.0
nuscenes-devkit               1.1.10
oauthlib                      3.2.2
opencv-python-headless        4.9.0.80
packaging                     24.0
pandas                        1.3.5
pandocfilters                 1.5.1
parso                         0.8.4
pathspec                      0.11.2
pccm                          0.4.11
pexpect                       4.9.0
pickleshare                   0.7.5
Pillow                        9.5.0
pip                           22.3.1
pkgutil_resolve_name          1.3.10
platformdirs                  4.0.0
plotly                        5.18.0
pluggy                        1.2.0
plyfile                       0.9
portalocker                   2.7.0
prometheus-client             0.17.1
prompt_toolkit                3.0.45
protobuf                      3.20.3
psutil                        5.9.8
ptyprocess                    0.7.0
pyasn1                        0.5.1
pyasn1-modules                0.3.0
pybind11                      2.12.0
pycocotools                   2.0.7
pycodestyle                   2.7.0
pycparser                     2.21
pyflakes                      2.3.1
Pygments                      2.17.2
pyparsing                     3.1.2
pyquaternion                  0.9.9
pyrsistent                    0.19.3
pytest                        7.4.4
python-dateutil               2.9.0.post0
pytz                          2024.1
PyWavelets                    1.3.0
PyYAML                        6.0.1
pyzmq                         26.0.3
qtconsole                     5.4.4
QtPy                          2.4.1
requests                      2.31.0
requests-oauthlib             2.0.0
rsa                           4.9
scikit-image                  0.19.3
scikit-learn                  1.0.2
scipy                         1.7.3
Send2Trash                    1.8.3
setuptools                    65.6.3
Shapely                       1.8.5
six                           1.16.0
sniffio                       1.3.1
soupsieve                     2.4.1
spconv-cu117                  2.3.3
tenacity                      8.2.3
tensorboard                   2.11.2
tensorboard-data-server       0.6.1
tensorboard-plugin-wit        1.8.1
termcolor                     2.3.0
terminado                     0.17.1
terminaltables                3.1.10
threadpoolctl                 3.1.0
tifffile                      2021.11.2
tinycss2                      1.2.1
tomli                         2.0.1
torch                         1.7.1+cu110
torchaudio                    0.7.2
torchvision                   0.8.2+cu110
tornado                       6.2
tqdm                          4.66.4
traitlets                     5.9.0
trimesh                       2.35.39
typed-ast                     1.5.5
typing_extensions             4.7.1
urllib3                       2.0.7
wcwidth                       0.2.13
webencodings                  0.5.1
websocket-client              1.6.1
Werkzeug                      2.2.3
wheel                         0.38.4
widgetsnbextension            4.0.11
yapf                          0.40.2
zipp                          3.15.0

当尝试按mmdet3d的QA降numpy版本时报错numpy与pycocotools的版本不匹配。 Traceback (most recent call last): File "tools/create_data.py", line 5, in from tools.data_converter import kitti_converter as kitti File "/home/palmdong/mmfusion/SparseFusion/tools/data_converter/kitti_converter.py", line 5, in from mmdet3d.core.bbox import box_np_ops File "/home/palmdong/mmfusion/SparseFusion/mmdet3d/core/init.py", line 1, in from .anchor import # noqa: F401, F403 File "/home/palmdong/mmfusion/SparseFusion/mmdet3d/core/anchor/init.py", line 1, in from mmdet.core.anchor import build_anchor_generator File "/home/palmdong/anaconda3/envs/sparsefusion/lib/python3.7/site-packages/mmdet/core/init.py", line 5, in from .mask import # noqa: F401, F403 File "/home/palmdong/anaconda3/envs/sparsefusion/lib/python3.7/site-packages/mmdet/core/mask/init.py", line 2, in from .structures import BaseInstanceMasks, BitmapMasks, PolygonMasks File "/home/palmdong/anaconda3/envs/sparsefusion/lib/python3.7/site-packages/mmdet/core/mask/structures.py", line 6, in import pycocotools.mask as maskUtils File "/home/palmdong/anaconda3/envs/sparsefusion/lib/python3.7/site-packages/pycocotools/mask.py", line 3, in import pycocotools._mask as _mask File "pycocotools/_mask.pyx", line 1, in init pycocotools._mask ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject Traceback (most recent call last): File "tools/combine_view_info.py", line 7, in from mmdet3d.core import Box3DMode, LiDARInstance3DBoxes File "/home/palmdong/mmfusion/SparseFusion/mmdet3d/core/init.py", line 1, in from .anchor import # noqa: F401, F403 File "/home/palmdong/mmfusion/SparseFusion/mmdet3d/core/anchor/init.py", line 1, in from mmdet.core.anchor import build_anchor_generator File "/home/palmdong/anaconda3/envs/sparsefusion/lib/python3.7/site-packages/mmdet/core/init.py", line 5, in from .mask import # noqa: F401, F403 File "/home/palmdong/anaconda3/envs/sparsefusion/lib/python3.7/site-packages/mmdet/core/mask/init.py", line 2, in from .structures import BaseInstanceMasks, BitmapMasks, PolygonMasks File "/home/palmdong/anaconda3/envs/sparsefusion/lib/python3.7/site-packages/mmdet/core/mask/structures.py", line 6, in import pycocotools.mask as maskUtils File "/home/palmdong/anaconda3/envs/sparsefusion/lib/python3.7/site-packages/pycocotools/mask.py", line 3, in import pycocotools._mask as _mask File "pycocotools/_mask.pyx", line 1, in init pycocotools._mask ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject

xtk8532704 commented 5 months ago

解决了,降版本+需要重新编译mmpycocotools:

pip uninstall numpy; pip install numpy==1.20.0 #需要在mmpycocotools前面装。 pip uninstall mmpycocotools pycocotools; pip install mmpycocotools==12.0.3 --no-cache-dir

想问一嘴为何用numpy==1.20.0,有什么讲究么？

yichen928 / SparseFusion

create_data.py #23