search

timmeinhardt / trackformer

Implementation of "TrackFormer: Multi-Object Tracking with Transformers”. [Conference on Computer Vision and Pattern Recognition (CVPR), 2022]
https://arxiv.org/abs/2101.02702
Apache License 2.0
487 stars 113 forks source link

Problems on running VisDrone2019 dataset #109

Open RicTimeMuseum opened 9 months ago

RicTimeMuseum commented 9 months ago

Am suffering from CUDA Error during I was running Trackformer on VisDrone2019 dataset on a server with 3 RTX3090 GPUs: 

Start training
/pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:312: operator(): block: [0,0,0], thread: [34,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:312: operator(): block: [0,0,0], thread: [0,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:312: operator(): block: [0,0,0], thread: [5,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:312: operator(): block: [1,0,0], thread: [27,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
Traceback (most recent call last):
  File "src/train.py", line 357, in 
    train(args)
  File "src/train.py", line 284, in train
    visualizers['train'], args)
  File "/root/autodl-tmp/trackformer/src/trackformer/engine.py", line 128, in train_one_epoch
    loss_dict = criterion(outputs, targets)
  File "/root/miniconda3/envs/tf/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/root/autodl-tmp/trackformer/src/trackformer/models/detr.py", line 406, in forward
    losses.update(self.get_loss(loss, outputs, targets, indices, num_boxes))
  File "/root/autodl-tmp/trackformer/src/trackformer/models/detr.py", line 380, in get_loss
    return loss_map[loss](outputs, targets, indices, num_boxes, **kwargs)
  File "/root/autodl-tmp/trackformer/src/trackformer/models/detr.py", line 252, in loss_labels_focal
    losses['class_error'] = 100 - accuracy(src_logits[idx], target_classes_o)[0]
RuntimeError: CUDA error: device-side assert triggered

My conda environment is listed as below: 

_libgcc_mutex             0.1                        main    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
_openmp_mutex             5.1                       1_gnu    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
argon2-cffi               20.1.0                   pypi_0    pypi
astroid                   2.4.2                    pypi_0    pypi
async-generator           1.10                     pypi_0    pypi
attrs                     19.3.0                   pypi_0    pypi
backcall                  0.2.0                    pypi_0    pypi
bleach                    3.2.3                    pypi_0    pypi
ca-certificates           2023.5.7             hbcca054_0    conda-forge
certifi                   2020.4.5.2               pypi_0    pypi
cffi                      1.14.4                   pypi_0    pypi
chardet                   3.0.4                    pypi_0    pypi
cloudpickle               1.6.0                    pypi_0    pypi
colorama                  0.4.3                    pypi_0    pypi
cycler                    0.10.0                   pypi_0    pypi
cython                    0.29.20                  pypi_0    pypi
decorator                 4.4.2                    pypi_0    pypi
defusedxml                0.6.0                    pypi_0    pypi
docopt                    0.6.2                    pypi_0    pypi
entrypoints               0.3                      pypi_0    pypi
filelock                  3.0.12                   pypi_0    pypi
flake8                    3.8.3                    pypi_0    pypi
flake8-import-order       0.18.1                   pypi_0    pypi
future                    0.18.2                   pypi_0    pypi
gdown                     3.12.2                   pypi_0    pypi
gitdb                     4.0.5                    pypi_0    pypi
gitpython                 3.1.3                    pypi_0    pypi
idna                      2.9                      pypi_0    pypi
imageio                   2.8.0                    pypi_0    pypi
importlib-metadata        1.6.1                    pypi_0    pypi
ipykernel                 5.4.3                    pypi_0    pypi
ipython                   7.19.0                   pypi_0    pypi
ipython-genutils          0.2.0                    pypi_0    pypi
ipywidgets                7.6.3                    pypi_0    pypi
isort                     5.6.4                    pypi_0    pypi
jedi                      0.18.0                   pypi_0    pypi
jinja2                    2.11.2                   pypi_0    pypi
jsonpatch                 1.25                     pypi_0    pypi
jsonpickle                1.4.1                    pypi_0    pypi
jsonpointer               2.0                      pypi_0    pypi
jsonschema                3.2.0                    pypi_0    pypi
jupyter                   1.0.0                    pypi_0    pypi
jupyter-client            6.1.11                   pypi_0    pypi
jupyter-console           6.2.0                    pypi_0    pypi
jupyter-core              4.7.0                    pypi_0    pypi
jupyterlab-pygments       0.1.2                    pypi_0    pypi
jupyterlab-widgets        1.0.0                    pypi_0    pypi
kiwisolver                1.2.0                    pypi_0    pypi
lap                       0.4.0           py37hfb7772e_1005    conda-forge
lapsolver                 1.1.0                    pypi_0    pypi
lazy-object-proxy         1.4.3                    pypi_0    pypi
ld_impl_linux-64          2.38                 h1181459_1    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
libblas                   3.9.0           15_linux64_openblas    conda-forge
libcblas                  3.9.0           15_linux64_openblas    conda-forge
libffi                    3.4.4                h6a678d5_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
libgcc-ng                 11.2.0               h1234567_1    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
libgfortran-ng            13.1.0               h69a702a_0    conda-forge
libgfortran5              13.1.0               h15d22d2_0    conda-forge
libgomp                   11.2.0               h1234567_1    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
liblapack                 3.9.0           15_linux64_openblas    conda-forge
libopenblas               0.3.20          pthreads_h78a6416_0    conda-forge
libstdcxx-ng              11.2.0               h1234567_1    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
markupsafe                1.1.1                    pypi_0    pypi
matplotlib                3.2.1                    pypi_0    pypi
mccabe                    0.6.1                    pypi_0    pypi
mistune                   0.8.4                    pypi_0    pypi
more-itertools            8.4.0                    pypi_0    pypi
motmetrics                1.2.0                    pypi_0    pypi
multiscaledeformableattention 1.0                      pypi_0    pypi
munch                     2.5.0                    pypi_0    pypi
nbclient                  0.5.1                    pypi_0    pypi
nbconvert                 6.0.7                    pypi_0    pypi
nbformat                  5.1.2                    pypi_0    pypi
ncurses                   6.4                  h6a678d5_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
nest-asyncio              1.5.1                    pypi_0    pypi
networkx                  2.4                      pypi_0    pypi
ninja                     1.10.0.post2             pypi_0    pypi
notebook                  6.2.0                    pypi_0    pypi
numpy                     1.18.5                   pypi_0    pypi
opencv-python             4.2.0.34                 pypi_0    pypi
openssl                   1.1.1u               h7f8727e_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
packaging                 20.4                     pypi_0    pypi
pandas                    1.0.5                    pypi_0    pypi
pandocfilters             1.4.3                    pypi_0    pypi
parso                     0.8.1                    pypi_0    pypi
pexpect                   4.8.0                    pypi_0    pypi
pickleshare               0.7.5                    pypi_0    pypi
pillow                    7.1.2                    pypi_0    pypi
pip                       22.3.1           py37h06a4308_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
pluggy                    0.13.1                   pypi_0    pypi
prometheus-client         0.9.0                    pypi_0    pypi
prompt-toolkit            3.0.14                   pypi_0    pypi
ptyprocess                0.7.0                    pypi_0    pypi
py                        1.8.2                    pypi_0    pypi
py-cpuinfo                6.0.0                    pypi_0    pypi
pyaml                     20.4.0                   pypi_0    pypi
pycocotools               2.0                      pypi_0    pypi
pycodestyle               2.6.0                    pypi_0    pypi
pycparser                 2.20                     pypi_0    pypi
pyflakes                  2.2.0                    pypi_0    pypi
pygments                  2.7.4                    pypi_0    pypi
pylint                    2.6.0                    pypi_0    pypi
pyparsing                 2.4.7                    pypi_0    pypi
pyrsistent                0.17.3                   pypi_0    pypi
pysocks                   1.7.1                    pypi_0    pypi
pytest                    5.4.3                    pypi_0    pypi
pytest-benchmark          3.2.3                    pypi_0    pypi
python                    3.7.16               h7a1cb2a_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
python-dateutil           2.8.1                    pypi_0    pypi
python_abi                3.7                     2_cp37m    conda-forge
pytz                      2020.1                   pypi_0    pypi
pywavelets                1.1.1                    pypi_0    pypi
pyyaml                    5.3.1                    pypi_0    pypi
pyzmq                     19.0.1                   pypi_0    pypi
qtconsole                 5.0.2                    pypi_0    pypi
qtpy                      1.9.0                    pypi_0    pypi
readline                  8.2                  h5eee18b_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
requests                  2.23.0                   pypi_0    pypi
sacred                    0.8.1                    pypi_0    pypi
scikit-image              0.17.2                   pypi_0    pypi
scipy                     1.4.1                    pypi_0    pypi
seaborn                   0.10.1                   pypi_0    pypi
send2trash                1.5.0                    pypi_0    pypi
setuptools                65.6.3           py37h06a4308_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
six                       1.15.0                   pypi_0    pypi
smmap                     3.0.4                    pypi_0    pypi
sqlite                    3.41.2               h5eee18b_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
submitit                  1.1.5                    pypi_0    pypi
terminado                 0.9.2                    pypi_0    pypi
testpath                  0.4.4                    pypi_0    pypi
tifffile                  2020.6.3                 pypi_0    pypi
tk                        8.6.12               h1ccaba5_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
toml                      0.10.2                   pypi_0    pypi
torch                     1.8.1+cu111              pypi_0    pypi
torchaudio                0.8.1                    pypi_0    pypi
torchfile                 0.1.0                    pypi_0    pypi
torchvision               0.9.1+cu111              pypi_0    pypi
tornado                   6.1                      pypi_0    pypi
tqdm                      4.46.1                   pypi_0    pypi
traitlets                 5.0.5                    pypi_0    pypi
typed-ast                 1.4.1                    pypi_0    pypi
typing-extensions         3.7.4.3                  pypi_0    pypi
urllib3                   1.25.9                   pypi_0    pypi
visdom                    0.1.8.9                  pypi_0    pypi
wcwidth                   0.2.5                    pypi_0    pypi
webencodings              0.5.1                    pypi_0    pypi
websocket-client          0.57.0                   pypi_0    pypi
wheel                     0.38.4           py37h06a4308_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
widgetsnbextension        3.5.1                    pypi_0    pypi
wrapt                     1.12.1                   pypi_0    pypi
xmltodict                 0.12.0                   pypi_0    pypi
xz                        5.4.2                h5eee18b_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
zipp                      3.1.0                    pypi_0    pypi
zlib                      1.2.13               h5eee18b_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main