ifzhang / ByteTrack

[ECCV 2022] ByteTrack: Multi-Object Tracking by Associating Every Detection Box
MIT License
4.69k stars 891 forks source link

DeepStream tracker library causes OOM error when compiled with -DCMAKE_BUILD_TYPE=Release option #253

Closed Klomi closed 3 months ago

Klomi commented 2 years ago

Description Build libByteTracker.so as per the instruction given in (README.md), but replace cmake .. with cmake -DCMAKE_BUILD_TYPE=Release to enable optimizations. Running a GStreamer pipeline with the built library file takes up all system memory immediately and crashes with OOM error.

To Reproduce

  1. Launch Docker container, set the memory limit to avoid screwing up the host system.
    docker run --rm -it --gpus=all --memory=4G nvcr.io/nvidia/deepstream:6.1-devel
  2. Install dependencies.
    apt update && apt install -y libeigen3-dev
  3. Clone repo and build libByteTracker.so.
    cd /root
    git clone --depth 1 https://github.com/ifzhang/ByteTrack/
    cd ByteTrack/deploy/DeepStream
    mkdir build && cd build
    # cmake -> /usr/local/bin/cmake version is 3.15.2
    # /usr/bin/cmake version is 3.16.3
    /usr/bin/cmake -DCMAKE_BUILD_TYPE=Release ..
    make
  4. Run GStreamer pipeline.
    gst-launch-1.0 -v \
    filesrc location=/opt/nvidia/deepstream/deepstream-6.1/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder \
    ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 \
    ! nvinfer config-file-path=/opt/nvidia/deepstream/deepstream-6.1/samples/configs/deepstream-app/config_infer_primary.txt \
    ! nvtracker \
        ll-lib-file=/root/ByteTrack/deploy/DeepStream/lib/libByteTracker.so \
        tracker-width=640 \
        tracker-height=384 \
    ! nvvideoconvert ! nvdsosd ! fpsdisplaysink sync=0 video-sink=fakesink text-overlay=0
  5. Observe memory usage. After a while, the program is killed and exit code is 137.

Additional Information The issue is probably caused by missing return value in NvMOTContext::processFrame, which is an undefined behaviour that could lead to bad compiler optimizations. https://github.com/ifzhang/ByteTrack/blob/72ca8b45d36caf5a39e949c6aa815d9abffd1ab5/deploy/DeepStream/src/NvMOTContext.cpp#L9-L56 Simply append return NvMOTStatus_OK; at the end of the function, rebuild the library, and the issue is fixed (tested on my local machine).

EmpireofKings commented 2 years ago

FYI appending return NvMOTStatus_OK; to the end of NvMOTContext::processFrame does not fix the memory leak.