search

tryolabs / norfair

Lightweight Python library for adding real-time multi-object tracking to any detector.
https://tryolabs.github.io/norfair/
BSD 3-Clause "New" or "Revised" License
2.41k stars 247 forks source link

[FIX][MOTION ESTIMATION] Fix error on homography calculation #278

Closed gfugante closed 10 months ago

gfugante commented 1 year ago

When estimating the motion transformation with the following code:

motion_estimator = MotionEstimator()
while True:
  frame = get_frame()
  detections = detect_frame(frame)
  mask = build_mask(frame, detection)
  coord_transformations = motion_estimator.update(frame=frame, mask=mask)

where frame can be any captured RGB frame and motion_estimation_mask is a mask representing all the bounding box of the people detected in the frame, the following two errors are sometimes raised:

cv2.error: OpenCV(4.8.0) /io/opencv/modules/calib3d/src/fundam.cpp:385: error: (-28:Unknown error code -28) The input arrays should have at least 4 corresponding point sets to calculate Homography in function 'findHomography'

and:

cv2.error: OpenCV(4.8.0) /io/opencv/modules/video/src/lkpyramid.cpp:1260: error: (-215:Assertion failed) (npoints = prevPtsMat.checkVector(2, CV_32F, true)) >= 0 in function 'calc'

This becomes increasingly frequent when there are lots of detections, making the tracker unusable.

The issue can be easily reproduced by istantiating the motion estimator as MotionEstimator(max_points=3), as the cv2.findHomography needs at least 4 points.

gfugante commented 1 year ago

Hi @aguscas! I tried to use the motion estimation a lot more and I found out that it breaks very easily even with the previous fix. Have a look at the new one! It's much more stable as it handles better the _get_sparse_flow function and the transformations_getter: on the second one the new handling is pretty strong, as I delete it and reinstantiate a copy, this has worked very well even in my usage code, where I delete the MotionEstimator when it break to instantiate a new one.

Anyways, the code is now stable even when you cover the camera, wave in front of it or have a mask that covers the entire frame. As said this is the best solution I found, it actually took a long time to get it and if you can break it please let me know how, also because touching these logics got me to Nan distances, and consequently to a very hard exit() in tracker.py, which is game over for the user.

gfugante commented 1 year ago

Hi @aguscas! Did you have any time to look or test this?