Implementing "anomaly" detection

I would like to implement a detector that can track anything that moves across multiple frames, and also build a semi-supervised neural network that can train itself on a collection of false positives.

Essentially, if a moving object is detected by a compute efficient motion detection kernel, the frames would then be passed through an additional filter (an LSTM or RNN) to try and filter out false positives. The second pass is intended to filter things such as bugs flying past the camera, switching to day/night vision, heavy rain, strong winds, or sudden changes in luminosity. I'm sure there are other things I'm forgetting... If a detection event is raised but it is a false positive, the recording can be flagged as a false positive. This would kick off a retraining of the neural network. I would like the kernel to have close to real-time performance on a Jetson Nano, if possible.

I could use help identifying the best algorithm(s) needed for something like this. Is the basic motion detector good enough or would optical flow be better?

Anyone interested?

elsampsa / valkka-live

Implementing "anomaly" detection #9