Reducing noise: looking for recommendations

[lucaperrin]

Hello there,

Thanks for this great TD plugin,

I am using a PTZ camera connected in NDI and I am experiencing a pretty high level of noise in the overlay rendering of tracking points. I am looking for recommendations in order to reduce the noise without compromising too much on response time ?

I experimented with a Guassian and One Euro filter CHOP but didn't achieve satisfying results.

Thanks in advance!

[domisjustanumber]

Hey @lucaperrin that sounds like a fun setup!

Which tracking model are you seeing noise in?

[lucaperrin]

Hi @domisjustanumber, I am experiencing some noise in the pose_tracking model. Even when my video flux is paused, the tracking points are still a bit jittery.

[domisjustanumber]

Ahh yeah, the single person pose tracking is pretty solid, but multi-person is extremely noisy. I already added a One Euro filter to the output, and it does help a lot (and is specifically designed for this sort of low-latency smoothing), but the underlying AI model is just not very good for multi-person tracking.

For my last project I ended up using RTMO instead. It's extremely good for multi-person tracking (2D, no hands or feet), but the learning curve to get it running is... steep. If you are very comfortable with Python programming, I can give you some pointers, but it's not an easy task.

[lucaperrin]

Thanks for this but I am still working on developing my python capabilities...

I am actually experiencing some noise with the single person pose tracking.

For my current project, I am trying to move a PTZ camera to keep a single point of interest (torso for example) in the center of the frame. I am getting closer to sucessfully achieve my objective but the noise is causing the camera to never fully stabilize on the target as the tracking point never really stops moving.

Now that you mention RTMO, any plans to develop a custom plugin for TouchDesigner ?

[domisjustanumber]

Ahh I understand what you're doing now. It's beyond the scope of MediaPipe, but in that case I'd suggest a lag CHOP to heavily smooth out the tracking data and/or some kind of threshold system so the camera only moves if there's more than x amount of change in the value for more than y amount of time. Depending on your use case, the PTZ camera probably doesn't need to be super responsive and/or you might be able to shoot a wider shot so the camera doesn't need to be as responsive.

RTMO... eh maybe? It's a Python library, so needs an entirely different pipeline/integration than MediaPipe. It (and a number of other machine vision libraries) do some really cool things, so it would be nice to find a way to integrate them, so we shall see how much time I have to play haha

[lucaperrin]

I'll try adding a lag CHOP and see how it improves my system. I'll keep you posted. I'm also comparing with the Nvidia Body Track CHOP and seeing which one is more relevant for my application.

Thanks again for the help and for this awesome free TD plugin!

[domisjustanumber]

You're more than welcome, and good luck! I'd love to see what you come up with too!