It is likely that we are going to have several "FFT frames" for a single video frame transition (at let's say 60fps). This means that if we simply read the FFT value at the time of the update, we lose the FFT values before that.
Instead, we could aggregate all the FFT values in between 2 video frames in some meaningful way (such as gradient / average), and use that value instead.
On the other hand, since the audio sampling and the FFT are very fast, the variation in the FFT frames in the video transition might be minimal, therefore making this issue irrelevant.
It is likely that we are going to have several "FFT frames" for a single video frame transition (at let's say 60fps). This means that if we simply read the FFT value at the time of the update, we lose the FFT values before that.
Instead, we could aggregate all the FFT values in between 2 video frames in some meaningful way (such as gradient / average), and use that value instead.
On the other hand, since the audio sampling and the FFT are very fast, the variation in the FFT frames in the video transition might be minimal, therefore making this issue irrelevant.