Add automatic frame estimation

[glenne]

It is desirable to get 120fps frames so each frame-to-frame time is less than 1/100th of a second. GIven most NDI cameras are 60fps, adding the ability to estimate the frame between two frames would achieve this goal. The same algorithm could be used to divide the video frames in succession to get 240,480,960 fps. This approach assumes that the boat speed does not change significantly over a 16ms period.

Initial tests show that frame estimation appears indistinguishable from the original. For example, this video constructed from two frames 100ms apart shows 600fps video.

Using opencv for this is a popular solution. Some possible starting points below:

• Opencv provides an opencv.js WASM based version that can run in the browser. This uses html5 canvas I understand.
• Possible react wrapper - https://www.npmjs.com/package/opencv-react
• Python interpolation using opencv
• I asked chatgpt to do it for me: https://chatgpt.com/share/1f762fa7-45dd-4e94-879b-94ce1d1e1066

[glenne]

I tried the opencv solution using calcOpticalFlowFarneback which computes the direction of movement for each pixel in the scene from frame to frame and then moves that pixel a fraction of the amount (e.g. 50%). In testing, it seems that some elements move as expected while others did not. For example, the bow number moved appropriately but the bow ball did not. Presumably the object detector didn't discern the bow ball movement.

[glenne]

Update - The images used for the calcOpticalFlowFarneback test included the finish line. As a result, the area near the finish line was dominated by the fact the finish line had zero movement. Testing again without the finish line in the image the interpolated image seemed reasonable. This approach now seems promising.

  const calculateOpticalFlowBetweenFrames = (frame1: cv.Mat, frame2: cv.Mat): cv.Mat => {
    const frame1Gray = new cv.Mat();
    const frame2Gray = new cv.Mat();
    cv.cvtColor(frame1, frame1Gray, cv.COLOR_RGBA2GRAY);
    cv.cvtColor(frame2, frame2Gray, cv.COLOR_RGBA2GRAY);

    const flow = new cv.Mat();
    cv.calcOpticalFlowFarneback(frame1Gray, frame2Gray, flow, 0.5, 3, 15, 3, 5, 1.2, 0);

    frame1Gray.delete();
    frame2Gray.delete();
    return flow;
  };

  const generateMidFrame = (frame1: cv.Mat, frame2: cv.Mat): cv.Mat => {
    const opticalFlow = calculateOpticalFlowBetweenFrames(frame1, frame2);
    const h = opticalFlow.rows;
    const w = opticalFlow.cols;
    const alpha = -0.50;
    console.log(`h: ${h}, w: ${w}, alpha: ${alpha}`);

    // Prepare maps for remap function
    const map1 = new cv.Mat(h, w, cv.CV_32FC1);
    const map2 = new cv.Mat(h, w, cv.CV_32FC1);
    console.log(JSON.stringify(opticalFlow.floatPtr(0, 0)))

    for (let y = 0; y < h; y++) {
      for (let x = 0; x < w; x++) {
        const flowVec = opticalFlow.floatPtr(y, x);
        // console.log(`(${x}, ${y}) -> (${flowVec[0]}, ${flowVec[1]})`);
        map1.floatPtr(y, x)[0] = x + alpha * flowVec[0];
        map2.floatPtr(y, x)[0] = y + alpha * flowVec[1];
        // console.log(`(${x}, ${y}) -> (${map1.floatPtr(y, x)[0]}, ${map2.floatPtr(y, x)[0]})`);
      }
    }

    console.log('map1/mat2 prepared');

    const interpolatedFrame = new cv.Mat();
    cv.remap(frame1, interpolatedFrame, map1, map2, cv.INTER_LINEAR);

    map1.delete();
    map2.delete();
    opticalFlow.delete();

    return interpolatedFrame;
  };

[glenne]

Another test was done with calcOpticalFlowFarneback where the flow results were processed to reject small movements and average larger movements. The net average movement was then used to adjust the image with similar results to applying the flow to every pixel.

const calculateAverageMotion = (flow, threshold) => {
    let sumX = 0.0;
    let sumY = 0.0;
    let count = 0;
    let max = 0.0;

    for (let y = 0; y < flow.rows; y++) {
      for (let x = 0; x < flow.cols; x++) {
        const flowAtXY = flow.floatPtr(y, x);
        const fx = flowAtXY[0];
        const fy = flowAtXY[1];
        const magnitude = Math.sqrt(fx * fx + fy * fy);
        if (magnitude > max) {
          max = magnitude;
        }
        if (magnitude > threshold) {
          sumX += fx;
          sumY += fy;
          count++;
        }
      }
    }

    if (count > 0) {
      return { x: sumX / count, y: sumY / count, max };
    } else {
      return { x: 0.0, y: 0.0, max: 0.0 };
    }
  }

  const applySceneShift = (frame, motion, percentage) => {
    const M = cv.matFromArray(2, 3, cv.CV_64F, [1, 0, motion.x * percentage, 0, 1, motion.y * percentage]);
    const shifted = new cv.Mat();
    const size = new cv.Size(frame.cols, frame.rows);
    cv.warpAffine(frame, shifted, M, size);
    // shifted.copyTo(frame);
    M.delete();
    // shifted.delete();
    return shifted;
  }

  const calculateOpticalFlowBetweenFrames = (frame1: cv.Mat, frame2: cv.Mat): cv.Mat => {
    const frame1Gray = new cv.Mat();
    const frame2Gray = new cv.Mat();
    cv.cvtColor(frame1, frame1Gray, cv.COLOR_RGBA2GRAY);
    cv.cvtColor(frame2, frame2Gray, cv.COLOR_RGBA2GRAY);

    const flow = new cv.Mat();
    cv.calcOpticalFlowFarneback(frame1Gray, frame2Gray, flow, 0.5, 3, 15, 3, 5, 1.2, 0);

    frame1Gray.delete();
    frame2Gray.delete();
    return flow;
  };

  const generateMidFrame = (frame1: cv.Mat, frame2: cv.Mat): cv.Mat => {
    const flow = calculateOpticalFlowBetweenFrames(frame1, frame2);
    const { max } = calculateAverageMotion(flow, 0);
    console.log(`max: ${max}`);
    const motion = calculateAverageMotion(flow, max * 0.5);
    const interpolatedFrame = applySceneShift(frame1, motion, 0.5);
    console.log(`motion: ${motion.x}, ${motion.y} (${motion.max})`);
    flow.delete();
    return interpolatedFrame;
  };