How to visualize optical flow?

[zzmicer]

Hello, I am struggling with optical flow visualization. I am tying to reproduce this visualization. Can you please share what graph did you use to create this tracking?

[sgowroji]

Hi @KastusKalinovski, Could you please elaborate your usecase with clear details. And we have very clear documentation with different solutions. And please let us know which solution are you referring to.

[zzmicer]

Hi @sgowroji, thank you for your response. I have a fixed camera and some object moving from frame to frame. My task is to detect this movement and track it with a bounding box. Now I am looking towards your box-tracking solution, and I am trying to reproduce motion detection part.

[zzmicer]

Firstly, I am trying to visualize optical flow(see attached graph). I get the following error: Input stream "optical_flow" of calculator "[FlowToImageCalculator, input stream: optical_flow, and output stream: output_video]" expects packets of type "mediapipe::OpticalFlowField" but the connected output stream will contain packets of type "mediapipe::RegionFlowFeatureList"

My question is how to use the motion analysis calculator to visualize the optical flow and further, region, where motion occurs?

[yegenzhi]

Hi, @KastusKalinovski you may want to introduce the input image to BoxTrackerCalculator as an input stream and set the following boolean to true in the BoxTrackerCalculatorOptions. Then you can obtain an output stream of the annotated image.

visualize_tracking_data: true visualize_state: true visualize_internal_state: true

[sgowroji]

Hi @KastusKalinovski, Did you get a chance to go through the above comment. Thanks!

[zzmicer]

Hello, @yegenzhi, thank you for your response. I've tried following graph: Here is my code:

input_stream: "VIDEO:input_video"
output_stream: "VIDEO:output_video"

node: {
  calculator: "BoxTrackerCalculator"
  input_stream: "VIDEO:input_video"
  input_stream: "TRACKING:input_video"
  output_stream: "VIZ:output_video"

  node_options: {
    [type.googleapis.com/mediapipe.BoxTrackerCalculatorOptions]: {
      tracker_options: {
        track_step_options {
          track_object_and_camera: true
          tracking_degrees: TRACKING_DEGREE_OBJECT_SCALE
          inlier_spring_force: 0.0
          static_motion_temporal_ratio: 3e-2
        }
      }
        visualize_tracking_data: true
        visualize_state: true
        visualize_internal_state: true
      streaming_track_data_cache_size: 100
    }
  }
}

I get the following error : Failed to run the graph: Input stream "input_video" of calculator "[BoxTrackerCalculator, input streams: <VIDEO:input_video,TRACKING:input_video>, and output stream: VIZ:output_video]" expects packets of type "mediapipe::ImageFrame" but the connected output stream will contain packets of type "[Same Type As mediapipe::TrackingData]"

I'm sorry for being silly, but am I supposed to combine MotionAnalysis, FlowPackager and BoxTracker?

[zzmicer]

I've also tried this combined graph, with visualization booleans enabled, but it just returns the input video without any visualization. Code:

input_stream: "VIDEO:input_video"
output_stream: "VIDEO:output_video"

# Performs motion analysis on an incoming video stream.
node: {
  calculator: "MotionAnalysisCalculator"
  input_stream: "VIDEO:input_video"
  output_stream: "CAMERA:camera_motion"
  output_stream: "FLOW:region_flow"

  node_options: {
    [type.googleapis.com/mediapipe.MotionAnalysisCalculatorOptions]: {
      analysis_options {
        analysis_policy: ANALYSIS_POLICY_CAMERA_MOBILE
        flow_options {
          fast_estimation_min_block_size: 100
          top_inlier_sets: 1
          frac_inlier_error_threshold: 3e-3
          verification_distance: 5.0
          verify_long_feature_acceleration: true
          verify_long_feature_trigger_ratio: 0.1
          tracking_options {
            max_features: 500
            adaptive_extraction_levels: 2
            min_eig_val_settings {
              adaptive_lowest_quality_level: 2e-4
            }
            klt_tracker_implementation: KLT_OPENCV
          }
        }
      }
    }
  }
}

# Reads optical flow fields defined in
# mediapipe/framework/formats/motion/optical_flow_field.h,
# returns a VideoFrame with 2 channels (v_x and v_y), each channel is quantized
# to 0-255.
node: {
  calculator: "FlowPackagerCalculator"
  input_stream: "FLOW:region_flow"
  input_stream: "CAMERA:camera_motion"
  output_stream: "TRACKING:tracking_data"

  node_options: {
    [type.googleapis.com/mediapipe.FlowPackagerCalculatorOptions]: {
      flow_packager_options: {
        binary_tracking_data_support: false
      }
    }
  }
}

# Tracks box positions over time.
node: {
  calculator: "BoxTrackerCalculator"
  input_stream: "TRACKING:tracking_data"
  input_stream: "VIDEO:input_video"
  output_stream: "VIZ:output_video"

  node_options: {
    [type.googleapis.com/mediapipe.BoxTrackerCalculatorOptions]: {
      tracker_options: {
        track_step_options {
          track_object_and_camera: true
          tracking_degrees: TRACKING_DEGREE_OBJECT_SCALE
          inlier_spring_force: 0.0
          static_motion_temporal_ratio: 3e-2
        }
      }
        visualize_tracking_data: true
        visualize_state: true
        visualize_internal_state: true
      streaming_track_data_cache_size: 100
    }
  }
}

Log:

I20210412 13:48:17.204370 25891 demo_run_graph_main.cc:54] Initialize the calculator graph.
I20210412 13:48:17.205118 25891 demo_run_graph_main.cc:58] Initialize the camera or load the video.
[ WARN:0] global /home/zmicer/opencv_build/opencv/modules/videoio/src/cap_gstreamer.cpp (1081) open OpenCV | GStreamer warning: Cannot query video position: status=0, value=-1, duration=-1
I20210412 13:48:19.824127 25891 demo_run_graph_main.cc:79] Start running the calculator graph.
I20210412 13:48:19.824381 25891 demo_run_graph_main.cc:84] Start grabbing and processing frames.
W20210412 13:48:19.824470 25892 motion_analysis_calculator.cc:336] No input video header found. Downstream calculators expecting video headers are likely to fail.
W20210412 13:48:19.851002 25892 region_flow_computation.cc:766] Compiled without OpenCV 3.0 but cv_tracking_algorithm was requested. Falling back to older algorithm
I20210412 13:48:19.851240 25892 motion_analysis_calculator.cc:552] Analyzed frame 1
E20210412 13:48:19.893671 25892 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:19.921649 25892 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:19.960402 25893 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:19.991989 25893 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:20.023031 25894 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:20.056699 25894 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:20.089020 25894 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:20.121562 25894 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:20.157258 25894 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:20.188791 25894 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:20.221220 25894 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:20.257305 25894 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:20.288550 25894 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:20.321455 25892 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:20.357326 25892 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:20.388392 25894 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:20.421864 25893 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:20.456892 25892 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:20.488687 25894 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:20.520169 25892 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:20.556659 25892 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:20.589349 25894 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:20.622043 25894 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:20.656780 25894 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:20.689489 25893 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:20.721148 25894 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:20.756757 25894 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:20.787941 25893 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:20.821660 25894 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:20.857470 25894 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:20.888913 25894 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:20.920120 25894 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:20.956406 25894 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:20.989048 25894 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:21.020762 25894 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:21.056771 25894 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:21.088418 25894 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:21.119925 25894 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:21.156098 25894 region_flow_computation.cc:2623] only cv tracking is supported.
E20210412 13:48:21.188031 25892 region_flow_computation.cc:2623] only cv tracking is supported.

[zzmicer]

@yegenzhi, @sgowroji Thank you for your help and provided flags, I've fixed the following error E20210412 13:48:20.888913 25894 region_flow_computation.cc:2623] only cv tracking is supported. by changing up CV_MAJOR_VERSION flag value in region_flow_computation.cc. Now everything works!

[samlaf]

@KastusKalinovski what do you mean by "changing up CV_MAJOR_VERSION flag value in region_flow_computation.cc"? In my case, CV_MAJOR_VERSION is 4, but region_flow_computation is set to only accept version 3... is this actually necessary? I tried changing all the #if CV_MAJOR_VERSION == 3 to #if CV_MAJOR_VERSION >= 3 and it seems to work so far. I'll start a new issue.

[laosanlyu]

Hello, I am struggling with optical flow visualization. I am tying to reproduce this visualization. Can you please share what graph did you use to create this tracking?

Have you successfully visualize the state of tracking state for each box?

[zzmicer]

Hey, @laosanlyu! Yes, I've extracted motion saliency points, and created a new bounding box around them.

[laosanlyu]

could you please share the graph you made?