Add pixel count to pre-verification

Background

At the moment, a LP broadcaster computes the # of pixels in a result by:

Decoding the video
Iterating through the list of decoded frames
For each frame, calculating the # of pixels in the frame by multiplying the height & width of the frame
Summing the pixel count of each frame in the video

The broadcaster compares the pixel count that it calculates with the pixel count reported by the orchestrator in order to ensure that the orchestrator is being honest with its pixel count reporting. It is important that the broadcaster can check the pixel count reported by an orchestrator because the orchestrator uses its reported pixel count to charge the broadcaster for transcoding - payment value is calculated based on the # of output pixels encoded by the orchestrator in the results (in the future we'll likely incorporate the # of input pixels decoded by the orchestrator in the source).

The broadcaster currently uses a pixel count implementation that relies on ffmpeg C libraries included in LPMS (Livepeer media server).

When we integrate verification (using the verifier API in this repo) into the LP broadcaster, there will be some redundant computation being done because the broadcaster will decode results from orchestrators once for pixel counting and another time for video content verification (since the verifier needs to extract features from the decoded frames of the video).

Feature

We can implement the pixel count check as a part of pre-verification and return it in the API response. Since the verifier will already be decoding results for feature extraction we can remove the extra decoding step that the broadcaster currently runs for pixel counting. The broadcaster can just delegate the pixel count check to the verifier.

Currently, the API request object looks like:

{
    ...
    "renditions: [
        {
            "uri": ...,
            "resolution": {},
            "frame_rate": ...
        }
    ]
}

We can update the API request object to:

{
    ...
    "renditions: [
        {
            "uri": ...,
            "resolution": {},
            "frame_rate": ...,
            "pixels": ...
        }
    ]
}

The API response object would be updated to include a field indicating if the pixel count check passed or failed.

Pixel count has been added using OpenCV's VideoCapture CAP_PROP_FRAME_COUNT, CAP_PROP_FRAME_HEIGHT and CAP_PROP_FRAME_WIDTH. As it is currently implemented, OpenCV is using the default Gstreamer backend, which has tools to read those properties directly from the metadata of the video capture. However, this may be subject to tampering provided that this data is not necessarily coincident with the contents of the capture. For this reason, a post-verification step has been implemented that takes the actual values from the decoding step for both resolution as pixel count.

In this manner, the current API call:

curl localhost:5000/verify -d '{
   "source":"https://storage.googleapis.com/livepeer-verifier-renditions/480p/-3MYFnEaYu4.mp4",
   "renditions":[
      {
         "uri":"https://storage.googleapis.com/livepeer-verifier-renditions/144p_black_and_white/-3MYFnEaYu4.mp4"
      },
      {
         "uri":"https://storage.googleapis.com/livepeer-verifier-renditions/144p/-3MYFnEaYu4.mp4",
         "resolution":{
            "height":"144",
            "width":"256"
         },
         "frame_rate":"24",
         "pixels":"1034500"
      }
   ],
   "model":"https://storage.googleapis.com/verification-models/verification.tar.xz"
}' 
-H 'Content-Type: application/json'

Returns the following:

[
{
   "https://storage.googleapis.com/livepeer-verifier-renditions/144p_black_and_white/-3MYFnEaYu4.mp4":{
      "tamper":-3.619838,
      "uri":"https://storage.googleapis.com/livepeer-verifier-renditions/144p_black_and_white/-3MYFnEaYu4.mp4"
   }
},
{
   "https://storage.googleapis.com/livepeer-verifier-renditions/144p/-3MYFnEaYu4.mp4":{
      "frame_rate":false,
      "pixels":"1034500",
      "pixels_post_verification":0.09354202835648148,
      "pixels_pre_verification":127119360.0,
      "resolution":{
         "height":"144",
         "height_post_verification":1.0,
         "height_pre_verification":1.0,
         "width":"256",
         "width_post_verification":1.0,
         "width_pre_verification":1.0
      },
      "tamper":1.431959,
      "uri":"https://storage.googleapis.com/livepeer-verifier-renditions/144p/-3MYFnEaYu4.mp4"
   }
}
]

Where the pre_verification values are the ratios between the required parameters in the call and the extracted values from the meta data (using Gstreamer) and the post verification values are the same ratio, extracted in the decoding stage.

Just curious if these numbers have been sanity checked against what our transcoder produces? Since it's a different implementation, there is always the possibility of small differences (eg, frame counts not matching). We have a transcoder sample program that prints out the number of pixels encoded, which can be checked against the decode count here.

livepeer / verification-classifier

Add pixel count to pre-verification #53

Background

Feature