Support per-frame QP configuration by VideoEncoder

[daijh]

Customize BRCs by applications, in addition to CBR/VBR provided by VideoEncoder.

Some applications prefer to implement its own BRC according to per-frame QP, to better response to network bandwidth estimation. One such per-frame QP BRC implement can be found: https://github.com/intel/libmebo/blob/master/explainer.md

dictionary VideoEncoderEncodeOptions {
  boolean keyFrame = false;

  // proposal
  unsigned long QP;
};

[daijh]

56 Refine encoder configuration of bitrate and/or quantization

[chcunningham]

Thanks for filing @daijh. Can you say more on why per-frame is important vs making this part of the config (affecting N frames). I'm surprised we would want to adjust it with every frame (e.g. 30 times a second).

[daijh]

@chcunningham The BRC is something intrinsic of the video encoder, and it usually cannot be much changed after platform shipped. Some developers would like to use their own BRC cross all platforms, instead of platform intrinsic BRC. Per-frame QP enables the developer to implement customized BRC, by set QP of every frame according to previous encoded frames size and bitrate budget, as mentioned in #56

[Djuffin]

As I see it we’ll need to make at least 3 following changes in the API. I'm curious to know what the community thinks

New bitrate mode.

• We’ll introduce a new value to BitrateMode enum : ‘qp’

  QP range discovery. Either

  1. A new optional field to VideoEncoderConfig dictionary : ULongRange qpRange This range is populated by VideoEncoder.isConfiguSupported() when an input config has bitrateMode=’qp’. It gives API users a range of expected QP values. VideoEncoder.configure() expects that config.qpRange value will be the same as the one returned by VideoEncoder.isConfiguSupported(). This works as an extra check that API users are aware of the expected QP range.
  2. QP range is always assumed to be between 1 and 100, and it is up to the implementation to scale it to values expected by underlying encoders.

     Setting a QP value. Either

  3. A new optional field in VideoEncoderEncodeOptions dictionary : unsigned long qp. API users can set a new QP value while submitting a new frame to VideoEncoder.encode(). If a new QP value is not given, it’s assumed to be the same as the previous frame.
  4. A new optional field in VideoEncoderConfig dictionary unsigned long qp. API users will need to reconfigure a video encoder by calling VideoEncoder.configure() each time they want to change the QP value.

It also might be a good idea to replace QP by a more generic term.

[aboba]

Some thoughts:

1. For setting the QP value, it seems like option 1 (using encode()) is easier than option 2 (configure()).
2. For the QP range, I have a mild preference for handling it on a per-codec basis (option 1).

[chcunningham]

Editors Call discussion w/ @aboba

Thanks @Djuffin for moving this along!

For bitrate mode: Could we just use "VBR" here? I like reasoning about both cases as "variable".

Related questions:

• should we define the behavior where setting per-frame QP comes to override any previously configured config.bitrate? (do underlying libraries support combining these things? My hunch is it's probably either-or).

For range discovery:

• I also like 1 better. I wonder if we need to return a supported range, or if we can instead get by with a static codec specific range defined in each codec registration?
• If range-support is impl specific, would you support throwing an error (rather than silent failure) for unsupported values.
• Do you know the semantic meaning of QP across codecs. For ex, is it always as simple as a single value? I've heard AV1 may have different quantizer knobs for y, u, and v (a matrix)? I expect that most codecs have a common notion, but if we find codec-specific the details have enough variance, we may prefer to have codec specific knob (e.g. config.avc.qp vs config.av1.qp[Y_PLANE]) as a way to cleanly define the semantics.

For setting a QP value, I also prefer option 1. This is a per-frame config, so doing it per-frame feels right.

[chcunningham]

Chrome folks met just now. Notes:

For bitrate mode: Could we just use "VBR" here? I like reasoning about both cases as "variable".

Simplicity of just doing VBR is clear, but also this is modeled on AVCodec (Mac), which has a distinct "mode" for setting QP. No super strong opinions though.

should we define the behavior where setting per-frame QP comes to override any previously configured config.bitrate? (do underlying libraries support combining these things? My hunch is it's probably either-or).

@Djuffin assumed they would be exclusive. Combining is an error.

I also like 1 better. I wonder if we need to return a supported range, or if we can instead get by with a static codec specific range defined in each codec registration?

Some preference for codec-specific route w/ ranges in registry.

If range-support is impl specific, would you support throwing an error (rather than silent failure) for unsupported values.

Throw an unsupported error.

Do you know the semantic meaning of QP across codecs....

Seems like another good reason to consider codec specific route (w/ codec specific names if appropriate).

[dalecurtis]

I spoke with James Zern from the encoding team today:

• He indicated that QP based configuration is uncommon, but necessary in the long tail of support. I.e., lets ensure we have engaged partners before undertaking this.
• He indicated there will be more than just QP parameters needed in the long tail, so codec specific dictionaries sounded good. Especially for real time tuning.
• Offloading the standardization of the parameters to the respective codec standards and codec registrations sounded good.
• He indicated most folks will probably just want a simple quality selector or some concept of presets. E.g., we could define a set of low, medium, high, ultra presets in each codec registration.

[Djuffin]

API proposal: https://github.com/w3c/webcodecs/pull/633

This is PR introduces codec specific quantizer fields in VideoEncoderEncodeOptions dictionary.

dictionary VideoEncoderEncodeOptions {
  boolean keyFrame;
  VideoEncoderEncodeOptionsForVp9 vp9;
  VideoEncoderEncodeOptionsForAv1 av1;
};

dictionary VideoEncoderEncodeOptionsForVp9 {
 unsigned short? quantizer;
}

[Djuffin]

Alternative API https://github.com/w3c/webcodecs/pull/642

This is PR introduces a generic quantizer field in VideoEncoderEncodeOptions dictionary.

dictionary VideoEncoderEncodeOptions {
  boolean keyFrame;
  double? quantizer;
};

[Djuffin]

On March 7th w3c media working group decided that codec specific quantizer fields are preferred. API proposal: https://github.com/w3c/webcodecs/pull/633

Rationale:

1. There might be codecs in the future that don't have the concept of QP
2. Having qp values for different codecs in different places will prevent websites from accidentally using qp value from one codec for another codec.

[aboba]

@dalecurtis Can we close this issue?

[dalecurtis]

Now landed.