bitrateType.eControlRate = OMX_Video_ControlRateConstant not implemented properly

[sdrsdr]

Hi I was having issues with a project of mine trying to do CBR H264 encoding. The encoder could not enter Idle state. I tried to change the provided example in /opt/vc/src/hello_pi/hello_encode/encode.c to do CBR, simple change on line 187

   bitrateType.eControlRate = OMX_Video_ControlRateConstant;

The hello_encode.bin compiles but it now hangs the same way as my code. So I assume this is a error in firmware somewhere. If the parameter is not supported this should be indicated in the SetParameter call. State changing should not block :)

[sdrsdr]

ping?

[F5OEO]

Have you find a solution ?..I have the same issue with Raspivid (which remove the code to set the control bitrate mode) and BCM2835-V4L2 driver. How can we know if it is implemented ?

[popcornmix]

I've dug into the code and:

     /* Prevent CBR in H.264 high profile - CABAC prevents us doing QP changes between MBs */
     if ((enc->params.Enc_Standard == VIDEO_ENCODER_STANDARD_H264) && (enc->params.profile == PROFILE_HIGH) && (enc->params.rc_unit == RC_UNIT_MB))
        level_valid = 0;

So I think you can't have CBR and cabac. Try adding this:

   OMX_VIDEO_PARAM_PROFILELEVELTYPE profile;
   memset(&profile, 0, sizeof(OMX_VIDEO_PARAM_PROFILELEVELTYPE));
   profile.nSize = sizeof(OMX_VIDEO_PARAM_PROFILELEVELTYPE);
   profile.nVersion.nVersion = OMX_VERSION;
   profile.nPortIndex = 201;
   profile.eProfile = OMX_VIDEO_AVCProfileBaseline;
   profile.eLevel = OMX_VIDEO_AVCLevel3;
   r = OMX_SetParameter(ILC_GET_HANDLE(video_encode),
            OMX_IndexParamVideoProfileLevelCurrent, &profile);   
   if (r != OMX_ErrorNone) {
      printf
     ("%s:%d: OMX_SetParameter() for video_encode port 201 failed with %x!\n",
      __FUNCTION__, __LINE__, r);
      exit(1);
   }

Just before the bitrate is set.

[6by9]

I think I've just found the reason for the lockup and asked @popcornmix to review.

As he has quoted, CBR in high profile is seemingly not supported, but it should now correctly cleanup and throw an error.

[F5OEO]

Thanks in digging in this issue. I will give it a try. I am still surprise that Profile High is not supported by the codec with CBR. It is a very common feature on H264 live encoders. Maybe this codec is focused on mass storage content and thus oriented VBR ?

[6by9]

VBR was certainly the more normal mode of operation that we saw the codec used in, though I think Miracast used CBR. The codec team aren't around any more to give more detail, so we can only go off the comment in the code unless the Pi Foundation have the complete chip datasheet, or the detail of supported codecs/encodings.

[popcornmix]

The hang when providing invalid settings should be fixed with rpi-update code. Unfortunately CBR with high profile is not going to be supported.

[F5OEO]

Thanks for update. CBR with High profile should be convenient...no way to help to implement it ? Other topic : Is there any plan to support B frame ? It could also improve quality ...Right now setting B Frame hang.

[6by9]

No, with Broadcom's decision we no longer have the experts on the codecs around to even tell us why (I suspect it is because CABAC is pipelined as a separate processing step after the main encode, so having to adjust QP would need to stall the pipe. Overall latency for 1080P was something like 40ms, so that would then mean no 1080P30). B-frames - the hardware doesn't support it, so not possible on 2835.

[F5OEO]

I still work on CBR, it seems to work with low IDR Periode (12). If I put 25 , bitrate is less constant. I would like to play wiith OMX_VIDEO_CONFIG_LEVEL_EXTEND and specially nCustomMaxBRandCPB. it is a 32 bit parameter which is in 2 parts : MaxBitrate in 1000b/s unit but what about CPB. How the two fields are encoded in the 4 Bytes parameter ?

[6by9]

Where's the confusion here?

The level sets up the max permitted bitrate as per http://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Levels. As per http://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Decoded_picture_buffering, this also controls the size of the codec picture buffer.

nCustomMaxBRandCPB allows you to specify a max bitrate greater than that specified by the level, and in doing so will alter the size of the buffer.

[brunogm0]

6by9 maybe issue #337 can interfere in encoding? Can you test it?

[6by9]

No link at all that I can think of.

337 is the decoder setting up how many images to allocate for DECODING images into. This issue is due to a design decision for CBR not to be supported in High profile due to CABAC making it not feasible (exact reason not totally clear, but it is clearly stated in the firmware source).

Why do you think those are linked?!

[brunogm0]

Well, basically all encoders have the decoder inside in loop. So it can make decisions based on what the information is available only on the user decoder. CABAC engines have most of its performance penalty from memory issues and the need to serialize output. So the link is obvious encoding needs more memory than decoding but as i pointed out and you mentioned DPB relation to CBR mode, there is at least 2 buffers or 6MB of wasted memory in the decode part. More if using less than 4REF frames.

[6by9]

Still irrelevant in this case. Yes, the encoder has to use the decode hardware block to reconstruct the reference frame, but as only a single reference frame is used for encode (no B-frame support), there are always a total two buffers allocated(*). The encoder does not reuse the whole decoder software stack - there is no need to re-decode the final stream, so you save the processing requirement for the CABAC decode and similar.

@F5OEO Are you happy now with what can and can't be done with respect CBR, and what nCustomMaxBRandCPB is nominally doing? If so, please close this issue.

(*) In theory I think it could be done with just one, but allocating two means that the pipe doesn't have to be reconfigured to do things like skippable P-frames where the new reconstructed frame is to be ignored and an older frame retained as reference.

[brunogm0]

In theory 337# would not be showing unexplained behavior. So if you tested and has proof on the amount of buffers in the codec firmware? can find an explanation to the extra buffers?

Yes the final CABAC is not decoded in loop in a simplistic rate control. But i found some patents on Broadcom CABAC engines and their reason is with a variable rate they can parallelize more. But they showed this engine block running at 42MHz built on 180nm tech. A revision of the control software to nowadays silicon can enable CABAC.

[6by9]

No I haven't run any tests., but I do have the firmware codec source in front of me. The encoder allocates two image buffers if immutable input is set (if not set, then it reuses the source image to store the recon frame - not great if it's on the display at the time). No additional buffers allocated beyond that.

I was not the codec expert at Broadcom, but I do happen to work with one of them. I have just pestered him. As suspected, the encoder is pipelined and CABAC is done effectively as a post processing step. So encode and reconstruction image for frame N is done at the same time as CABAC is being done on frame N-1. Rate control therefore doesn't know how many bits it took to encode frame N until the end of frame N+1. The interpretation taken for CBR (rightly or wrongly) was constant per frame. In which case it needs to know the number of bits taken to encode frame N before it can proceed to N+1. That precludes using CABAC due to the above pipeline. VBR will keep the bitrate at the target bitrate over a period of a second or so, and using roughly constant quality.

That is the situation. Without significant effort it won't be changing. Feel free to hypothesize about what might be possible with current silicon technologies, but it isn't going to change - BCM2835 is what you have on the Pi, and it was released about 4 years ago in 40nm.

(I can see no reason why rate control ever needs to decode the CABAC bitstream again. Rate control wants to look at the final bitstream. Recon frames are independent of CABAC.)

That's my final word on any link with #337. There simply isn't one. Please stick to #337 for discussion of your issue with extra buffers in video decode. I may look at it if I find some time, but I do this voluntarily, and camera and video encode are my main areas of knowledge.

[brunogm0]

I dont want to put you personally in any way responsible. Sorry if i sounded that way.

To fix or change this Broadcom has to be involved so lets leave to someone who willing to pay for it.

(RDO optimization looks to final CABAC "sizes" to mode decision and so on)

[Ruffio]

@sdrsdr is this still an issue?

[sdrsdr]

I have no interest in this any more. :( Can't test sorry