mfxExtPictureTimingSEI and interlaced

[mliu2020]

This is a question from IDZ forum about mfxExtPictureTimingSEI() and I believe this is a good place to ask:

Customer has following scenario: "I'm using mfxExtPictureTimingSEI on the mfxEncodeCtrl struct that I send to MFXVideoENCODE_EncodeFrameAsync, in order to insert pic_timing info on the frames.

For progressive video, everything works OK, but for interlaced, both fields have identical pic_timing values, while I need the time_offset of the first field to be 0, and the second field to be 1.

I've noticed that mfxExtPictureTimingSEI has 3 timestamp structs, but whatever I set on the two latter doesn't seem to have any effect. Only the first timestamp is used for both fields.

Can you help me find out if there's a way to set different data on each field? I wouldn't like to modify the bytes after encoding to find the pic_timing and change the 0 for a 1."

"- What's the point of having the timestamp as an array of three timestamps? Only TimeStamp[0] seems to be used. What should I set in TimeStamp[1] and TimeStamp[2]. The manual doesn't say anything about it, and I couldn't find it used on the samples.

• How can I make top and bottom field to have different values? They both have the values set in TimeStamp[0]."

Mark

[dmitryermilov]

@mliu2020 , it seems it's about old https://community.intel.com/t5/Media-Intel-oneAPI-Video/H264-Timecode-absent-in-QSV/m-p/950913

[mliu2020]

Thanks Dmitry,

I read through all the post and scenario is similar. If my understanding is correct, mfxExtPictureTimingSEI is not very relevant to this case. The customer in new post seems more confused by mfxExtPictureTimingSEI.

• What's the point of having the timestamp as an array of three timestamps? Only TimeStamp[0] seems to be used. What should I set in TimeStamp[1] and TimeStamp[2]. The manual doesn't say anything about it, and I couldn't find it used on the samples.

I have forwarded this post to customer.

Mark

[Fernickk]

Hello, I'm the "customer" ^_^. Thank you for taking the time to address my issue.

In my case, the timecode is applied to the hardware-encoded video. It's just a case of a single parameter (time_offset), that I can't generate as I like.

I think I understand now the need for the three timestamps on the struct, reading the H264 standard (ISO 14496-10). The pic_struct allows a single frame to be displayed multiple times consecutively (up to three times), and each of those times uses one of the elements on the TimeStamp array. This however is not my case, as every frame/field should be displayed only once.

Whatever I put on the TimeStamp[0] of the mfxExtPictureTimingSEI is applied to the encoded output. That is, when I add this to a frame to carry the timecode 10:01:14:23:

timestamp.ClockTimestampFlag = 1;
timestamp.CtType = isProgressive ? 0 : 1; // Progressive or interlaced
timestamp.NuitFieldBasedFlag = 1;
timestamp.CountingType = 1;
timestamp.FullTimestampFlag = 1;
timestamp.DiscontinuityFlag = 0;
timestamp.CntDroppedFlag = 0;
timestamp.NFrames = 23;
timestamp.SecondsFlag = 1;
timestamp.MinutesFlag = 1;
timestamp.HoursFlag = 1;
timestamp.SecondsValue = 14;
timestamp.MinutesValue = 1;
timestamp.HoursValue = 10;
timestamp.TimeOffset = 0;

For progressive, I get on the pic_timing of every frame:

pic_struct = 0 (frame)
clock_timestamp_flag = 1
ct_type = 0
nuit_field_based_flag = 1
counting_type = 1
full_timestamp_flag = 1
discontinuity_flag = 0
cnt_dropped_flag = 0
n_frames = 23
seconds_value = 14
minutes_value = 1
hours_value = 10
time_offset = 0

For interlaced, every frame has two pic_timing's, one for each field. The top field has:

pic_struct = 1 (top field)
clock_timestamp_flag = 1
ct_type = 0
nuit_field_based_flag = 1
counting_type = 1
full_timestamp_flag = 1
discontinuity_flag = 0
cnt_dropped_flag = 0
n_frames = 23
seconds_value = 14
minutes_value = 1
hours_value = 10
time_offset = 0

and the bottom field has:

pic_struct = 2 (bottom field)
clock_timestamp_flag = 1
ct_type = 0
nuit_field_based_flag = 1
counting_type = 1
full_timestamp_flag = 1
discontinuity_flag = 0
cnt_dropped_flag = 0
n_frames = 23
seconds_value = 14
minutes_value = 1
hours_value = 10
time_offset = 0

See that the pic_struct is generated by the encoder, and the rest of the fields are taken from what I set on mfxExtPictureTimingSEI for that particular frame.

What I'm trying to do here, is to set a different time_offset for the second field (top or bottom depending on if it's bottom field first or top field first). I want the first field to have time_offset = 0 and the second field to have time_offset = 1. But the encoder takes the info I put in mfxExtPictureTimingSEI and applies to both fields.

I tried to add two mfxExtPictureTimingSEI to the frame, setting ExtParam as an array of two mfxExtPictureTimingSEI*, and NumExtParam = 2, and in that case the encoding of the frame gave me a MFX_WRN_INCOMPATIBLE_VIDEO_PARAM warning, and only the first mfxExtPictureTimingSEI was used.

Setting TimeOffset as 0xFFFF sets the time_offset to 255, besides the MFX_WRN_INCOMPATIBLE_VIDEO_PARAM warning.

So, in conclusion: is there a way to tell the encoder to automatically increase time_offset, as it sets the pic_struct value automatically? Or do I have to resort to dig through the encoded data, find the pic_timing and edit the byte in post?

Thanks in advance.

[mliu2020]

Hi @Fernickk,

I understand your request better now, If you see the old post in IDZ from @dmitryermilov , it has following statement: "For picture timing, if I insert two payload data in the mfxExtBuffer (instead of only one) in an interlaced resolution, it works for SW and QSV H264 encoder. I see the timecode in playback."

Is it the same request as yours?

Mark Liu

[dmitryermilov]

Hi @Fernickk ,

I tried to add two mfxExtPictureTimingSEI to the frame, setting ExtParam as an array of two mfxExtPictureTimingSEI*, and NumExtParam = 2, and in that case the encoding of the frame gave me a MFX_WRN_INCOMPATIBLE_VIDEO_PARAM warning, and only the first mfxExtPictureTimingSEI was used.

This is an issue of encode implementation. I mean the solution would be to allow application to pass a pair of mfxExtPictureTimingSEI per frame. For some mfxExt buffers it's allowed but not mfxExtPictureTimingSEI. The old post was about a similar problem. I think we can draft a fix this or next week.

[mliu2020]

Hi @dmitryermilov,

From the post, it seems like for Interlaced stream on hardware decodec, there is an issue to mfxEctPictureTimingSEI which doesn't allow a pair of this structure per frame.

Do you have an update to this issue?

Mark

[daleksan]

Hi @kovakimy can you take a look on this request?

[kovakimy]

Hi @Fernickk, Could you, please, tell me which platform, OS you are running on? Could you also, please, share me reproducer?

Thanks & Regards, Kristina