H264 test_client <-> Firefox fails with high motion (snow pattern)

[AndrewJDR]

When test_client is a peer connecting to Firefox, high motion video will cause the H264 encoder element or parser (I'm not sure which is at fault) to fail with the following messages:

0:00:05.565335879 23831      0x2087cf0 WARN       codecparsers_h264 gsth264parser.c:692:slice_parse_ref_pic_list_modification_1: error parsing "Reference picture list 0 modification"
0:00:05.565373042 23831      0x2087cf0 WARN       codecparsers_h264 gsth264parser.c:2274:gst_h264_parser_parse_slice_hdr: error parsing "Slice header"
0:00:05.565398128 23831      0x2087cf0 WARN       codecparsers_h264 gsth264parser.c:1578:gst_h264_parse_sps_data: value greater than max. value: 30, max 2
0:00:05.565406793 23831      0x2087cf0 WARN       codecparsers_h264 gsth264parser.c:1811:gst_h264_parse_sps: error parsing "Sequence parameter set"
0:00:05.565413668 23831      0x2087cf0 WARN               h264parse gsth264parse.c:726:gst_h264_parse_process_nal:<parser_h264parse_1> failed to parse SPS:
0:00:05.596761596 23831      0x2087cf0 WARN       codecparsers_h264 gsth264parser.c:1578:gst_h264_parse_sps_data: value greater than max. value: 6, max 2
0:00:05.596787327 23831      0x2087cf0 WARN       codecparsers_h264 gsth264parser.c:1811:gst_h264_parse_sps: error parsing "Sequence parameter set"
0:00:05.596794681 23831      0x2087cf0 WARN               h264parse gsth264parse.c:726:gst_h264_parse_process_nal:<parser_h264parse_1> failed to parse SPS:
0:00:05.629930719 23831      0x2087d40 WARN                 basesrc gstbasesrc.c:2943:gst_base_src_loop:<videotestsrc0> error: Internal data flow error.
0:00:05.629967039 23831      0x2087d40 WARN                 basesrc gstbasesrc.c:2943:gst_base_src_loop:<videotestsrc0> error: streaming task paused, reason not-negotiated (-4)
==== Error message start ====
0:00:05.630067874 23831      0x2087d40 WARN                   queue gstqueue.c:970:gst_queue_handle_sink_event:<send-input-video-queue-1> error: Internal data flow error.
Error in element videotestsrc0.
0:00:05.630095390 23831      0x2087d40 WARN                   queue gstqueue.c:970:gst_queue_handle_sink_event:<send-input-video-queue-1> error: streaming task paused, reason not-negotiated (-4)
Error: Internal data flow error.
Debugging info: gstbasesrc.c(2943): gst_base_src_loop (): /GstPipeline:transport-agent-2/GstBin:bin0/GstVideoTestSrc:videotestsrc0:
streaming task paused, reason not-negotiated (-4)
==== Error message stop ====
==== Error message start ====
Error in element send-input-video-queue-1.
Error: Internal data flow error.
Debugging info: gstqueue.c(970): gst_queue_handle_sink_event (): /GstPipeline:transport-agent-2/GstBin:transport_bin_2/GstBin:send-input-bin-1/GstQueue:send-input-video-queue-1:
streaming task paused, reason not-negotiated (-4)

Here is a patch against test_client to reproduce this issue using a videotestsrc with either "pattern=snow" or "pattern=zone-plate kx2=20 ky2=20 kt=1": https://gist.githubusercontent.com/ajohnson23/bff7d086f7c6477db707/raw/e01646617342513b32a70ca68a1f6e29f70b8a3f/openwebrtc-h264-high-motion-failurecase.patch

To reproduce:

1. Apply patch to latest openwebrtc (62a0bc6501379cbbb3322a69a53ed3d981f8b3af) and build test-client
2. The patch is designed to work against the latest channel_server.js running on localhost:8080, not the one on demo.openwebrtc.org, so please clone latest openwebrtc-examples (66d715643e6fa763a477812911f94d8fe9692fc0) and run "nodejs channel_server.js"
3. Run GST_DEBUG=3 test-client [room_id]
4. In firefox 39, connect to localhost:8080 and join the same room id. Click Call.
5. You should hopefully see errors similar to the ones I'm seeing. If you set the FORCE_H264 define to 0 and rebuild/rerun test-client, that will cause VP8 to be used, which works fine with pattern=snow. Other patterns, such as ball or white, work fine with both VP8 and H264.

This is not a "pattern=snow" only problem -- it happens in a lot of other situations when there is high-motion. This was just the easiest test case I could come up with to show the issue.

A few notes:

• test-client must be linked with -lopenwebrtc_gst,... This is included in the patch but I just wanted to mention it separately. This also means you need to build openwebrtc_gst...
• I also tested this with the latest firefox nightly and get similar results. Firefox nightly seems less reliable in general with WebRTC, sometimes failing to negotiate even with VP8 whereas Firefox 39 works fine.
• After step 5 above, you can set the FORCE_H264 define back to 0 and should find that the snow pattern works fine with VP8.
• Naturally, this patch won't work in Chrome with FORCE_H264 set to 1.

[superdump]

I am fairly sure this is a bug with the frame size in the H.264 headers as written by the encoder. It can use 2, 3 or 4 bytes. x264 uses 4 always. I think openh264enc uses 2 always as written, so it overflows if the frame gets too large, which is precisely the case snow would trigger.