H264 video access unit delimeters missing causes ExoPlayer to get stuck in preparing state

[micolous]

udpxy is a simple multicast UDP to HTTP proxy service. It takes a multicast UDP stream (encapsulated as RTP MPEG-TS), strips off the RTP and pushes out raw MPEG-TS on the other side.

The service doesn't do anything fancy like HLS or DASH in order to improve stream, you just get an infinite MPEG-TS stream on the other side. You can even record it with wget and play back with mplayer:

$ wget http://fnog:8000/udp/233.XXX.XXX.XXX:1234/ -O test.ts
--2016-02-15 00:38:14--  http://fnog:8000/udp/233.XXX.XXX.XXX:1234/
Resolving fnog (fnog)... 172.20.XXX.XXX
Connecting to fnog (fnog)|172.20.XXX.XXX|:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/octet-stream]
Saving to: 'test.ts’

test.ts            [                    <=>          ]   5.50M   165KB/s             ^C

$ mplayer test.ts
Playing test.ts.
Detected file format: MPEG-TS (MPEG-2 Transport Stream) (libavformat)
[h264 @ 0x7fb6a10a6a40]non-existing PPS 0 referenced
[h264 @ 0x7fb6a10a6a40]non-existing PPS 0 referenced
[h264 @ 0x7fb6a10a6a40]decode_slice_header error
[h264 @ 0x7fb6a10a6a40]no frame!
...
LAVF: Program 1 
Selected video codec: H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 [libavcodec]
Selected audio codec: MPEG 1.0/2.0/2.5 layers I, II, III [mpg123]
AUDIO: 48000 Hz, 2 ch, s16le, 256.0 kbit/16.67% (ratio: 32000->192000)

When using the demo application to view the stream in TYPE_OTHER mode, the player is stuck in playbackState=preparing. It appears to be buffering the stream, but doesn't get any further into the actual playback of the stream after this.

Eventually, the buffering stops (after around three minutes), and playback never starts. The UI still says "preparing"...

vlc and totem (gstreamer) seem to be quite happy playing this same stream. mplayer crashes on the live stream (another bug), but will play back a previously recorded stream.

[ojw28]

You'll need to provide more information if we're to look at this. We'll probably need an accessible test stream. Thanks.

[micolous]

I'm trying to build a portable pipeline as I don't have a way to give you access to the streams in question. I also don't have access to the encoding pipelines on the other side, as they come from an IPTV provider.

However, my own test streams appear to work correctly locally. I'm still trying to understand what the difference between my streams and the provider's streams are.

I'm having difficulty muxing an audio track into the test stream, and I suspect this may be what causes ExoPlayer to fail.

For reference, this was the test pipeline I built using gstreamer:

gst-launch-1.0 \
    videotestsrc \
    ! video/x-raw,format=I420,width=720,height=576,framerate=25/1,pixel-aspect-ratio=1/1 \
    ! x264enc bitrate=2000 \
    ! mux. \
    \
    mpegtsmux name=mux \
    ! rtpmp2tpay \
    ! udpsink host=233.1.1.1 port=1234 auto-multicast=true

Then udpxy was started with:

udpxy -p 8000 -v

Then the URL in the client looks like (demo/Samples.java):

new Sample("IPTV", "http://172.20.0.238:8000/udp/233.1.1.1:1234/", Util.TYPE_OTHER),

[micolous]

Still not broken yet, even with audio:

gst-launch-1.0 \
    videotestsrc \
    ! video/x-raw,format=I420,width=720,height=576,framerate=25/1,pixel-aspect-ratio=\(fraction\)64/45 \
    ! x264enc bitrate=2000 \
    ! mux. \
    \
    audiotestsrc \
    ! audio/x-raw,format=S16LE,channels=1,layout=interleaved,rate=44100 \
    ! twolamemp2enc \
    ! mpegaudioparse \
    ! mpegtsmux name=mux \
    ! rtpmp2tpay \
    ! udpsink host=233.1.1.1 port=1234 auto-multicast=true

Interestingly looking at the metadata for the different streams, this is one from a udpxy'd stream from the IPTV provider:

$ ffmpeg -i test.ts 
Input #0, mpegts, from 'test.ts':
  Duration: 00:00:23.33, start: 35812.197589, bitrate: 1985 kb/s
  Program 1 
    Stream #0:0[0xe0]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p, 720x576 [SAR 64:45 DAR 16:9], 25 fps, 25 tbr, 90k tbn, 180k tbc
    Stream #0:1[0xc0]: Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, s16p, 256 kb/s

$ tsinfo test.ts Reading from test.ts
Scanning 1000 TS packets

Packet 219 is PAT
Program list:
    Program 1 -> PID 0020 (32)

Packet 220 is PMT with PID 0020 (32)
  Program 1, version 2, PCR PID 00e0 (224)
  Program streams:
    PID 00e0 ( 224) -> Stream type 1b ( 27) H.264/14496-10 video (MPEG-4/AVC)
    PID 00c0 ( 192) -> Stream type 03 (  3) 11172-3 audio (MPEG-1)

Found 4 PAT packets and 2 PMT packets in 1000 TS packets

Compared to my stream:

$ ffmpeg -i homebrew3.ts
Input #0, mpegts, from 'homebrew3.ts':
  Duration: 00:00:10.09, start: 3649.345300, bitrate: 2290 kb/s
  Program 1 
    Stream #0:0[0x41]: Video: h264 (High) (HDMV / 0x564D4448), yuv420p(tv), 720x576 [SAR 64:45 DAR 16:9], 25 fps, 25 tbr, 90k tbn, 50 tbc
    Stream #0:1[0x42]: Audio: mp2 ([3][0][0][0] / 0x0003), 44100 Hz, mono, s16p, 192 kb/s

$ tsinfo homebrew3.ts 
Reading from homebrew3.ts
Scanning 1000 TS packets

Packet 60 is PAT
Program list:
    Program 1 -> PID 0020 (32)

Packet 61 is PMT with PID 0020 (32)
  Program 1, version 0, PCR PID 0041 (65)
     Program info (12 bytes): 05 04 48 44 4d 56 88 04 0f ff fc fc
     Registration HDMV
     Descriptor tag 88 (136) (4 bytes): 0f ff fc fc
  Program streams:
    PID 0041 (  65) -> Stream type 1b ( 27) H.264/14496-10 video (MPEG-4/AVC)
        ES info (10 bytes): 05 08 48 44 4d 56 ff 1b 44 3f
        Registration HDMV ff 1b 44 3f
    PID 0042 (  66) -> Stream type 03 (  3) 11172-3 audio (MPEG-1)

Found 12 PAT packets and 6 PMT packets in 1000 TS packets

I'm not sure if there's any obvious smoking gun there as to what is strange about the provider's stream that is causing ExoPlayer to fail.

I did also try using SimpleHTTPServer to host the ts files in question, homebrew3.ts (my file) plays reliably, but test.ts (the file I recorded from the provider) fails in the same manner that streaming did.

As a result, I think I have a portable test case, which I will send through via email to you tomorrow morning, when I have access to a computer with my work email account.

[ojw28]

Please send to dev.exoplayer@gmail.com when you get round to it; thanks.

[micolous]

Files sent.

[louaybassbouss]

I have similar issue. I am using ExoPlayer r1.5.5.

[ojw28]

The issue with the test4.ts file is that the H264 stream doesn't contain any access unit delimiters. We currently rely on the existence of access unit delimiters to split the stream up into frames. They aren't strictly required, according to the spec. The audio in the file is audio/mpeg-L2, which I don't think many Android devices include a decoder for.

Given the latter point and depending on what devices you're targeting, you may need to do some transcoding work regardless to get the audio into a suitable format. In which case you should probably insert AUDs into the video stream as part of that process.

I'm unsure whether it's worth the effort for us to support H264 streams that don't contain AUDs. Any thoughts @andrewlewis (context: this is in ts.H264Reader)?

[micolous]

I updated the title because it's evidently not a udpxy issue now, and other udpxy streams seem to work fine when correctly formatted.

I should clarify my use case -- I'm not affiliated with the IPTV provider at all, and I'm interested in writing an Android TV client to play this. I think transcoding the stream would be impractical, and severely limit the utility of the software.

The provider is unlikely to fix their streams if I were to report it to them, as it is likely generated by some appliance in their datacentre. The provider recommends playback with VLC, where playback works without issue.

The video component of the stream in question does play with the Android MediaPlayer API, provided I can fill it's buffers quickly enough. In the case of using udpxy, I've been having the stream open through udpxy on my PC with VLC, then opening the stream using the MediaPlayer API. This allows MediaPlayer to buffer immediately, and have a chance at playing video.

MediaPlayer also plays the ts files fine from the sdcard, albeit without audio.

However, the buffering of MediaPlayer is poor and unreliable (for live streaming), which is why I turned to ExoPlayer given it is billed as a MediaPlayer replacement.

I'm not sure how to address the audio component of the stream. Though on that point alone I may have to try another video playback system which has some software codecs in it.

I was really hoping on a way I could integrate this with the Android TV Live Channels API, but at this point I think it may be too difficult.

[andrewlewis]

It should be possible to alter H264Reader so it doesn't rely on AUDs to identify samples. This would be consistent with our H265Reader (though detecting the start of an access unit is easier for that format). I'll take a look at implementing this.

Regarding the audio stream, if you can find a suitable software codec it may be possible to use that with ExoPlayer. The opus extension is an example of doing this. (Tangentially, we are currently working on reducing the amount of boilerplate code needed to implement extensions like this, but don't have a firm ETA yet.)

[micolous]

Ah, cool.

I ended up getting this running in Kodi's Live TV API, and it took me less than an hour to do (compared to 12+ hours I've spent on trying to integrate this with Android TV properly). The sum total of code I wrote for that was a small Python script which converted the playlist file to the correct format for Kodi's standard IPTV plugins to work, and rewrite the URLs so it worked with udpxy.

I would like proper Android TV integration, but it is very difficult for me to pull off right now. It looks like I'll have to wait for H264Reader to get fixed up, and an API to bundle software audio codecs.

[ojw28]

You can now work around the issue described here using the WORKAROUND_DETECT_ACCESS_UNITS flag in TsExtractor.

[micolous]

Cool, I'll have to have another play with this another day and see how this works.

I'll have a try talking to the provider about their audio codec selection but I don't like my chances.