alternate audio track support

[mangui]

test streams: http://chkhu.connectmedia.hu/6094/index.m3u8 Duna Word HD (1280x720) http://service02-live.connectmedia.hu/6176/index.m3u8 Euronews 704x396 http://service02-live.connectmedia.hu/6119/index.m3u8

http://encode1.hk1.tvb.com/stream/silver/playlist.m3u8

[mangui]

• create 2 FLV Tags queue : one for audio , another one for video
• create a mixer that will unstack and inject into NetStream tags from both queues when NetStream buffer becomes empty
• derive AlternateFragmentLoader from FragmentLoader and use it to load alternate audio fragments

[mangui]

should come soon, stay tuned

[scan]

Staying tuned!

[mangui]

196 has been merged in dev branch

[mangui]

you can now check with latest swf/swc

[yverbin]

I have a problem with playing a live hls streams with an alternative audiotracks. Chunks both audio and video stored in ts format, these streams are normally played on ipad, iphone and using OSMF HLS Plugin. During playback, sometimes the values of audioBufferLength and videoBufferLength are close to zero, which leads to video/audio freezes.

[mangui]

Hi @yverbin i never had the opportunity to test alt-audio with live streams... so i cannot guarantee the results. could you provide debug/debug2 logs of your issue ? then eventually i would need an access to your streams as well, cheers, Mangui

[mangui]

i rethinked about it and definitely there is a potential flaw in the design for live streams with altaudio tracks. there is only one variable to track audio and video live playlist sliding.

the issue is that video and audio live playlists might not be synchronized, depending on the playlist reloading timing, meaning that relative position in the live audio and video playlists might not link to the same timestamps.

https://github.com/mangui/flashls/blob/dev/src/org/mangui/hls/stream/StreamBuffer.as#L150

_time_sliding variable would need to be adjusted to take into account potential offsets betweeen audio and video : two sliding variables needs to be defined : one for audio playlist and another one for video.

this will need some rework in StreamBuffer.as each time _time_sliding variable is used. i will check how this could be handled.

Cheers, Mangui

[yverbin]

Please tell me when you need the test live streams. And, how i can send it to you ?

[mangui]

curl https://api.github.com/users/mangui/events/public | grep email

[ydelwiche]

I'm up for some extra testing as well!

[jlacivita]

hello. I'm testing 0.4.0.4 w/ Apple's advanced HLS test stream: http://devimages.apple.com.edgekey.net/streaming/examples/bipbop_16x9/bipbop_16x9_variant.m3u8

using flashlsOSMF

And I see getAlternativeAudioItemAt() working (although info.language isn't set, only info.title) I'm not hearing the tracks switch when i call switchAlternativeAudioIndex. I'm just getting silence after calling it.

Does 0.4.0.4 actually support alternate audio, or are we waiting for another update based on your comments above?

[mangui]

Hi @jlacivita there are some identified issues with live alt/audio, but VoD alt/audio should work with flashls/dev (some issues have been fixed in /dev since 0.4.0.4) I didn't recheck with this 'complex' apple stream. could you check whether you also observe the issue with Chromeless Player / dev ? (just to nail down the pb) Cheers, Mangui

[jlacivita]

Hi,

It works in Chromeless fine, as well as Grind.

I'll keep digging.

[jlacivita]

The only thing i noticed was that language is set to null. not sure if that's because it's missing from Apple's HLS file, or if we need a table to translate from HLS language names to OSMF names. It'd be nice to have in there if the data is available.

[mangui]

@yverbin @MTYannick i did some changes to support altaudio live playlists, these are blind changes as I don't have any live altaudio playlist in hands. could you check @ your hand with flashls/dev ? and report any issue(s) ? thanks Cheers, Mangui

[yverbin]

i have try to play live stream without success - player is hung with message "initialization". After the stream has been closed by EXT-X-ENDLIST tags, all works fine.

I will try to keep the stream opens for you. Debug logs: Failed to load resource: the server responded with a status of 404 (Not Found) (program):1 818 DEBUG [object MediatorMap] [object TypeFilter] mapped to object MediatorMapping:1 822 DEBUG [object MediatorMap] [object TypeFilter] mapped to object MediatorMapping:1 830 DEBUG [object MediatorMap] [object TypeFilter] mapped to object MediatorMapping:1 832 DEBUG [object MediatorMap] [object TypeFilter] mapped to object MediatorMapping:1 833 DEBUG [object MediatorMap] [object TypeFilter] mapped to object MediatorMapping:1 834 DEBUG [object MediatorMap] [object TypeFilter] mapped to object MediatorMapping:1 835 DEBUG [object MediatorMap] [object TypeFilter] mapped to object MediatorMapping:1 837 DEBUG [object MediatorMap] [object TypeFilter] mapped to object MediatorMapping:1 838 DEBUG [object MediatorMap] [object TypeFilter] mapped to object MediatorMapping:1 839 DEBUG [object MediatorMap] [object TypeFilter] mapped to object MediatorMapping:1 840 DEBUG [object MediatorMap] [object TypeFilter] mapped to object MediatorMapping:1 841 DEBUG [object MediatorMap] [object TypeFilter] mapped to object MediatorMapping:1 844 DEBUG [object MediatorMap] [object TypeFilter] mapped to object MediatorMapping:1 845 DEBUG [object MediatorMap] [object TypeFilter] mapped to object MediatorMapping:1 847 DEBUG [object MediatorMap] [object TypeFilter] mapped to object MediatorMapping:1 848 DEBUG [object MediatorMap] [object TypeFilter] mapped to object MediatorMapping:1 849 DEBUG [object MediatorMap] [object TypeFilter] mapped to object MediatorMapping:1 850 DEBUG [object MediatorMap] [object TypeFilter] mapped to object MediatorMapping:1 851 DEBUG [object MediatorMap] [object TypeFilter] mapped to object MediatorMapping:1 852 DEBUG [object MediatorMap] [object TypeFilter] mapped to object MediatorMapping:1 853 DEBUG [object MediatorMap] [object TypeFilter] mapped to object MediatorMapping:1 854 DEBUG [object ConfigManager] Now initializing. Injecting into config object object ContextView:1 854 INFO Context-0-f1 Initialize complete GrindPlayer.html?src=http%3A%2F%2F80.251.113.51%2Fhls%2Fpublisher%2F3591.m3u8:61 volumeChange Objectmuted: falsevolume: 0.41proto: ObjectdefineGetter: function defineGetter() { [native code] }defineSetter: function defineSetter() { [native code] }lookupGetter: function lookupGetter() { [native code] }lookupSetter: function lookupSetter() { [native code] }constructor: function Object() { [native code] }hasOwnProperty: function hasOwnProperty() { [native code] }isPrototypeOf: function isPrototypeOf() { [native code] }propertyIsEnumerable: function propertyIsEnumerable() { [native code] }toLocaleString: function toLocaleString() { [native code] }toString: function toString() { [native code] }valueOf: function valueOf() { [native code] }get proto: function proto() { [native code] }set proto: function proto() { [native code] } (program):1 INFO:HLSSettings.maxBufferLength = 300 (program):1 INFO:HLSSettings.logDebug2 = true (program):1 INFO:HLSSettings.flushLiveURLCache = false (program):1 INFO:HLSSettings.startFromLevel = -1 (program):1 INFO:HLSSettings.seekFromLevel = -1 (program):1 INFO:HLSSettings.logDebug = true (program):1 INFO:HLSSettings.minBufferLength = -1 (program):1 INFO:HLSSettings.maxLevelCappingMode = downscale (program):1 INFO:HLSSettings.logWarn = true (program):1 INFO:HLSSettings.logError = true (program):1 INFO:HLSSettings.logInfo = true (program):1 INFO:HLSSettings.manifestLoadMaxRetry = -1 (program):1 INFO:HLSSettings.capLevelToStage = false (program):1 INFO:HLSSettings.fragmentLoadMaxRetry = -1 (program):1 INFO:HLSSettings.seekMode = KEYFRAME (program):1 INFO:HLSSettings.lowBufferLength = 3 (program):1 INFO:HLSNetStream:close (program):1 DEBUG:cancel any manifest load in progress (program):1 DEBUG:adaptive playlist:

EXTM3U

EXT-X-VERSION:4

EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="audio",LANGUAGE="ru",NAME="Russian",AUTOSELECT=YES,DEFAULT=YES,URI="3591/audio_l.m3u8"

EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="audio",LANGUAGE="en",NAME="English",AUTOSELECT=YES,DEFAULT=NO,URI="3591/audio_r.m3u8"

EXT-X-STREAM-INF:PROGRAM-ID=1,AUDIO="audio",DEFAULT=NO,BANDWIDTH=98304

3591/160p_wa.m3u8

EXT-X-STREAM-INF:PROGRAM-ID=1,AUDIO="audio",DEFAULT=NO,BANDWIDTH=548864

3591/360p_wa.m3u8

EXT-X-STREAM-INF:PROGRAM-ID=1,AUDIO="audio",DEFAULT=NO,BANDWIDTH=393216

3591/240p_wa.m3u8

EXT-X-STREAM-INF:PROGRAM-ID=1,AUDIO="audio",DEFAULT=YES,BANDWIDTH=3072000

3591/source_wa.m3u8

EXT-X-STREAM-INF:PROGRAM-ID=1,AUDIO="audio",DEFAULT=NO,BANDWIDTH=1011712

3591/480p_wa.m3u8

(program):1 DEBUG:alternate audio level found (program):1 DEBUG:2 alternate audio tracks found GrindPlayer.html?src=http%3A%2F%2F80.251.113.51%2Fhls%2Fpublisher%2F3591.m3u8:61 loading undefined (program):1 DEBUG:level 0 playlist:

EXTM3U

EXT-X-VERSION:4

EXT-X-MEDIA-SEQUENCE:0

EXT-X-TARGETDURATION:5

EXTINF:5.0667,

16883/0.ts

EXTINF:5.0000,

16883/1.ts

EXTINF:5.0000,

16883/2.ts

EXTINF:5.0000,

16883/3.ts

EXTINF:5.0000,

16883/4.ts

EXTINF:5.0000,

16883/5.ts {cropped}

(program):1 DEBUG:updateFragments: unknown PTS info for this level (program):1 DEBUG:Level 0 Live Playlist parsing finished: reload in NaN ms

[yverbin]

Sorry, fixed in 427f2733230d90619db1a04cf33b5dcb5028ed8e

[yverbin]

problem is still here, but reproduced much less

[mangui]

ok , i rechecked with your stream and it is working fine. plz check with latest flashls/dev i did some other fixes. if you still observe some issues let me know

[yverbin]

the problem appears even less, but still here. scenario: run player with my stream, do not touch anything, and after about 2-3 minutes audio-video freeze for about 5 seconds sometimes.

[jlacivita]

One thing I'm seeing in OSMF: the MediaPlayer.hasAlternativeAudio isn't set to true until right after the MediaPlayerStateChangeEvent is fired with MediaPlayerState.PLAYING

This is problematic, because that's when we typical check for all the attributes of a Media.

Is there anyway to get this set slightly earlier?

[mangui]

@jlacivita , strange. this event should be fired when the AlternativeAudioTrait is created.

https://github.com/willwh/osmf/blob/fac113c5fb7d6724992247be39106e1caa00fcfa/framework/OSMF/org/osmf/media/MediaPlayer.as#L1723

https://github.com/willwh/osmf/blob/fac113c5fb7d6724992247be39106e1caa00fcfa/framework/OSMF/org/osmf/media/MediaPlayer.as#L1607

could you dump the backtrace when this state is changing ?

[dtniland]

Hello mangui, I work with jlacivita and am working with the alternativeAudio in your hls plugin. Your code has been great to work with so far, so thanks. jlacivita's comment was about some work I've been doing. The issue is that we're keying off of MediaPlayerState.PLAYING to tell the rest of our app that alternative audio is present, but when we look at hasAlternativeAudio at that instant the value is false. If we listen for: MediaPlayerCapabilityChangeEvent.HAS_ALTERNATIVE_AUDIO_CHANGE, it fires just miliseconds later, but by then it's too late: we've already told our app that there was no alternative audio.

Looking at the stack traces, the relevant spot seems to be at:

at org.osmf.media::MediaPlayer/updateTraitListeners()[/org/osmf/media/MediaPlayer.as:1502] at org.osmf.media::MediaPlayer/onTraitAdd()[/org/osmf/media/MediaPlayer.as:1469] at flash.events::EventDispatcher/dispatchEventFunction() at flash.events::EventDispatcher/dispatchEvent() at org.osmf.elements::ProxyElement/processTraitsChangeEvent()[/org/osmf/elements/ProxyElement.as:475] at org.osmf.elements::ProxyElement/onTraitAdd()[/org/osmf/elements/ProxyElement.as:436] at flash.events::EventDispatcher/dispatchEventFunction() at flash.events::EventDispatcher/dispatchEvent() at org.osmf.media::MediaElement/setLocalTrait()[/org/osmf/media/MediaElement.as:647] at org.osmf.media::MediaElement/addTrait()[/org/osmf/media/MediaElement.as:424] at org.mangui.osmf.plugins::HLSMediaElement/initTraits()

After that, the path that leads to the PLAYING state change event is: at org.osmf.media::MediaPlayer/play() while the path that leads to the HAS_ALTERNATIVE_AUDIO_CHANGE is: at org.osmf.media::MediaPlayer/updateCapabilityForTrait()

It seems to be a simple matter of order -- the play trait is hit first. I'm not intimately familiar with the inner workings of OSMF so I don't know how hard it would be, but is there anything you can do to affect that sequence so that the .hasAlternativeAudio is already set to true when the PLAYING state is fired? I realize that there may be parse order issues that are special to what you're trying to do, but the f4m files that we use seem to handle this in the order we need.

Thanks dtniland

[mangui]

Hi @dtniland plz recheck with flashls/dev. after this change, SMP and GrindPlayer are still working as expected... but it might improve your case.

[dtniland]

@mangui yes thank you, the trait order change worked. The hasAlternativeAudio property is now set correctly. I've noticed another issue, however. When we set mediaPlayer.switchAlternativeAudioIndex(0) to get the alternate audio, the switch works correctly (if we seek to the currentPosition at the same time) but the mediaPlayer.currentAlternativeAudioStreamIndex gets set to -2 and will stay there no matter what we do afterwards. Looking through the call stack debugger, it seems like there's an internal variable: _activeTransitionIndex that is doing the real works of switching, but the variables _lastTransitionIndex and _indexToSwitchTo somehow stay stuck at -2, and these are surfaced to the mediaPlayer.

Sorry to continue this but there is another issue: the switching property of the AlternativeAudioEvent.AUDIO_SWITCHING_CHANGE event is always false, both the first time it is called and the second. One would assume the first time the event fires, the property would be true.

We can work around all of these issues by keeping the state ourselves outside of your event structure, but that can lead to complications of state, especially when mixed with hitting alternative audio from an f4m file that gets handled by OSMF itself.

[mangui]

@dtniland you might recheck your first issue with latest flashls/dev there was indeed something wrong with _lastTransitionIndex for the second issue, i m not sure as this event is fired by OSMF itself, but i would suggest to recheck if this issue is still there after my changes cheers, mangui

[dtniland]

Thanks, the stream index is fixed! The switching is still always false, but it's not as serious an issue...

[mangui]

closing, alt-audio tracks are now supported

[jlacivita]

yay!

[Prendo93]

What is the behaviour when there is audio muxed into the video stream AND alternate audio is provided?

Testing that here shows that it will not play: http://www.flashls.org/latest/examples/chromeless/?src=http%3A%2F%2Fvideo.spalk.co%2Fhls%2Ftest-stream%2Fmaster.m3u8

This stream plays correctly on Safari and iOS clients, and has been validated by the apple mediastreamvalidator.

[mangui]

Hi @Prendo93 using latest flashls/dev you should now be able to play the two alternate audio tracks. However the main track (muxed with video) is not playable since my fix, as it is not signaled in the variant manifest.

[Prendo93]

Thanks @mangui. I missed the whitespace. I have amended the variant manifest and all works as expected.