iOS / Webkit - No Media Source Extensions Support

[eshaz]

icecast-metadata-player uses the Media Source Extensions API to play the stream audio in the browser. This works great in most browsers, except iOS / Webkit, which is the only modern browser that doesn't support MSE. iOS should still play the stream, but there will not be any metadata updates. An alternative will need to be found in order to support real-time metadata updates in iOS.

This is a continuation of #40.

Possible Alternatives:

• Use a service worker to intercept the fetch request, parse / schedule / trigger the metadata using icecast-metadata-js, and send the stream data as the response to an HTML5 Audio element.
  • I tried this once here, but abandoned the effort for MSE. The stream would just stop randomly when playing in Firefox.
  • https://github.com/eshaz/icecast-metadata-js/tree/service-worker
• Decode the stream data into PCM using a WASM module, and then play the PCM via a ScriptProcessorNode or AudioWorkletNode.
  • decodeAudioData() won't work for MP3 because of the bit reservoir that spans multiple frames. See: "4. Why cant MP3 files be seamlessly spliced together?"
  • I tried playing raw PCM via the AudioWorkletNode here. The audio was choppy and it was not very performative.
  • https://github.com/eshaz/icecast-metadata-js/tree/firefox-compatibility
• Make two requests, one for metadata, and one for audio playback.
  • The audio request can just be the src of an HTML5 Audio element.
  • The metadata request can be parsed through icecast-metadata-js
  • The two requests can be synchronized by matching the current / buffered time in the audio element with the audio time of the metadata request.
  • Audio time of the metadata request can be determined by reading the frame headers. This would involve reading the audio data frame by frame and determining the duration of each frame. This is already being done in mse-audio-wrapper to support the current MSE solution and it could be adapted to expose the frame by frame durations and byte offsets.

[W3AXL]

Option 3 above might be the best thing to try first. Before I stumbled upon your set of js libraries I was trying to do something similar, however I was unaware timestamps were embedded in the mp3 data. I was simply trying to come up with a fixed time offset related to the amount of time it took the audio element to buffer, so that the received metadata would align with the audio. For various reasons that approach was unreliable.

Unfortunately, the annoying part is with HTML5 audio you are now at the whim of the browser for buffering times, which for lower-bitrate streams can be in excess of 10 seconds. The nice benefit of directly streaming the data chunks as they come in is that you have virtually no buffering times.

[eshaz]

I agree. I've been meaning to make the changes to allow the frame by frame durations in mse-audio-wrapper. I'm going to split out the existing codec parsing logic that exists in mse-audio-wrapper into a new repo codec-parser that can be used independently to parse the audio codec data into individual frames. Then it should just be a matter of syncing the two requests together.

[eshaz]

I made the initial release of codec-parser that will allow you to calculate the duration of the incoming audio without having to decode it. The onStream callback result can be passed into the CodecParser.iterator() and the iterator will start to yield each frame once there's enough data to sync and parse.

I was thinking a little more into this, and I realized it might be difficult to perfectly sync the HTML5 audio and the metadata fetch request. Both request will have to start at exactly the same time in order to be properly synced, since the audio durations are relative and calcualted on a frame-by-frame basis. There isn't any absolute timestamp in a MP3 stream.

Maybe one of the media events could be used to either trigger or sync the metadata request to the audio element: https://developer.mozilla.org/en-US/docs/Web/Guide/Events/Media_events

[eshaz]

Another update, I just merged a new feature in icecast-metadata-player that uses the two-request method to play audio and display metadata when the MediaSource API isn't supported. I don't have an iOS device to test with, but the demo and all of the code is updated with this new feature. Please give it a try when you get a chance and let me know about the results!

[W3AXL]

I haven't had much time to play around with my player this week. Definitely on my todo list to check out the new additions.

[W3AXL]

So I've got a beta version of my player switched over to using IcecastMetadataPlayer and so far I'm in love with how much it simplifies things.

On desktop, it seems to be working fine on both Firefox and Chrome which is great.

On Safari however, still no dice. No errors to see in the console (enableLogging is set to true), but I never get the onStreamStart() or onPlay() callbacks from the player. Just sits in some limbo state after player.play() is called.

[eshaz]

Ok that's interesting. Are you able to see the network activity on your phone to see if it's at least requesting the stream? Is there any way to pause the script execution with your debugger to see which part of the code it's stuck on? Also, which versions of Safari and iOS are you running?

I was recently able to get macOS running on a virtual machine, and I'm installing Xcode on it so I can emulate an iPhone. I should be able to be more helpful once that's installed.

[eshaz]

Great news! I got it to work in the Xcode emulation! In iOS Safari, the load() method on the audio element has to be called after setting the src. All other browsers automatically started loading when src is set. Once that was fixed, everything else for the HTML5 audio, metadata, and retry logic worked perfectly. I'll cut a patch release shortly.

[eshaz]

I've merged in the fixes in #64 and updated the demo. I also went ahead and tested all the way back to iOS 12.4, which should cover most iOS users. My emulator setup has a delay in the audio, so I'm not able to hear if the metadata is actually synced with the audio, but everything looked correct.

[W3AXL]

Will pull this weekend and test out. For reference I'm currently running iOS 14.0.1 (and whatever Safari version that entails)

[W3AXL]

FYI - things seem to be generally working on Safari for iOS. Metadata is updating properly and audio works fine. Buffering takes a while but I'm guess that's just due to the fallback to HTML5 audio playback.

[eshaz]

That's great to hear. It might help to reduce the buffering time by increasing the burst-size value in your Icecast configuration. https://icecast.org/docs/icecast-latest/config-file.html#limits

Since this works with iOS, I'm going to close this issue. Thanks for the help on getting this to work!