LL-HLS streams on Apple devices played only in lowest quality even with good network conditions

[mbakumenkov]

Describe the bug LL-HLS streams on Apple devices played only in lowest quality even with good network conditions although there are no such problems on other windows/linux/android devices with the same stream.

To Reproduce Just open a LL-HLS link from safari browser or paste in LL-HLS player (e.g. OvenPlayer or TheoPlayer)

Expected behavior Stream is played in highest quality provided by playlist or at least switches to high quality after some time

Logs Please upload the entire OvenMediaEngine.log. You may delete important personal information.

Server (please complete the following information):

• OS: Ubuntu 22.04
• OvenMediaEngine Version: [e.g. v0.16.0] (docker image)

Player (please complete the following information):

• Device: Apple devices, tested on Iphone 11 and Macbook pro M1
• OS: IOS 15.6.1, macOS
• Browser Safari
• Version [e.g. 22]

Additional context My Server.xml

<?xml version="1.0" encoding="UTF-8"?>

<Server version="8">
        <Name>OvenMediaEngine</Name>
        <!-- Host type (origin/edge) -->
        <Type>origin</Type>
        <!-- Specify IP address to bind (* means all IPs) -->
        <IP>*</IP>
        <PrivacyProtection>false</PrivacyProtection>

        <!--
        To get the public IP address(mapped address of stun) of the local server.
        This is useful when OME cannot obtain a public IP from an interface, such as AWS or docker environment.
        If this is successful, you can use ${PublicIP} in your settings.
        -->
        <StunServer>stun.l.google.com:19302</StunServer>

        <Modules>
                <!--
                Currently OME only supports h2 like all browsers do. Therefore, HTTP/2 only works on TLS ports.
                -->
                <HTTP2>
                        <Enable>true</Enable>
                </HTTP2>

                <LLHLS>
                        <Enable>true</Enable>
                </LLHLS>

                <!-- P2P works only in WebRTC and is experiment feature -->
                <P2P>
                        <!-- disabled by default -->
                        <Enable>false</Enable>
                        <MaxClientPeersPerHostPeer>2</MaxClientPeersPerHostPeer>
                </P2P>
        </Modules>

        <!-- Settings for the ports to bind -->
        <Bind>
                <!-- Enable this configuration if you want to use API Server -->

                <Managers>
                        <API>
                                <Port>${env:OME_API_PORT:8081}</Port>
                                <WorkerCount>1</WorkerCount>
                        </API>
                </Managers>

                <Providers>
                        <RTMP>
                                <Port>1935</Port>
                                <WorkerCount>1</WorkerCount>
                        </RTMP>
                </Providers>

                <Publishers>
                        <LLHLS>
                                <!--
                                OME only supports h2, so LLHLS works over HTTP/1.1 on non-TLS port.
                                Note that LLHLS runs higher performance over HTTP/2.
                                Therefore, it is recommended to use TLS Port.
                                -->
                                <Port>${env:OME_LLHLS_STREAM_PORT:8080}</Port>
                                <TLSPort>${env:OME_LLHLS_STREAM_TLS_PORT:443}</TLSPort>
                                <WorkerCount>1</WorkerCount>
                        </LLHLS>
                </Publishers>
        </Bind>

        <Managers>
                <Host>
                        <Names>
                                <Name>*</Name>
                        </Names>
                </Host>
                <API>
                        <AccessToken>lala:blabla</AccessToken>
                        <CrossDomains>
                                <Url>*</Url>
                        </CrossDomains>
                </API>
        </Managers>

        <!-- P2P works only in WebRTC -->
        <!--
        <P2P>
                <MaxClientPeersPerHostPeer>2</MaxClientPeersPerHostPeer>
        </P2P>
        -->

        <VirtualHosts>
                <!-- You can use wildcard like this to include multiple XMLs -->
                <VirtualHost include="VHost*.xml" />
                <VirtualHost>
                        <Name>default</Name>
                        <!--Distribution is a value that can be used when grouping the same vhost distributed across multiple servers. This value is output to the events log, so you can use it to aggregate statistics. -->
                        <Distribution>ovenmediaengine.com</Distribution>

                        <!-- Settings for multi ip/domain and TLS -->
                        <Host>
                                <Names>
                                        <!-- Host names
                                                <Name>stream1.airensoft.com</Name>
                                                <Name>stream2.airensoft.com</Name>
                                                <Name>*.sub.airensoft.com</Name>
                                                <Name>192.168.0.1</Name>
                                        -->
                                        <Name>*</Name>
                                </Names>
                                <TLS>
                                        <CertPath>/opt/ovenmediaengine/bin/origin_conf/certs/cert.pem</CertPath>
                                        <KeyPath>/opt/ovenmediaengine/bin/origin_conf/certs/privkey.pem</KeyPath>
                                        <ChainCertPath>/opt/ovenmediaengine/bin/origin_conf/certs/fullchain.pem</ChainCertPath>
                                </TLS>
                        </Host>

                        <!-- Refer https://airensoft.gitbook.io/ovenmediaengine/signedpolicy
                        -->

                        <SignedPolicy>
                                <PolicyQueryKeyName>${env:SIGNED_POLICY_POLICY_QUERY_KEY:lala}</PolicyQueryKeyName>
                                <SignatureQueryKeyName>${env:SIGNED_POLICY_SIGNATURE_QUERY_KEY:blabla}</SignatureQueryKeyName>
                                <SecretKey>${env:SIGNED_URL_CRYPTO_KEY:lalala}</SecretKey>

                                <Enables>
                                        <Providers>rtmp</Providers>
                                </Enables>
                        </SignedPolicy>

                        <!-- Settings for ProxyPass (It can specify origin for each path) -->
                        <!--
                        When Edge tries to pull a stream, it looks first in <Origins> in local, and in OriginMapStore.
                        Therefore, it is recommended to disable Origins when using OriginMapStore.
                        <OriginMapStore>
                                In order to use OriginMap, you must enable OVT Publisher in Origin and OVT Provider in Edge.
                                <RedisServer>
                                        <Host>192.168.0.160:6379</Host>
                                        <Auth>!@#ovenmediaengine</Auth>
                                </RedisServer>
                        </OriginMapStore>
                        -->

                        <!--
                                Apply the following settings for /app/<stream_name> requests.

                                This demonstrates setting <OutputProfiles>/<Providers>/<Pubishers> for a specific application.

                                (https://github.com/AirenSoft/OvenMediaEngine/issues/621#issuecomment-1026892776)
                        -->
                        <Applications>
                                <Application>
                                        <!--
                                        Origins and OriginMapStore can dynamically create an application if no application exists
                                        when creating a stream. They create a new application by copying the Application configuration
                                        where <Name> is *.
                                        In other words, an application with <Name> as * serves as a dynamic application template.
                                        -->
                                        <Name>live</Name>
                                        <Type>live</Type>
                                        <OutputProfiles>
                                                <OutputProfile>
                                                        <Name>stream</Name>
                                                        <OutputStreamName>${OriginStreamName}</OutputStreamName>
                                                        <Playlist>
                                                                <Name>playlist</Name>
                                                                <FileName>playlist</FileName>
                                                                <Rendition>
                                                                        <Name>720p</Name>
                                                                        <Video>720p</Video>
                                                                        <Audio>pass</Audio>
                                                                </Rendition>
                                                                <Rendition>
                                                                        <Name>480p</Name>
                                                                        <Video>480p</Video>
                                                                        <Audio>pass</Audio>
                                                                </Rendition>
                                                                <Rendition>
                                                                        <Name>360p</Name>
                                                                        <Video>360p</Video>
                                                                        <Audio>pass</Audio>
                                                                </Rendition>
                                                        </Playlist>
                                                        <Playlist>
                                                                <Name>smoker</Name>
                                                                <FileName>smoker</FileName>
                                                                <Rendition>
                                                                        <Name>Quality</Name>
                                                                        <Video>720p</Video>
                                                                        <Audio>pass</Audio>
                                                                </Rendition>
                                                        </Playlist>

                                                        <Encodes>
                                                                <Video>
                                                                        <Name>720p</Name>
                                                                        <Bypass>true</Bypass>
                                                                </Video>
                                                                <Video>
                                                                        <Name>480p</Name>
                                                                        <Codec>h264</Codec>
                                                                        <Bitrate>1000000</Bitrate>
                                                                        <Framerate>30</Framerate>
                                                                        <Width>854</Width>
                                                                        <Height>480</Height>
                                                                        <Preset>medium</Preset>
                                                                        <Profile>main</Profile>
                                                                        <KeyFrameInterval>60</KeyFrameInterval>
                                                                        <ThreadCount>8</ThreadCount>
                                                                </Video>
                                                                <Video>
                                                                        <Name>360p</Name>
                                                                        <Codec>h264</Codec>
                                                                        <Bitrate>400000</Bitrate>
                                                                        <Framerate>30</Framerate>
                                                                        <Width>640</Width>
                                                                        <Height>360</Height>
                                                                        <Preset>fast</Preset>
                                                                        <Profile>main</Profile>
                                                                        <KeyFrameInterval>60</KeyFrameInterval>
                                                                        <ThreadCount>8</ThreadCount>
                                                                </Video>

                                                                <Audio>
                                                                        <Name>pass</Name>
                                                                        <Bypass>true</Bypass>
                                                                </Audio>
                                                        </Encodes>
                                                </OutputProfile>
                                        </OutputProfiles>
                                        <Providers>
                                                <RTMP />
                                        </Providers>
                                        <Publishers>
                                                <AppWorkerCount>1</AppWorkerCount>
                                                <StreamWorkerCount>16</StreamWorkerCount>
                                                <LLHLS>
                                                        <OriginMode>true</OriginMode>
                                                        <ChunkDuration>1.0</ChunkDuration>
                                                        <SegmentDuration>2</SegmentDuration>
                                                        <SegmentCount>10</SegmentCount>
                                                        <CrossDomains>
                                                                <Url>*</Url>
                                                        </CrossDomains>
                                                </LLHLS>
                                        </Publishers>
                                </Application>
                        </Applications>
                </VirtualHost>
        </VirtualHosts>

OBS encoder settings:

I also tried with ChunkDuration 0.2, SegmentDuration 6 - same result :(

[getroot]

I've been researching this for a while and was assuming it was a bug in Safari. ABR is working properly in all players except Safari. And if you play with ?_HLS_legacy=YES after OME's LLHLS URL, ABR appears to work normally.

I still don't know what trick I need to use to get this to work properly. Do you have a URL where ABR works properly in Safari using another streaming server? If so, it would be very helpful if you could provide it.

[getroot]

Below is the LL-HLS demo URL provided by Apple. On my iOS Safari, this always plays at 488 kbps. In comparison, it plays at 2Mbps on PC /Chrome /demo.ovenplayer.com. What does your iOS Safari play with?

https://ll-hls-test.cdn-apple.com/llhls4/ll-hls-test-04/multi.m3u8

[dygy]

@getroot it plays maximum quality as i can see

[dygy]

@getroot but i believe best example of a stream i saw that improves in Safari is it https://cph-p2p-msl.akamaized.net/hls/live/2000341/test/master.m3u8

[getroot]

@getroot it plays maximum quality as i can see

I think 488Kbps is lowest quality. Your Safari plays the lowest quality, just like mine.

[getroot]

And unfortunately, https://cph-p2p-msl.akamaized.net/hls/live/2000341/test/master.m3u8 is not a LL-HLS URL, it is just legacy HLS URL.

[dygy]

on reload it does like this. so it still can improve, but we have the problem that it's never 720 for our streams.

[getroot]

After some time it will drop back to 488 or if you reload it will drop to 488. My and my colleagues' 5 iPhones always behave this strangely. I don't think OME can improve this yet.

[getroot]

How can I see the information box floating in the middle of the player in your screenshot?

[dygy]

@getroot on safari mac-os you also can put a url and it work similar to iOS, but it has tooltip with information

[getroot]

As far as I know, iOS Safari and MAC Safari are quite different. And since MSE is supported on MAC Safari, you have an alternative to using other players. However, iOS Safari does not support MSE, so this is obviously a problem.

Anyway, I've been analyzing this for quite some time, and so far I don't have any ideas on how to improve it on the OME side. To temporarily resolve this, I often create multiple playlists and serve them by generating LLHLS URLs for each quality.

If anyone has any ideas on how to improve this, please let me know. I hope iOS Safari supports MSE as soon as possible. Alternatively, I'd love for them to reveal more details about how the player in iOS Safari works.

[getroot]

I think I've figured out a way to bypass Safari's LLHLS ABR problem. It will be patched in the next week or so.

[getroot]

After a lot of analysis and testing, it is very likely that this is a bug or non-implementation in iOS Safari. It appears that they are not doing Bandwidth Estimation properly when loading media with #EXT-X-PRELOAD-HINT.

Sometimes they don't use #EXT-X-PRELOAD-HINT and then it plays just fine at highest quality. I saw this and thought there was a way to get around it. In fact, if you do not use partial segments using ?_HLS_legacy=YES, Safari plays the video well at the highest quality.

I'll look into it a little further, but this is very likely out of my control. I'm spending too much time on this.

[dygy]

@getroot any chance we could report it to Apple?

[dygy]

i made request to apple, I've choose Webkit as source of problem (but i'm not sure it's correct :DDD) i sent them link to this discussion and hope someone from apple will look on our discussion and maybe join it

[getroot]

I added a special option called <EnablePreloadHint>true</EnablePreloadHint> . The default value is true. Setting this to false OME will not add #EXT-X-PRELOAD-HINT to the playlist. This will also make ABR work properly in iOS Safari. But the delay will be longer.

From this I am guessing that Safari is doing bandwidth estimation wrong or not implemented when loading PRELOAD-HINT. When a player loads a PRELOAD-HINT, the server must block the request until the resource is ready and respond with the resource all at once as soon as it is ready. Therefore, players must measure bandwidth from the moment the first byte is received, not from the moment of request. THEO Player appears to do this well. hls.js does not yet use PRELOAD-HINT. But Safari seems to be doing this wrong.

You can test this with the master branch and configuration below:

<Publishers>
    <LLHLS>
        <EnablePreloadHint>false</EnablePreloadHint>
        <ChunkDuration>0.5</ChunkDuration>
        <PartHoldBack>1.5</PartHoldBack>
        <SegmentDuration>6</SegmentDuration>
        <SegmentCount>10</SegmentCount>
        <CrossDomains>
            <Url>*</Url>
        </CrossDomains>

[getroot]

I just installed iOS 17.2 Beta3 and tested it. Surprisingly, I found that the problem was resolved in this version. It plays well with high quality rendition.

[getroot]

This is fixed in iOS 17.2, so close. If you need further discussion, please start a new thread on the Discussion board.