search

AirenSoft / OvenMediaEngine

OvenMediaEngine (OME) is a Sub-Second Latency Live Streaming Server with Large-Scale and High-Definition. #WebRTC #LLHLS
https://airensoft.com/ome.html
GNU Affero General Public License v3.0
2.57k stars 1.06k forks source link

Low-Latency HLS (LLHLS) has been released #766

Closed getroot closed 1 year ago

getroot commented 2 years ago

LLHLS is now available in the latest master branch. LLHLS is a low-latency streaming protocol that aims at latency of about 2 to 4 seconds, unlike WebRTC's less than 1 second. But, as you know, it's HTTP-based, so you can deploy with an existing CDN.

image

LLHLS requires HTTP/2. And browsers only support TLS-based H2 so TLSPort is essential. [2022.05.21 edited] LLHLS runs much higher performance over HTTP/2. Therefore, it is recommended to use TLS Port. OME supports LLHLS over HTTP/1.1 as some CDNs only use HTTP/1.1 to connect to Origin.

LLHLS playback URLs are in the following format:

https://domain[:TLS Port]/<App Name>/<Stream Name>/llhls.m3u8

The Server.xml example below is the minimum setting for RTMP Input / LLHLS Output. If HTTP2 or LLHLS is set to <Enable>false</Enable> in <Modules>, it will not work, so please check this part. If <Modules> is not in Server.xml, it is enabled by default.

<?xml version="1.0" encoding="UTF-8"?>

<Server version="8">
    <Name>OvenMediaEngine</Name>
    <Type>origin</Type>
    <IP>*</IP>
    <PrivacyProtection>false</PrivacyProtection>

    <StunServer>stun.l.google.com:19302</StunServer>

    <Modules>
        <HTTP2>
            <Enable>true</Enable>
        </HTTP2>
        <LLHLS>
            <Enable>true</Enable>
        </LLHLS>
    </Modules>

    <Bind>
        <Providers>
            <RTMP>
                <Port>1935</Port>
                <WorkerCount>1</WorkerCount>
            </RTMP>
        </Providers>

        <Publishers>
            <LLHLS>
                <TLSPort>443</TLSPort>
                <WorkerCount>1</WorkerCount>
            </LLHLS>
        </Publishers>
    </Bind>

    <VirtualHosts>
        <VirtualHost include="VHost*.xml" />
        <VirtualHost>
            <Name>default</Name>
            <Distribution>ovenmediaengine.com</Distribution>

            <Host>
                <Names>
                    <Name>your.domain.com</Name>
                </Names>
                <TLS>
                    <CertPath>path/to/file.crt</CertPath>
                    <KeyPath>path/to/file.key</KeyPath>
                    <ChainCertPath>path/to/file.crt</ChainCertPath>
                </TLS>
            </Host>

            <Applications>
                <Application>
                    <Name>app</Name>
                    <Type>live</Type>
                    <OutputProfiles>
                        <HardwareAcceleration>false</HardwareAcceleration>
                        <OutputProfile>
                            <Name>bypass_stream</Name>
                            <OutputStreamName>${OriginStreamName}</OutputStreamName>
                            <Encodes>
                                <Audio>
                                    <Bypass>true</Bypass>
                                </Audio>
                                <Video>
                                    <Bypass>true</Bypass>
                                </Video>
                            </Encodes>
                        </OutputProfile>
                    </OutputProfiles>
                    <Providers>
                        <RTMP />
                    </Providers>
                    <Publishers>
                        <AppWorkerCount>1</AppWorkerCount>
                        <StreamWorkerCount>8</StreamWorkerCount>
                        <LLHLS>
                            <ChunkDuration>0.2</ChunkDuration>
                            <SegmentDuration>6</SegmentDuration>
                            <SegmentCount>10</SegmentCount>
                            <CrossDomains>
                                <Url>*</Url>
                            </CrossDomains>
                        </LLHLS>
                    </Publishers>
                </Application>
            </Applications>
        </VirtualHost>
    </VirtualHosts>
</Server>

You can test with the following players.

OvenPlayer: https://demo.ovenplayer.com THEO Player: https://www.theoplayer.com/ll-hls-test-page Mac, iOS Safari Browser : Older versions do not support LLHLS, so please use the latest version of Safari. And many other players support LLHLS.

Thanks a lot for your feedback!

getroot commented 2 years ago

Browsers in same device share a single tcp connection when connecting to the same server over HTTP/2. Therefore, when testing by running multiple chrome on one PC, please note that one concurrent user may be output to the log. [2022.05.21] This has been improved and the number of concurrent users is displayed normally.

getroot commented 2 years ago

And AdmissionWebhooks and SignedPolicy in LLHLS are still under development. [2022.05.21] It has been implemented.

getroot commented 2 years ago

In LLHLS, SegmentCountand SegmentDurationdo not affect latency. Only ChunkDurationand the player's buffer size settings (eg TargetLatency) affect latency. So we recommend keeping SegmentCount:10 and SegmentDuration:6 as is recommended by Apple.

nums commented 2 years ago

Amazing work ! thanks ! I will make test next week

IanMitchell77 commented 2 years ago

@getroot - Is this in the v0.13.2 release?

danruser commented 2 years ago

Hello Very good work. Now I am testing. Already 3 hours and without problems at the moment. Thank you!!

getroot commented 2 years ago

@IanMitchell77 No. It was released today in pre-alpha version. This is currently available in the master branch. It will be fully tested and will be included in the next release.

heye commented 2 years ago

Hi, great work! Amazing how active development is! I am currently testing LLHLS with an edge configuration with ovt provider. normal HLS works fine, but calls to /<App Name>/<Stream Name>/llhls.m3u8 are failing with a 404 not found error (tested with firefox/chrome & postman). Is it supposed to work on an edge already?

getroot commented 2 years ago

@heye

LLHLS should also work well on the edge. Of course, I recommend using a different HTTP cache server or CDN (like CloudFront) rather than OME as an edge for LLHLS.

Please share your Server.xml and ovenmediaengine.log files.

heye commented 2 years ago

Sure, here they are. Let me know if I can do anything else. Log: ovenmediaengine.log Config: Server_edge.txt

getroot commented 2 years ago

@heye Thanks to you I found the cause of LLHLS not working in OME Edge. I will fix this problem. Until then, please test in Origin.

bchah commented 2 years ago

@getroot congratulations and thank you for the hard work! Testing now and I see OME crashes with LLHLS added to my configuration. There is an output profile for Thumbnails in the same application that is using LLHLS and that is where the crash seems to happen:

[2022-05-16 14:49:31.716] I [OvenMediaEngine:1604] Config | config_manager.cpp:205  | Trying to set logfile in directory... (/log/engine)
[2022-05-16 14:49:31.716] I [OvenMediaEngine:1604] Config | config_manager.cpp:227  | Trying to load configurations... (/opt/ovenmediaengine/bin/conf/Server.xml)
[2022-05-16 14:49:31.739] I [OvenMediaEngine:1604] OvenMediaEngine | banner.cpp:25   | OvenMediaEngine v0.13.2 (v0.13.2-103-g2113e9bb) is started on [omeserver.live] (Linux x86_64 - 5.4.0-107-generic, #121-Ubuntu SMP Thu Mar 24 16:04:27 UTC 2022)
[2022-05-16 14:49:31.739] I [OvenMediaEngine:1604] OvenMediaEngine | banner.cpp:27   | With modules:
[2022-05-16 14:49:31.739] I [OvenMediaEngine:1604] OvenMediaEngine | banner.cpp:28   |   FFmpeg 4.4.1
[2022-05-16 14:49:31.739] I [OvenMediaEngine:1604] OvenMediaEngine | banner.cpp:29   |     Configuration: --prefix=/opt/ovenmediaengine --extra-cflags='-I/opt/ovenmediaengine/include ' --extra-ldflags='-L/opt/ovenmediaengine/lib -Wl,-rpath,/opt/ovenmediaengine/lib' --extra-libs=-ldl --enable-shared --disable-static --disable-debug --disable-doc --disable-programs --disable-avdevice --disable-dct --disable-dwt --disable-lsp --disable-lzo --disable-rdft --disable-faan --disable-pixelutils --enable-zlib --enable-libopus --enable-libvpx --enable-libfdk_aac --enable-libopenh264 --enable-openssl --disable-nvdec --disable-nvdec --disable-vaapi --disable-vdpau --disable-cuda-llvm --disable-cuvid --disable-ffnvcodec --disable-everything --disable-fast-unaligned --enable-encoder='libvpx_vp8,libopus,libfdk_aac,libopenh264,mjpeg,png' --enable-decoder='aac,aac_latm,aac_fixed,h264,hevc,opus,vp8' --enable-parser='aac,aac_latm,aac_fixed,h264,hevc,opus,vp8' --enable-network --enable-protocol=tcp --enable-protocol=udp --enable-protocol='rtp,file,rtmp,tls,rtmps' --enable-demuxer='rtsp,flv,live_flv' --enable-muxer='mp4,webm,mpegts,flv,mpjpeg' --enable-filter='asetnsamples,aresample,aformat,channelmap,channelsplit,scale,transpose,fps,settb,asettb,format'
[2022-05-16 14:49:31.739] I [OvenMediaEngine:1604] OvenMediaEngine | banner.cpp:30   |     libavformat: 58.76.100
[2022-05-16 14:49:31.739] I [OvenMediaEngine:1604] OvenMediaEngine | banner.cpp:31   |     libavcodec: 58.134.100
[2022-05-16 14:49:31.739] I [OvenMediaEngine:1604] OvenMediaEngine | banner.cpp:32   |     libavutil: 56.70.100
[2022-05-16 14:49:31.739] I [OvenMediaEngine:1604] OvenMediaEngine | banner.cpp:33   |     libavfilter: 7.110.100
[2022-05-16 14:49:31.739] I [OvenMediaEngine:1604] OvenMediaEngine | banner.cpp:34   |     libswresample: 3.9.100
[2022-05-16 14:49:31.739] I [OvenMediaEngine:1604] OvenMediaEngine | banner.cpp:35   |     libswscale: 5.9.100
[2022-05-16 14:49:31.739] I [OvenMediaEngine:1604] OvenMediaEngine | banner.cpp:36   |   SRT: 1.4.4
[2022-05-16 14:49:31.739] I [OvenMediaEngine:1604] OvenMediaEngine | banner.cpp:37   |   SRTP: libsrtp2 2.4.2
[2022-05-16 14:49:31.739] I [OvenMediaEngine:1604] OvenMediaEngine | banner.cpp:38   |   OpenSSL: OpenSSL 3.0.2 15 Mar 2022
[2022-05-16 14:49:31.739] I [OvenMediaEngine:1604] OvenMediaEngine | banner.cpp:39   |     Configuration: compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -O3 -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG
[2022-05-16 14:49:31.739] I [OvenMediaEngine:1604] OvenMediaEngine | banner.cpp:40   |   JsonCpp: 1.9.3
[2022-05-16 14:49:31.739] I [OvenMediaEngine:1604] OvenMediaEngine | banner.cpp:41   |   jemalloc: 5.2.1-0-gea6b3e973b477b8061e0076bb257dbd7f3faa756
[2022-05-16 14:49:31.739] C [OvenMediaEngine:1604] OvenMediaEngine | main.cpp:297  | Current kernel version: 5.4.0-107-generic
[2022-05-16 14:49:31.739] C [OvenMediaEngine:1604] OvenMediaEngine | main.cpp:298  | Linux kernel version 5.3 through 5.6 have a critical bug. Please consider using a different version. (https://bugzilla.kernel.org/show_bug.cgi?id=205933)
[2022-05-16 14:49:31.739] I [OvenMediaEngine:1604] OvenMediaEngine | main.cpp:72   | Server ID : 6194bcc0-91a3-4e1c-aa59-a9f8ba8cdf40
[2022-05-16 14:49:31.748] I [OvenMediaEngine:1604] OvenMediaEngine | main.cpp:111  | Trying to create MediaRouter...
[2022-05-16 14:49:31.748] I [OvenMediaEngine:1604] MediaRouter | mediarouter.cpp:40   | Mediarouter has been started.
[2022-05-16 14:49:31.748] I [OvenMediaEngine:1604] OvenMediaEngine | main.cpp:114  | Trying to create WebRTC Publisher...
[2022-05-16 14:49:31.749] I [OvenMediaEngine:1604] Signalling.P2P | rtc_p2p_manager.cpp:27   | P2P is disabled in the configuration
[2022-05-16 14:49:31.751] I [OvenMediaEngine:1604] ICE | ice_port.cpp:86   | ICE port is bound to 0.0.0.0:10010/UDP (0x55b4bcf26930)
[2022-05-16 14:49:31.751] I [OvenMediaEngine:1604] ICE | ice_port.cpp:128  | ICE port is bound to *:3333/TCP (0x55b4bcf2b840)
[2022-05-16 14:49:31.751] I [OvenMediaEngine:1604] ICE | ice_port_manager.cpp:94   | RelayServer is created successfully: host:3333?transport=tcp
[2022-05-16 14:49:31.751] I [OvenMediaEngine:1604] WebRTC Publisher | webrtc_publisher.cpp:161  | WebRTC Publisher is listening on TLS: *:3334...
[2022-05-16 14:49:31.751] I [OvenMediaEngine:1604] Publisher | publisher.cpp:18   | WebRTC Publisher has been started.
[2022-05-16 14:49:31.752] I [OvenMediaEngine:1604] OvenMediaEngine | main.cpp:115  | Trying to create LLHLS Publisher...
[2022-05-16 14:49:31.752] E [OvenMediaEngine:1604] Socket | socket.cpp:419  | [#13] [0x55b4bcf72df0] Could not bind to *:443 (-1)
[2022-05-16 14:49:31.753] E [OvenMediaEngine:1604] LLHLS Publisher | llhls_publisher.cpp:81   | Could not initialize LLHLS https server
[2022-05-16 14:49:31.753] I [OvenMediaEngine:1604] Publisher | publisher.cpp:18   | LLHLS Publisher has been started.
[2022-05-16 14:49:31.753] I [OvenMediaEngine:1604] OvenMediaEngine | main.cpp:116  | Trying to create HLS Publisher...
[2022-05-16 14:49:31.754] I [OvenMediaEngine:1604] Publisher | publisher.cpp:18   | HLS Publisher has been started.
[2022-05-16 14:49:31.754] I [OvenMediaEngine:1604] OvenMediaEngine | main.cpp:117  | Trying to create MPEG-DASH Publisher...
[2022-05-16 14:49:31.755] W [OvenMediaEngine:1604] DASH | dash_publisher.cpp:36   | DASH Publisher is disabled by configuration
[2022-05-16 14:49:31.755] I [OvenMediaEngine:1604] OvenMediaEngine | main.cpp:118  | Trying to create Low-Latency MPEG-DASH Publisher...
[2022-05-16 14:49:31.755] W [OvenMediaEngine:1604] LLDASH | cmaf_publisher.cpp:39   | LLDASH Publisher is disabled by configuration
[2022-05-16 14:49:31.755] I [OvenMediaEngine:1604] OvenMediaEngine | main.cpp:119  | Trying to create OVT Publisher...
[2022-05-16 14:49:31.756] W [OvenMediaEngine:1604] OVT | ovt_publisher.cpp:38   | OVTPublisher is disabled by configuration
[2022-05-16 14:49:31.756] I [OvenMediaEngine:1604] OvenMediaEngine | main.cpp:120  | Trying to create File Publisher...
[2022-05-16 14:49:31.756] I [OvenMediaEngine:1604] Publisher | publisher.cpp:18   | FilePublisher has been started.
[2022-05-16 14:49:31.756] I [OvenMediaEngine:1604] OvenMediaEngine | main.cpp:121  | Trying to create MpegtsPush Publisher...
[2022-05-16 14:49:31.756] I [OvenMediaEngine:1604] Publisher | publisher.cpp:18   | MPEGTSPushPublisher has been started.
[2022-05-16 14:49:31.757] I [OvenMediaEngine:1604] OvenMediaEngine | main.cpp:122  | Trying to create RtmpPush Publisher...
[2022-05-16 14:49:31.757] I [OvenMediaEngine:1604] Publisher | publisher.cpp:18   | RTMPPushPublisher has been started.
[2022-05-16 14:49:31.757] I [OvenMediaEngine:1604] OvenMediaEngine | main.cpp:123  | Trying to create Thumbnail Publisher...
[2022-05-16 14:49:31.758] I [OvenMediaEngine:1604] Thumbnail | thumbnail_publisher.cpp:105  | Thumbnail publisher is listening on *
[2022-05-16 14:49:31.758] I [OvenMediaEngine:1604] Publisher | publisher.cpp:18   | ThumbnailPublisher has been started.
[2022-05-16 14:49:31.758] I [OvenMediaEngine:1604] OvenMediaEngine | main.cpp:126  | Trying to create Transcoder...
[2022-05-16 14:49:31.758] I [OvenMediaEngine:1604] OvenMediaEngine | main.cpp:129  | Trying to create WebRTC Provider...
[2022-05-16 14:49:31.759] I [OvenMediaEngine:1604] Signalling.P2P | rtc_p2p_manager.cpp:27   | P2P is disabled in the configuration
[2022-05-16 14:49:31.759] I [OvenMediaEngine:1604] ICE | ice_port.cpp:86   | ICE port is bound to 0.0.0.0:10010/UDP (0x55b4bcf26930)
[2022-05-16 14:49:31.759] I [OvenMediaEngine:1604] ICE | ice_port.cpp:128  | ICE port is bound to *:3333/TCP (0x55b4bcf2b840)
[2022-05-16 14:49:31.760] I [OvenMediaEngine:1604] ICE | ice_port_manager.cpp:94   | RelayServer is created successfully: host:3333?transport=tcp
[2022-05-16 14:49:31.762] I [OvenMediaEngine:1604] WebRTC Provider | webrtc_provider.cpp:166  | WebRTCProvider is listening on TLS: *:3334...
[2022-05-16 14:49:31.762] I [OvenMediaEngine:1604] Provider | provider.cpp:37   | WebRTCProvider has been started.
[2022-05-16 14:49:31.763] I [OvenMediaEngine:1604] OvenMediaEngine | main.cpp:130  | Trying to create MPEG-TS Provider...
[2022-05-16 14:49:31.763] W [OvenMediaEngine:1604] MpegtsProvider | mpegts_provider.cpp:97   | MPEGTSProvider is disabled by configuration
[2022-05-16 14:49:31.763] I [OvenMediaEngine:1604] OvenMediaEngine | main.cpp:131  | Trying to create SRT Provider...
[2022-05-16 14:49:31.764] I [OvenMediaEngine:1604] SrtProvider | srt_provider.cpp:66   | SrtProvider is listening on *:9999/SRT
[2022-05-16 14:49:31.764] I [OvenMediaEngine:1604] Provider | provider.cpp:37   | SrtProvider has been started.
[2022-05-16 14:49:31.765] I [OvenMediaEngine:1604] OvenMediaEngine | main.cpp:132  | Trying to create RTMP Provider...
[2022-05-16 14:49:31.766] I [OvenMediaEngine:1604] RTMPProvider | rtmp_provider.cpp:101  | RTMPProvider is listening on *:1935/TCP
[2022-05-16 14:49:31.766] I [OvenMediaEngine:1604] Provider | provider.cpp:37   | RTMPProvider has been started.
[2022-05-16 14:49:31.766] I [OvenMediaEngine:1604] OvenMediaEngine | main.cpp:133  | Trying to create OVT Provider...
[2022-05-16 14:49:31.766] I [OvenMediaEngine:1604] Provider | provider.cpp:37   | OVTProvider has been started.
[2022-05-16 14:49:31.767] I [OvenMediaEngine:1604] OvenMediaEngine | main.cpp:134  | Trying to create RTSPC Provider...
[2022-05-16 14:49:31.767] I [OvenMediaEngine:1604] Provider | provider.cpp:37   | RTSPCProvider has been started.
[2022-05-16 14:49:31.767] I [OvenMediaEngine:1604] OvenMediaEngine | main.cpp:135  | Trying to create File Provider...
[2022-05-16 14:49:31.768] I [OvenMediaEngine:1604] Provider | provider.cpp:37   | FileProvider has been started.
[2022-05-16 14:49:31.768] I [OvenMediaEngine:1604] OvenMediaEngine | main.cpp:142  | All modules are initialized successfully
[2022-05-16 14:49:31.771] I [OvenMediaEngine:1604] Certificate | certificate.cpp:43   | A certificate has been created for VirtualHost [default]:
    Cert file path: /etc/letsencrypt/live/omeserver.xyz/cert.pem
    Chain cert file path: /etc/letsencrypt/live/omeserver.xyz/chain.pem
    Private key file path: /etc/letsencrypt/live/omeserver.xyz/privkey.pem
[2022-05-16 14:49:31.784] I [OvenMediaEngine:1604] Orchestrator | orchestrator_internal.cpp:701  | Trying to create an application: [#default#live]
[2022-05-16 14:49:31.784] I [OvenMediaEngine:1604] MediaRouter | mediarouter_application.cpp:44   | Created Mediarouter application. application id(0), app(#default#live), worker(1)
[2022-05-16 14:49:31.784] I [OvenMediaEngine:1604] MediaRouter | mediarouter_application.cpp:99   | Started Mediarouter application. application id(0), app(#default#live)
[2022-05-16 14:49:31.784] I [OvenMediaEngine:1604] MediaRouter | mediarouter.cpp:83   | Created Mediarouter. app(#default#live)
[2022-05-16 14:49:31.786] I [OvenMediaEngine:1604] Publisher | application.cpp:29   | WebRTC ApplicationWorker has been created
[2022-05-16 14:49:31.786] I [OvenMediaEngine:1604] Publisher | application.cpp:29   | WebRTC ApplicationWorker has been created
[2022-05-16 14:49:31.786] I [OvenMediaEngine:1604] Publisher | application.cpp:161  | WebRTC Publisher Application has created [#default#live] application
[2022-05-16 14:49:31.786] I [OvenMediaEngine:1604] Publisher | application.cpp:29   | LLHLS ApplicationWorker has been created
[2022-05-16 14:49:31.786] I [OvenMediaEngine:1604] Publisher | application.cpp:29   | LLHLS ApplicationWorker has been created
[2022-05-16 14:49:31.786] I [OvenMediaEngine:1604] Publisher | application.cpp:161  | LLHLS Publisher Application has created [#default#live] application
[2022-05-16 14:49:31.787] I [OvenMediaEngine:1604] Publisher | application.cpp:29   | HLS ApplicationWorker has been created
[2022-05-16 14:49:31.787] I [OvenMediaEngine:1604] Publisher | application.cpp:29   | HLS ApplicationWorker has been created
[2022-05-16 14:49:31.787] I [OvenMediaEngine:1604] Publisher | application.cpp:161  | HLS Publisher Application has created [#default#live] application
[2022-05-16 14:49:31.787] I [OvenMediaEngine:1604] Publisher | application.cpp:29   | File ApplicationWorker has been created
[2022-05-16 14:49:31.787] I [OvenMediaEngine:1604] Publisher | application.cpp:29   | File ApplicationWorker has been created
[2022-05-16 14:49:31.787] I [OvenMediaEngine:1604] Publisher | application.cpp:161  | FilePublisher Application has created [#default#live] application
[2022-05-16 14:49:31.787] W [OvenMediaEngine:1604] Publisher | publisher.cpp:70   | MPEGTSPush publisher is disabled in #default#live application, so it was not created
[2022-05-16 14:49:31.787] I [OvenMediaEngine:1604] Publisher | application.cpp:29   | RTMPPush ApplicationWorker has been created
[2022-05-16 14:49:31.787] I [OvenMediaEngine:1604] Publisher | application.cpp:29   | RTMPPush ApplicationWorker has been created
[2022-05-16 14:49:31.787] I [OvenMediaEngine:1604] Publisher | application.cpp:161  | RTMPPushPublisher Application has created [#default#live] application
[2022-05-16 14:49:31.788] I [OvenMediaEngine:1604] Publisher | application.cpp:29   | Thumbnail ApplicationWorker has been created
[2022-05-16 14:49:31.788] I [OvenMediaEngine:1604] Publisher | application.cpp:29   | Thumbnail ApplicationWorker has been created
[2022-05-16 14:49:31.788] I [OvenMediaEngine:1604] Publisher | application.cpp:161  | ThumbnailPublisher Application has created [#default#live] application
[2022-05-16 14:49:31.788] I [OvenMediaEngine:1604] Transcoder | transcoder_application.cpp:31   | Created transcoder application. app(#default#live)
[2022-05-16 14:49:31.788] I [OvenMediaEngine:1604] Transcoder | transcoder.cpp:86   | Transcoder has created [config][#default#live] application
[2022-05-16 14:49:31.788] I [OvenMediaEngine:1604] Provider | application.cpp:46   | WebRTCProvider has created [#default#live] application
[2022-05-16 14:49:31.788] I [OvenMediaEngine:1604] Provider | application.cpp:46   | SrtProvider has created [#default#live] application
[2022-05-16 14:49:31.788] I [OvenMediaEngine:1604] Provider | application.cpp:46   | RTMPProvider has created [#default#live] application
[2022-05-16 14:49:31.788] W [OvenMediaEngine:1604] Provider | provider.cpp:94   | OVT provider is disabled in #default#live application, so it was not created
[2022-05-16 14:49:31.788] W [OvenMediaEngine:1604] Provider | provider.cpp:94   | RTSP Pull provider is disabled in #default#live application, so it was not created
[2022-05-16 14:49:31.789] I [OvenMediaEngine:1604] Orchestrator | orchestrator_internal.cpp:701  | Trying to create an application: [#default#chat]
[2022-05-16 14:49:31.789] I [OvenMediaEngine:1604] MediaRouter | mediarouter_application.cpp:44   | Created Mediarouter application. application id(1), app(#default#chat), worker(1)
[2022-05-16 14:49:31.789] I [OvenMediaEngine:1604] MediaRouter | mediarouter_application.cpp:99   | Started Mediarouter application. application id(1), app(#default#chat)
[2022-05-16 14:49:31.789] I [OvenMediaEngine:1604] MediaRouter | mediarouter.cpp:83   | Created Mediarouter. app(#default#chat)
[2022-05-16 14:49:31.791] I [OvenMediaEngine:1604] Publisher | application.cpp:29   | WebRTC ApplicationWorker has been created
[2022-05-16 14:49:31.791] I [OvenMediaEngine:1604] Publisher | application.cpp:161  | WebRTC Publisher Application has created [#default#chat] application
[2022-05-16 14:49:31.791] W [OvenMediaEngine:1604] Publisher | publisher.cpp:70   | LLHLS publisher is disabled in #default#chat application, so it was not created
[2022-05-16 14:49:31.791] W [OvenMediaEngine:1604] Publisher | publisher.cpp:70   | HLS publisher is disabled in #default#chat application, so it was not created
[2022-05-16 14:49:31.791] W [OvenMediaEngine:1604] Publisher | publisher.cpp:70   | File publisher is disabled in #default#chat application, so it was not created
[2022-05-16 14:49:31.791] W [OvenMediaEngine:1604] Publisher | publisher.cpp:70   | MPEGTSPush publisher is disabled in #default#chat application, so it was not created
[2022-05-16 14:49:31.791] W [OvenMediaEngine:1604] Publisher | publisher.cpp:70   | RTMPPush publisher is disabled in #default#chat application, so it was not created
[2022-05-16 14:49:31.791] W [OvenMediaEngine:1604] Publisher | publisher.cpp:70   | Thumbnail publisher is disabled in #default#chat application, so it was not created
[2022-05-16 14:49:31.791] I [OvenMediaEngine:1604] Transcoder | transcoder_application.cpp:31   | Created transcoder application. app(#default#chat)
[2022-05-16 14:49:31.791] I [OvenMediaEngine:1604] Transcoder | transcoder.cpp:86   | Transcoder has created [config][#default#chat] application
[2022-05-16 14:49:31.791] I [OvenMediaEngine:1604] Provider | application.cpp:46   | WebRTCProvider has created [#default#chat] application
[2022-05-16 14:49:31.791] W [OvenMediaEngine:1604] Provider | provider.cpp:94   | SRT provider is disabled in #default#chat application, so it was not created
[2022-05-16 14:49:31.791] W [OvenMediaEngine:1604] Provider | provider.cpp:94   | RTMP provider is disabled in #default#chat application, so it was not created
[2022-05-16 14:49:31.791] W [OvenMediaEngine:1604] Provider | provider.cpp:94   | OVT provider is disabled in #default#chat application, so it was not created
[2022-05-16 14:49:31.791] W [OvenMediaEngine:1604] Provider | provider.cpp:94   | RTSP Pull provider is disabled in #default#chat application, so it was not created
[2022-05-16 14:49:31.795] I [OvenMediaEngine:1604] Certificate | certificate.cpp:43   | A certificate has been created for VirtualHost [api_server]:
    Cert file path: /etc/letsencrypt/live/omeserver.xyz/cert.pem
    Chain cert file path: /etc/letsencrypt/live/omeserver.xyz/chain.pem
    Private key file path: /etc/letsencrypt/live/omeserver.xyz/privkey.pem
[2022-05-16 14:49:31.800] I [OvenMediaEngine:1604] APIServer | api_server.cpp:113  | API Server is listening on TLS: *:8081/TCP...
[2022-05-16 14:49:40.889] I [SPSRT-T9999:1622] SrtProvider | srt_provider.cpp:113  | The SRT client has connected : <ClientSocket: 0x7fe57c0010f0, #621434660, Connected, SRT, Nonblocking, 174.88.37.212:61517> [srt%3A//omeserver.xyz%3A9999/live/helloWorld%3Fp%3DeyJ1cmxfZXhwaXJlIjo3MjQ1NTk0MTMwMDAwfQ%26s%3DiFeE5aFBCuw-uMRACa2Pe2QzMgk]
[2022-05-16 14:49:40.891] I [SPSRT-T9999:1622] Provider | stream.cpp:49   | Unknown/helloWorld(621434660) has been started stream
[2022-05-16 14:49:40.982] I [SPSRT-T9999:1622] MediaRouter | mediarouter_application.cpp:244  | Trying to create a stream: [#default#live/helloWorld(621434660)]
[2022-05-16 14:49:40.982] I [SPSRT-T9999:1622] MediaRouter | mediarouter_application.cpp:245  | 
[Stream Info]
id(621434660), msid(0), output(helloWorld), SourceType(SRT), Created Time (Mon May 16 14:49:40 2022) UUID(6194bcc0-91a3-4e1c-aa59-a9f8ba8cdf40/default/#default#live/helloWorld/i)

    Video Track #256: Bypass(false) Bitrate(0b) codec(1, H264) resolution(0x0) framerate(0.00fps) timebase(1/90000)
    Audio Track #257: Bypass(false) Bitrate(0b) codec(6, AAC) samplerate(0) format(none, 0) channel(unknown, 0) timebase(1/90000)
[2022-05-16 14:49:40.982] I [SPSRT-T9999:1622] MediaRouter | mediarouter_stream.cpp:54   | Trying to create media route stream: name(helloWorld) id(621434660)
[2022-05-16 14:49:40.983] I [SPSRT-T9999:1622] Transcoder | transcoder_stream.cpp:557  | [#default#live/helloWorld(621434660)] -> [#default#live/helloWorld(714771552)] Output stream has been created.
[2022-05-16 14:49:40.983] I [SPSRT-T9999:1622] Transcoder | transcoder_stream.cpp:557  | [#default#live/helloWorld(621434660)] -> [#default#live/helloWorld_preview(1766724565)] Output stream has been created.
[2022-05-16 14:49:40.986] I [SPSRT-T9999:1622] MediaRouter | mediarouter_application.cpp:244  | Trying to create a stream: [#default#live/helloWorld(714771552)]
[2022-05-16 14:49:40.986] I [SPSRT-T9999:1622] MediaRouter | mediarouter_application.cpp:245  | 
[Stream Info]
id(714771552), msid(0), output(helloWorld), SourceType(Transcoder), Created Time (Mon May 16 14:49:40 2022) UUID(6194bcc0-91a3-4e1c-aa59-a9f8ba8cdf40/default/#default#live/helloWorld/o)
    >> Origin Stream Info
    id(621434660), output(helloWorld), SourceType(SRT), Created Time (Mon May 16 14:49:40 2022)

    Video Track #0: Bypass(true) Bitrate(0b) codec(1, H264) resolution(0x0) framerate(0.00fps) timebase(1/90000)
    Audio Track #1: Bypass(true) Bitrate(0b) codec(6, AAC) samplerate(90.0K) format(none, 0) channel(unknown, 0) timebase(1/90000)
    Audio Track #2: Bypass(false) Bitrate(128.00Kb) codec(8, OPUS) samplerate(48.0K) format(none, 0) channel(stereo, 2) timebase(1/48000)
[2022-05-16 14:49:40.986] I [SPSRT-T9999:1622] MediaRouter | mediarouter_stream.cpp:54   | Trying to create media route stream: name(helloWorld) id(714771552)
[2022-05-16 14:49:40.986] I [SPSRT-T9999:1622] Publisher | stream.cpp:202  | ThumbnailPublisher Application application has started [helloWorld(714771552)] stream (MSID : 0)
[2022-05-16 14:49:40.986] I [SPSRT-T9999:1622] MediaRouter | mediarouter_application.cpp:244  | Trying to create a stream: [#default#live/helloWorld_preview(1766724565)]
[2022-05-16 14:49:40.986] I [SPSRT-T9999:1622] MediaRouter | mediarouter_application.cpp:245  | 
[Stream Info]
id(1766724565), msid(0), output(helloWorld_preview), SourceType(Transcoder), Created Time (Mon May 16 14:49:40 2022) UUID(6194bcc0-91a3-4e1c-aa59-a9f8ba8cdf40/default/#default#live/helloWorld_preview/o)
    >> Origin Stream Info
    id(621434660), output(helloWorld), SourceType(SRT), Created Time (Mon May 16 14:49:40 2022)

    Video Track #3: Bypass(false) Bitrate(0b) codec(9, JPEG) resolution(192x108) framerate(2.00fps) timebase(1/90000)
[2022-05-16 14:49:40.986] I [SPSRT-T9999:1622] MediaRouter | mediarouter_stream.cpp:54   | Trying to create media route stream: name(helloWorld_preview) id(1766724565)
[2022-05-16 14:49:40.986] I [SPSRT-T9999:1622] Publisher | stream.cpp:202  | ThumbnailPublisher Application application has started [helloWorld_preview(1766724565)] stream (MSID : 0)
[2022-05-16 14:49:40.987] I [SPSRT-T9999:1622] WebRTC Publisher | rtc_stream.cpp:192  | Unsupported codec(Video/JPEG) is being input from media track
[2022-05-16 14:49:40.987] I [SPSRT-T9999:1622] WebRTC Publisher | rtc_stream.cpp:367  | WebRTC Stream has been created : helloWorld_preview/1766724565
Rtx(false) Ulpfec(false) JitterBuffer(true) PlayoutDelay(false min:0 max: 0)
[2022-05-16 14:49:40.987] I [SPSRT-T9999:1622] Publisher | stream.cpp:202  | WebRTC Publisher Application application has started [helloWorld_preview(1766724565)] stream (MSID : 0)
[2022-05-16 14:49:40.987] I [SPSRT-T9999:1622] LLHLS Publisher | llhls_stream.cpp:80   | LLHlsStream(#default#live/helloWorld_preview) - Ignore unsupported codec(JPEG)
[2022-05-16 14:49:40.987] C [SPSRT-T9999:1622] OvenMediaEngine | signals.cpp:114  | OME received signal 11 (SIGSEGV), interrupt.
[2022-05-16 14:49:40.987] E [SPSRT-T9999:1622] OvenMediaEngine | signals.cpp:152  | Could not open dump file to write
getroot commented 2 years ago

@bchah The helloWorld_previewstream crashes in the LLHLS module because there is no codec that LLHLS can support. Thanks for finding this bug.

getroot commented 2 years ago

@heye

I've fixed LLHLS to work in edge mode. Edge of OME receives packets from OVT and repackages them into LLHLS. Therefore, Edge requests a stream to Origin, and the player's request is pending until at least two segments are completed.

To operate LLHLS in Edge mode, the following settings are required.

<Origins><Properties><UnusedStreamDeletionTimeout>15000</UnusedStreamDeletionTimeout>

<Publishers><LLHLS><SegmentDuration>3</SegmentDuration>

getroot commented 2 years ago

@bchah I solved this problem. Please confirm.

bchah commented 2 years ago

@getroot The crashes are fixed but the Thumbnail publisher now has a new problem:

[2022-05-16 17:11:15.800] W [SPSegPub-T8000:1333] Thumbnail | thumbnail_publisher.cpp:184 | Failed to parse hostname

Calls to the thumbnail URL (e.g. https://myapp.xyz:8000/live/helloWorld_preview/thumb.jpg) return a 200 response but the contents of x.jpg are empty.

I noticed a new <Distribution></Distribution> tag in your default Server.xml but the issue seems the same with or without it.

getroot commented 2 years ago

@bchah This problem was that the Thumbnail publisher did not work over HTTP2. I solved this. Thank you very much!

heye commented 2 years ago

@getroot It works very well now, amazing. Thank you for the fast fix!

bchah commented 2 years ago

@getroot just confirming a few tests for you:

Thumbnail publisher fixed on HTTP/2 ✅ LL-HLS working with RTMP Provider ✅ SignedPolicy working with LL-HLS ✅ LL-HLS working with SRT Provider ❌

When trying to load LL-HLS with SRT Provider, I see this in the OME log:

[2022-05-18 12:20:14.242] W [StreamWorker:1951] LLHLS Publisher | llhls_stream.cpp:198 | Could not find segment for track_id = 1, segment_number = 430

I hope this provides some good clues 🥇

getroot commented 2 years ago

@bchah

Thanks for the hard testing! Can you provide Server.xml and OvenMediaEngine.log when testing LL-HLS working with SRT Provider? If it is difficult to upload it to a public place, please send it to getroot@airensoft.com.

bchah commented 2 years ago

@getroot Here you go!

One more clue I found is that I could not produce the above error in the log with SRT Provider, unless SignedPolicy was also enabled. Without SignedPolicy there is no error in the logs (but the symptom of playback failed is the same).

log.txt config.txt

naanlizard commented 2 years ago

Browsers in same device share a single tcp connection when connecting to the same server over HTTP/2. Therefore, when testing by running multiple chrome on one PC, please note that one concurrent user may be output to the log.

For our use case, we can sometimes have multiple streams/players on one page, does this mean the log will only show one connection per device, no matter how many streams they watch?

That seems like it could be significant performance improvement over HLS with multiple requests per player

getroot commented 2 years ago

@naanlizard

To be precise, that means watching the same stream on multiple players (browsers) on one device. That is, even if sessions share the same TCP connection, if each session plays a different stream, OME can distinguish them individually. But if you run multiple players (browsers) on one PC and request LLHLS playback on the same stream, it is difficult to distinguish them into individual sessions because the multiple players download the same playlist and chunk over one TCP connection.

Oh, that's how it's implemented now, and that doesn't mean it's forever impossible. I have a few ideas and will experiment with them.

Adam1901 commented 2 years ago

Is it possible to set up the LLHLS port without TLS and use a CDN/Load balencer for TLS support? And still get the benefits of LLHLS?

getroot commented 2 years ago

https://datatracker.ietf.org/doc/html/draft-pantos-hls-rfc8216bis

I misunderstood in this document that using HTTP/2 in LLHLS is mandatory. But it was a recommendation.

I have modified some code and tested LLHLS on HTTP/1.1, and it has been confirmed that it works well. When playing LLHLS over HTTP/1.1, chrome (hls.js) connects 4 connections at the same time. So HTTP/2 has a high performance advantage.

However, there will be cases where HTTP/1.1 must be used. Several CDNs, including CloudFront for example, still only support HTTP/1.1 between CDNs and Origin.

I'll let you know when this is done.

@Adam1901 After this is done, it should work fine on non-TLS ports as well. Of course, it will work with HTTP/1.1.

getroot commented 2 years ago

[Update]

danruser commented 2 years ago

Hello. After your last update. LLHLS Works only in TLS port

LLHLS does not work on Non-TLS HTTP Ports. Ignores the "LLHLS Port 8080 /Port /LLHLS -setting". In LLHLS, TlsPort must be enabled since LLHLS works only over HTTP/2.

Thank you.

getroot commented 2 years ago

@danruser Oh I didn't commit one thing by mistake. Thank you for telling me. I will fix this soon.

JIEgOKOJI commented 2 years ago

Constant crashes whem pushing streams from nginx-rtmp ovenmediaengine.log .

getroot commented 2 years ago

@JIEgOKOJI Thank you for reporting. Could you share your binary and crash_2022xxxx.dump file?

getroot commented 2 years ago

@danruser I have committed the missing code. But I haven't done enough testing yet. I plan to start again next week. Have a great weekend!

JIEgOKOJI commented 2 years ago

crashandbin.zip Here crash dump and binary. Compiled on CentOS7 kernel 5.16.10-1

getroot commented 2 years ago

@JIEgOKOJI

Do you have any more information that can help me analyze this issue? For example, does it crash if you push only 1 stream? Do you have enough memory?

At this time, there is insufficient information to analyze the cause of the crash. Unfortunately the crash dump did not generate enough information for analysis as shown below. So if you compile OME in debug mode and provide the generated crash dump, it will be of great help to me.

***** Crash dump *****
OvenMediaEngine v0.13.2 (v0.13.2-108-g0fb25e7) received signal 6 (SIGABRT)
- Time: 2022-05-21T04:49:56+0300, pid: 26151, tid: 26160
- Stack trace
#0 /lib64/libpthread.so.0 0x7f6223433630 ? + 0xf630
#1 /lib64/libc.so.6 0x7f621fae2387 gsignal + 0x37
#2 /lib64/libc.so.6 0x7f621fae3a78 abort + 0x148
#3 /lib64/libstdc++.so.6 0x7f62203f2a95 __gnu_cxx::__verbose_terminate_handler() + 0x165
#4 /lib64/libstdc++.so.6 0x7f62203f0a06 ? + 0x5ea06
#5 /lib64/libstdc++.so.6 0x7f62203f0a33 ? + 0x5ea33
#6 /usr/bin/OvenMediaEngine 0x76aa3e ? + 0x0
#7 /lib64/libpthread.so.0 0x7f622342bea5 ? + 0x7ea5
#8 /lib64/libc.so.6 0x7f621fbaab0d clone + 0x6d
getroot commented 2 years ago

[Update]

getroot commented 2 years ago

@bchah Now LL-HLS works with SRT Provider. Please confirm!

bchah commented 2 years ago

@getroot Thank you! LL-HLS now working with SRT Provider except for playing back in Safari natively. The error in the browser is "Failed to Load Resource, Plugin Handled Load" and there is no information on the OME side.

I wonder if this thread provides any clues?

https://stackoverflow.com/questions/18103103/failed-to-load-resource-plugin-handled-load-on-ios

For context, this issue was occurring intermittently before but now seems to be a persistent behaviour. The other players like OvenPlayer, VLC and TheoPlayer work great 👍

getroot commented 2 years ago

@bchah Thank you. In my env that problem is not reproduced but I will fight safari from tomorrow. I will win and let you know again.

cwpenhale commented 2 years ago

Cleaning up the comment on #766 to keep it on-topic and opened a new issue (#774)

getroot commented 2 years ago

@cwpenhale It is better to create this as a new issue. (This is far from an LLHLS issue.) Origin-Relay-Edge structures are supported. Please share your Server.xml and OvenMediaEngine.log files with new issues.

cwpenhale commented 2 years ago

Sounds good, thank you!

getroot commented 2 years ago

@bchah I reproduced the problem on my iOS and it seems to have solved it. Did you solve the problem too? Thanks for the hard testing!!

bchah commented 2 years ago

@getroot You win, Safari loses! LL-HLS is now working perfectly for me on the latest commit :)

getroot commented 2 years ago

I did a test using CloudFront as an Edge and found it to work fine. During testing I added a Cache-Control header to LLHLS.

JIEgOKOJI commented 2 years ago

@JIEgOKOJI

Do you have any more information that can help me analyze this issue? For example, does it crash if you push only 1 stream? Do you have enough memory?

At this time, there is insufficient information to analyze the cause of the crash. Unfortunately the crash dump did not generate enough information for analysis as shown below. So if you compile OME in debug mode and provide the generated crash dump, it will be of great help to me.

***** Crash dump *****
OvenMediaEngine v0.13.2 (v0.13.2-108-g0fb25e7) received signal 6 (SIGABRT)
- Time: 2022-05-21T04:49:56+0300, pid: 26151, tid: 26160
- Stack trace
#0 /lib64/libpthread.so.0 0x7f6223433630 ? + 0xf630
#1 /lib64/libc.so.6 0x7f621fae2387 gsignal + 0x37
#2 /lib64/libc.so.6 0x7f621fae3a78 abort + 0x148
#3 /lib64/libstdc++.so.6 0x7f62203f2a95 __gnu_cxx::__verbose_terminate_handler() + 0x165
#4 /lib64/libstdc++.so.6 0x7f62203f0a06 ? + 0x5ea06
#5 /lib64/libstdc++.so.6 0x7f62203f0a33 ? + 0x5ea33
#6 /usr/bin/OvenMediaEngine 0x76aa3e ? + 0x0
#7 /lib64/libpthread.so.0 0x7f622342bea5 ? + 0x7ea5
#8 /lib64/libc.so.6 0x7f621fbaab0d clone + 0x6d

Memory is enough, no signs about oom killer. For single stream from obs or nginx-rtmp everything is ok. I will try to compile in debug mode today

JIEgOKOJI commented 2 years ago

ahh and when i disable lhls everything works fine. Not tested with traditional hls or dash

JIEgOKOJI commented 2 years ago

debug.zip here debug logs and crashdump

getroot commented 2 years ago

@JIEgOKOJI Hmm.... Detailed information is not available even in the crash dump in debug mode. However, looking at the log, it is noticeable that the samplerate information is empty. I will try to reproduce the problem. Thank you!

[Stream Info]
id(53), msid(0), output(154280_prem), SourceType(Rtmp), Created Time (Thu May 26 07:52:40 2022) UUID(c1d7d690-8c56-4751-9bcb-696243d59386/default/#default#lhls/154280_prem/i)

    Video Track #0: Bypass(false) Bitrate(0b) codec(1, H264) resolution(0x0) framerate(0.00fps) timebase(1/1000)
    Audio Track #1: Bypass(false) Bitrate(0b) codec(6, AAC) samplerate(0) format(s16, 16) channel(unknown, 0) timebase(1/1000)
getroot commented 2 years ago

@JIEgOKOJI Forcing the samplerate to 0 does not reproduce the crash. I'll have to test it with nginx-rtmp. Can you share your nginx-rtmp configuration?

getroot commented 2 years ago

@JIEgOKOJI I tested with more than 50 stream inputs using the following nginx configuration. But I couldn't reproduce the crash. I would appreciate it if you could provide me with the following information to reproduce the issue.

  1. With which encoder do you send the stream to nginx? And what codec settings do you use?
  2. Please tell me your nginx settings. (I hope this setup reproduces the same problem.)
  3. Make sure that you are using the latest OME by git pull the master branch.
  4. Please provide your operating system information. Package version, kernel version.
getroot@OME-Dev:~$ cat /usr/local/nginx/conf/nginx.conf

#user nobody;
worker_processes 1;

#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;

#pid logs/nginx.pid;

events {
    worker_connections 1024;
}

rtmp {
        server {
                listen 11936;
                chunk_size 4096;

                application app {
                        live on;
                        record off;
                        meta copy;
                        push rtmp://ome-dev.airensoft.com:11935/app/;
                }
        }
}
getroot commented 2 years ago

ABR is now available on LLLHS. Discuss ABR related issues at https://github.com/AirenSoft/OvenMediaEngine/issues/777.

(Issue number 777!)