search

AirenSoft / OvenMediaEngine

OvenMediaEngine (OME) is a Sub-Second Latency Live Streaming Server with Large-Scale and High-Definition. #WebRTC #LLHLS
https://airensoft.com/ome.html
GNU Affero General Public License v3.0
2.57k stars 1.06k forks source link

Scale out Origin servers for more incoming streams #343

Closed command-tab closed 1 year ago

command-tab commented 3 years ago

I have a question about Origin-Edge mode: If I understand the architecture correctly, the Origin server performs any configured transcoding, and the Edge servers fetch that content over OVT and scale out to handle all the necessary viewer connections. Let's say I wanted to support dozens of incoming RTMP streams. According to the docs, there can be only one primary Origin and one failover. So how would you recommend scaling of Origin servers to accommodate lots of incoming streams?

Thank you!

basisbit commented 3 years ago

According to the docs, there can be only one primary Origin and one failover.

Could you please provide a link to where you did read this? As far as I understand the docs, you should be able to just set up a bunch of Origin servers in different regions close to your streamers to avoid RTMP streaming issues, and on the edge servers in the config, just add a bunch of origins and an application for each of those origins, differentiating them by url.

command-tab commented 3 years ago

In the Clustering docs, the diagram shows Primary and Secondary, and I didn't realize I could have more than that pair.

If I have a bunch of similarly located people streaming to one Origin server and it starts being overwhelmed, can I add more Origin servers in that same area and load balance the incoming RTMP traffic across them? If so, what's the best method to do that — round robin? IP hash?

basisbit commented 3 years ago

You could probably use HAProxy or similar TCP-based load-balancer to round-robin balance different incoming RTMP streams to different origin servers running OME. I'd suggest avoiding just using IP address hash because thanks to carrier grade NAT, many users could show up having the same source IP address.

The question however would be, how does your user get the stream-url which points to the correct origin server (as part of the path which the client requests from the edge server)?

An alternative approach would be to assign different customers to different origin servers (thus they always stream to the same one, you don't necessarily need a tcp-load-balancer in that case and the stream-url for the webclient would stay the same.

getroot commented 3 years ago

We also solve this problem by assigning a separate domain to each Origin server when operating a commercial service and then providing a different input URL to the sender.

Of course, this process is automated in the web service. When the sender clicks Create Channel, it provides the URL of the Origin server that is being used the least at the moment.

alexthedamager commented 3 years ago

We also solve this problem by assigning a separate domain to each Origin server when operating a commercial service and then providing a different input URL to the sender.

Of course, this process is automated in the web service. When the sender clicks Create Channel, it provides the URL of the Origin server that is being used the least at the moment.

It would be great to be able to scale origin servers as @command-tab mentioned. We currently have a few origin OMEs on different domain names in different regions (one origin per region) and use the same approach you mentioned (let the user choose their preferred ingest region when they create a room) and then show them the relevant Ingest URL.

But the concern is that at a later point (days / weeks later) the traffic changes and you may run into a situation where too many users are streaming to origin-1.ome and overloading the server. We could make it so that we spin up more origin servers as required and set our web app to not give the same origin url to more than X users, but then we run into a situation where we have many origins that become largely unused in the future, but we can't shut them down because some users are still streaming to them.

I think an ideal implementation for us would be if we could route incoming streams to origin servers dynamically, based on the current usage of the server. For example: in our web app the user selects "US West" as the ingest region, which always gives the same url "us-west.ome". And then within the US-West region we have multiple Origin servers. When a user streams to us-west.ome, the stream is automatically routed to the least-used origin server within that region.

bchah commented 3 years ago

I’d write that straight into your own application.

The flow would be something like:

{server_id: “eu-5” server_url: “wss://eu5.com/live” signed_key: “abc123”}

Does that make sense?

On Jun 2, 2021, at 8:17 AM, llspalex @.***> wrote:

We also solve this problem by assigning a separate domain to each Origin server when operating a commercial service and then providing a different input URL to the sender.

Of course, this process is automated in the web service. When the sender clicks Create Channel, it provides the URL of the Origin server that is being used the least at the moment.

It would be great to be able to scale origin servers as @command-tab https://github.com/command-tab mentioned. We currently have a few origin OMEs on different domain names in different regions (one origin per region) and use the same approach you mentioned (let the user choose their preferred ingest region when they create a room) and then show them the relevant Ingest URL.

But the concern is that at a later point (days / weeks later) the traffic changes and you may run into a situation where too many users are streaming to origin-1.ome and overloading the server. We could make it so that we spin up more origin servers as required and set our web app to not give the same origin url to more than X users, but then we run into a situation where we have many origins that become largely unused in the future, but we can't shut them down because some users are still streaming to them.

I think an ideal implementation for us would be if we could route incoming streams to origin servers dynamically, based on the current usage of the server.

For example, user selects "US West" as the ingest region, which always gives the same url "us-west.ome". And then within the US-West region we have multiple Origin servers. When a user streams to us-west.ome, the stream is automatically routed to the least-used origin server within that region.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/AirenSoft/OvenMediaEngine/issues/343#issuecomment-852977629, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIAMIQTA6C22Q34Q4QVZXZTTQYOL5ANCNFSM43D2JSNA.

basisbit commented 3 years ago

for that to work, the streaming protocols like RTMP, RTSP or SRT would have to support something like following HTTP 302 redirects or similar. Does this exist in the mentioned protocols? Then a load distributor service could be written, which distributes the streams to a server with enough capacity in the geographic region of the streamer.

The discovery of the correct stream edge server and endpoint/app can already be done similarly with HTTP 302 forwarding and that scales well to thousands of concurrent users (for example last weekend I had an event with ~35000 viewers over HLS using a custom load balancer like this)..

bchah commented 3 years ago

Oh, I meant these checks could take place before a stream URL is issued - not as a redirect, you are correct.

On Jun 2, 2021, at 8:45 AM, basisbit @.***> wrote:

for that to work, the streaming protocols like RTMP, RTSP or SRT would have to support something like following HTTP 302 redirects or similar. Does this exist in the mentioned protocols? Then a load distributor service could be written, which distributes the streams to a server with enough capacity in the geographic region of the streamer.

The discovery of the correct stream edge server and endpoint/app can already be done similarly with HTTP 302 forwarding and that scales well to thousands of concurrent users.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/AirenSoft/OvenMediaEngine/issues/343#issuecomment-852996311, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIAMIQR7BQZK3OLKB7KAVLTTQYRXNANCNFSM43D2JSNA.

alexthedamager commented 3 years ago

Thanks @bchah - I actually forgot that we could leverage the OME API to check current usage. Though as you said, it wouldn't take care of dynamically routing streams after the stream URL is issued.

getroot commented 3 years ago

This is a very good discussion thread.

Since redirect is not provided by all protocols, it is difficult to use other than HTTP-based protocols. Also the implementation between Encoder - Provider is more difficult.

So as a way to automatically scale the origin I have the following idea.

To scale Origin automatically, Edge needs to be a bit smarter. If origin is bundled into one domain, when a request to play “/app/stream” comes to Edge, Edge does not know which origin has “app/stream”. There are several ways to make Edge know this.

  1. Develop a server that maps the app/stream list to the origin server, and let edge automatically find the origin from this server.

  2. The mapping between the app/stream list and the origin server is handled by each service developer, and edge provides the following playback methods. ws://edge.com/app/stream?origin="origin.com/app/stream"

Please let me know if you have any better ideas or have similar examples of other media servers that have addressed this.

command-tab commented 3 years ago

Another thing to consider: If you go the route where you arrange to provide users with a different Origin URL every time they stream (based on current usage stats), they will have to adjust the settings of their encoder every time they stream (to point to the new Origin URL you give them), and that adds some friction. By contrast, if I stream to YouTube in OBS, I don't have to change anything every time I start streaming.

I really like the ideas about making Edge smarter and being able to figure out which Origin has the requested stream 😍

alexthedamager commented 3 years ago

Bundling origin servers into one domain (or a few different domains based on region) would be amazing. I agree that to the streamer, the process should be invisible and the Stream URL and Stream Key used should not have to change between streaming sessions.

@getroot, I'm not sure which of your two proposed ideas would be better, though in our current implementation we do not use Edge servers (viewers connect to our Origins) because a) we have a low number of viewers per room (our platform is designed for many rooms with <20 viewers per room rather than a few rooms with many viewers per room) and b) we are trying to keep latency as low as possible - every ms counts.

That said, we are not opposed to using Edge servers, and we will probably eventually need Edge servers.

Auto-scaling / load-balancing of OME ingest in a way that we can make it invisible to the streamer would be a very welcome feature.

basisbit commented 3 years ago

YouTube uses Anycast DNS for load-balancing based on regions, as well as 10Gb/s connected servers. If a region needs more capacity, they can add more servers to the domain name entry at the resolver. However, Anycast DNS only works if you are your own autonomous system and buy transit to various places, or pay quite a bit of money for that service.

You can also partially solve the ingest congestion issue by using much more subdomain names than you have servers, and on demand changing the subdomains to different servers, to distribute the load evenly. However, this requires the "smart" edge server solution mentioned above, which knows where to get the stream from. For scaling this nicely, this should probably be implemented using event streams (server-sent events), so each edge only holds one http request to each configured origin server. The origin servers then should send stream-started and stream-stopped events over that event channel - as well as initially send a list of all the currently already running streams to the edge at initial connection.

This ways, you could dynamically load-balance the ingest side to the origin servers and the egress side to the edge servers. This should scale well up to the point where you would run out of bandwidth of the origin server sending video streams to the edge servers, and be rather easy to implement.

basisbit commented 3 years ago

Uh, the ingress side could even be changed to requiring much less subdomains than mentioned above for smaller setups:

You give the streamers/customers only the region-specific origin address. Then you dynamically change the address of the region specific subdomain to the IP address of the origin server which currently has the least amount of load. You could also write your own small dns server for that, which returns the IP address of the least used origin server to dns resoution requests. For these solutions, however, you will have to learn how to use the API of your DNS provider.

getroot commented 3 years ago

@command-tab @bchah @llspalex @basisbit Thank you all for providing some great insights. I'll think about this topic in more depth when I'm done with my urgent task now.

alexthedamager commented 3 years ago

@getroot Know you're super busy - just wondered if you've had time to think more about this, or if it's possibly on the roadmap? 🙏

getroot commented 3 years ago

@llspalex

These are grouped in the advanced features category on our roadmap. The advanced features category isn't that high of a priority.

Here's why:

Scale out origin servers is a very nice feature, but I don't think it's essential. Because this is solved by the operator manually assigning an origin server for each stream.

alexthedamager commented 3 years ago

@getroot Thank you for the info - good to know.

alexthedamager commented 3 years ago

Uh, the ingress side could even be changed to requiring much less subdomains than mentioned above for smaller setups:

* one subdomain address per origin server which always points to that origin server so that the edge server can contact those

* and one subdomain per geographic region.

You give the streamers/customers only the region-specific origin address. Then you dynamically change the address of the region specific subdomain to the IP address of the origin server which currently has the least amount of load. You could also write your own small dns server for that, which returns the IP address of the least used origin server to dns resoution requests. For these solutions, however, you will have to learn how to use the API of your DNS provider.

@basisbit When dynamically changing the address of the region specific subdomain to the IP address of the 'least-load' origin server, what method would you suggest - changing the A record value, or changing the CNAME?

We are thinking of using your proposed setup to scale origin servers. Because A / CNAME changes aren't instant, we will be conservative with our server switching threshold (eg. when server load hits 70%, change the A or CNAME record so that the next incoming stream goes to a lower load server).

basisbit commented 3 years ago

Because A / CNAME changes aren't instant, [...]

They can be instant, it just depends on your DNS setup. You can set the DNS cache time to 1 Second or to 1 Minute or whatever time you want, if either your DNS provider supports that, or if you operate your own name server. I'd suggest using CNAME, because that makes it easier to debug issues later on by the resolved CNAME being readable (could be for example RegionCityNumber.domain.tld (for example europefrankfurt42.example.org)). But IP would also be fine.

However, you still have the issue of not knowing what origin the edge should pull the stream from. For HLS/LL-DASH, you can work around that using HTTP 301 forwarding and regularly polling the Origin server API for available streams. Adding a feature to OvenMediaEngine would be the better and more scalable and faster solution (less time after starting the stream until the stream can be watched). A C++ developer should be able to design that together with feedback from AirenSoft (OME core devs), then implement that and preferably publish it with a pull request here, so that the community can do the majority of future code maintenance work for you (and you thus help improve OvenMediaEngine for all).

alexthedamager commented 3 years ago

@basisbit Thank you for the info.

Just so I better understand the requirements (I'm not a dev) - would dyanmic changing of DNS records be required regardless of how scaling the origin is implemented, or is there an implementation in which dynamic DNS routing would not be required?

In our current implementation, we allow RTMP, SRT + WebRTC for ingest and only WebRTC for playback and we do not use edge servers (origin only, because low number of viewers, and because we are trying to keep latency as low as possible). I'm aware that 'making the edge smarter' is required to properly implement this feature, as many users do use edge servers.

We would love to contribute this to OME. The main question at the moment is the stuff around dynamic DNS.

basisbit commented 3 years ago

Instead of changing DNS records, you could

basisbit commented 3 years ago

[...] we do not use edge servers [...] because we are trying to keep latency as low as possible [...]

Uhm, using an Edge server doesn't noticeably increase latency. If your users are all not just in one country, you will even need local edge servers to be able to do usable and reliable enough low-latency streaming.

alexthedamager commented 3 years ago

At the moment our users are geographically very close to one another. But yes, we do realise that we will need to add edge servers as the userbase grows.

It sounds like dynamically changing DNS might actually be the best solution for us. Possibly using something like AWS Route 53 or DNSMadeEasy.

If that's the case, is it worth us trying to contribute an 'origin scaling' feature to OME if it relies on users needing to dynamically change their DNS records? Might be a question for getroot.

Wallacy commented 2 years ago

FWIW: Im current running my own custom media server to handle the needs of our product, but i do think the solution the we got for scale our servers can be used here:

In our user case, we run our API server using Cloudflare Workers, and our API workflow is based on WebRTC. So in that case, we can redirect the websocket very easy using the Workers. Each machine has a performance monitor and when any machine has more than 80% os CPU (or memory) usage our scripts just setup another media-server automatically (all that using the Workers), and if any machine is unused for some time we just remove for our pool. And when any user request the websocket connection we just resolve that pointing to any available server that has spare capacity, but we try to keep the machine usage above of 20%.

In that way we save some money when theres only few users, and we can grow until the datacenter stops to create new VMs. (Depending of the provider we are talking about hundreds of thousands machines)

Thats for sure does not work if we plan to ingest using SRT or other protocols that just resolve the DNS. We can use a proxy to resolve that but the cost to do that is not in our target so, in that case we just make change the DNS like already talked here (1 minute TTL).

In any case, the OvenMediaEngine cluster should be able to be add/remove machines dynamic instead of static files. Auto scale is a bonus over that.

basisbit commented 2 years ago

In our user case, we run our API server using Cloudflare Workers, and our API workflow is based on WebRTC. So in that case, we can redirect the websocket

Websocket does not support HTTP 302 redirects, so, do you route all your traffic through Cloudflare? If yes, what customer base do you have so that they are okay with paying Cloudflares high traffic prices? oO

Wallacy commented 2 years ago

In our user case, we run our API server using Cloudflare Workers, and our API workflow is based on WebRTC. So in that case, we can redirect the websocket

Websocket does not support HTTP 302 redirects, so, do you route all your traffic through Cloudflare? If yes, what customer base do you have so that they are okay with paying Cloudflares high traffic prices? oO

You can redirect the initial Upgrade offer. Maybe that will not worker for all clients btw. But yes, in our case our websocket is proxied by the Worker itself. And that does not incur in any "high traffic prices", well... Cloudflares does not have any traffic price, except for Argo. We run terabytes using the basic plan without pay any cent for traffic. Also, the WebSocket for WebRTC is just the signaling, there's not much going on here.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

basisbit commented 2 years ago

*bump* I'd like to keep this open for now, because this is indeed a not yet easily solved problem. Scaling up to support many concurrent viewers is easy, but doing this for high amounts of concurrent inbound streams (with transcoding, to more it more difficult of an example) at low fix-costs is still a difficult task (and requires anycast or your own bgp setup plus your own origin-lookup process + frequent polling).

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

alexthedamager commented 2 years ago

bump Agree that this issue should be kept open.

getroot commented 2 years ago

I committed a new feature today called OriginMapStore.

This is a function where the origin server updates its app/stream information to the Redis server, and the Edge retrieves the app/stream information from the Redis server and connects to the origin server.

With this feature, Edge no longer needs to know in advance the URL of the origin server that has a specific app/stream. In other words, even if you create a stream in any origin server, Edge can pull the app/stream in the cluster. So you are free to expand your Origin server. Note that the app/stream name is used as a key, so it must be unique within the cluster.

I tested this with Redis Server 5.0.7.

Set <OriginMapStore> in <VirtualHost> of Origin server as follows.

<OriginMapStore>
    <!-- 
    In order to use OriginMap, you must enable OVT Publisher in Origin and OVT Provider in Edge.
    -->
    <RedisServer>
        <Host>192.168.0.160:6379</Host>
        <Auth>airensoft!@#</Auth>
    </RedisServer>

    <!-- 
        This is only needed for the origin server and used to register the ovt address of the stream. 
    -->
    <OriginHostName>ome-dev.airensoft.com</OriginHostName>
</OriginMapStore>

And also in Edge, set it like this:

<OriginMapStore>
    <RedisServer>
        <Host>192.168.0.160:6379</Host>
        <Auth>airensoft!@#</Auth>
    </RedisServer>
</OriginMapStore>

Edge first looks for app/stream in the local <Origins>, and if not, then it looks for app/stream in <OriginMapStore>. Therefore, if <Location>/</Location> is exist in <Origins> setting, <OriginMapStore> does not work. So you may need to disable the <Origins> setting or modify it appropriately.

getroot commented 2 years ago

You must rerun prerequisite.sh for redis client library to test OringinMapStore feature.

command-tab commented 2 years ago

Oh, this is very cool! I look forward to experimenting with this 😍

getroot commented 2 years ago

@command-tab You can experiment with the master branch. Many comments are welcome! :-)

basisbit commented 2 years ago

Very nice improvement! Congratulations! Any suggestions / best practices how to secure and encrypt the redis communication, because these tend to go across country/region borders when having edge servers across the globe?

command-tab commented 2 years ago

I'm not sure that securing communication with Redis needs to be in scope for OME. Redis's own docs say:

Redis is designed to be accessed by trusted clients inside trusted environments. This means that usually it is not a good idea to expose the Redis instance directly to the internet or, in general, to an environment where untrusted clients can directly access the Redis TCP port or UNIX socket. [...] Access to the Redis port should be denied to everybody but trusted clients in the network, so the servers running Redis should be directly accessible only by the computers implementing the application using Redis.

Historically, securing that network path is up to the admin, not the particular software, as Redis is best used not exposed directly to the internet. Maybe a VPN, Wireguard tunnel, or other private networking system would be in order. This is all good, since it helps OME retain focus.

getroot commented 2 years ago

@basisbit I haven't implemented it yet, but Redis also supports TLS. But in our case I am wondering if it is necessary. This is because the only information OME sends and receives with Redis is the app/stream name and OVT URL of Origin. Origin can exist in a secure group that only Edge can access, so even if it is exposed, there will be no problem. Is this sensitive information that needs to be protected? Let us know what you think.

guyguy333 commented 2 years ago

Hello @getroot,

I’m currently trying OME and OriginMapStore feature. Origin server publishes well the stream on Redis and I can found it browsing keys:

Screenshot 2022-10-14 at 17 41 51

However, when I try to connect through Edge also using OriginMapStore feature, it’s unable to find stream:

[2022-10-14 15:35:08.530] I [SPRtcSig-T4334:11] Publisher | publisher.cpp:186 | Try to pull stream from local origin map: [#default#app/azerty] [2022-10-14 15:35:08.530] I [SPRtcSig-T4334:11] Publisher | publisher.cpp:190 | Try to pull stream from origin map store: [#default#app/azerty] [2022-10-14 15:35:08.530] E [SPRtcSig-T4334:11] Orchestrator | orchestrator.cpp:635 | Could not find Origin for the stream: [#default#app/azerty]

I don’t explain it. I didn’t have issue with a static Origin config. Origin part seems to works well, it looks like issue is only on Edge side but I’ve no error message and so I don’t know how to debug issue.

I did monitor Redis query, and I found GET is similar to SET so it should find it:

1665763544.425932 [0 100.64.3.125:46300] "PING"
1665763544.426872 [0 100.64.3.125:46300] "SET" "app/azerty" "ovt://100.64.3.125:9000/app/azerty" "EX" "10"
1665763546.929070 [0 100.64.3.125:46300] "PING"
1665763546.929912 [0 100.64.3.125:46300] "SET" "app/azerty" "ovt://100.64.3.125:9000/app/azerty" "EX" "10"
1665763547.227483 [0 100.64.2.69:54072] "AUTH" "(redacted)"
1665763547.228671 [0 100.64.2.69:54072] "GET" "app/azerty"
1665763548.165683 [0 100.64.9.11:58406] "AUTH" "(redacted)"
1665763548.166009 [0 100.64.9.11:58406] "ping"
1665763548.207847 [0 100.64.9.11:58410] "AUTH" "(redacted)"
1665763548.208705 [0 100.64.9.11:58410] "ping"

Thanks

getroot commented 2 years ago

@guyguy333 plz upload your server.xml files of origin and edge

guyguy333 commented 2 years ago

@getroot Yes sure. I'm using Redis 7.

Edge:

<?xml version="1.0" encoding="UTF-8"?>
        <Server version="8">
            <Name>edge</Name>
            <IP>*</IP>
            <Type>edge</Type>
            <PrivacyProtection>false</PrivacyProtection>
            <StunServer>stun.l.google.com:19302</StunServer>
            <Modules>
                <!-- 
                Currently OME only supports h2 like all browsers do. Therefore, HTTP/2 only works on TLS ports.         
                -->
                <HTTP2>
                    <Enable>true</Enable>
                </HTTP2>
                <!-- P2P works only in WebRTC and is experiment feature -->
                <!-- 
                <P2P>
                    <Enable>false</Enable>
                    <MaxClientPeersPerHostPeer>2</MaxClientPeersPerHostPeer>
                </P2P>
                -->
            </Modules>
            <Bind>
                <Providers />
                <Publishers>
                    <WebRTC>
                        <Signalling>
                            <Port>4333</Port>
                            <TLSPort>4334</TLSPort>
                            <WorkerCount>1</WorkerCount>
                        </Signalling>
                        <IceCandidates>
                            <IceCandidate>*:11000-11005/udp</IceCandidate>
                            <TcpRelay>*:4478</TcpRelay>
                            <TcpForce>true</TcpForce>
                            <TcpRelayWorkerCount>1</TcpRelayWorkerCount>
                        </IceCandidates>
                    </WebRTC>
                </Publishers>
            </Bind>
            <VirtualHosts>
                <VirtualHost>
                    <Name>default</Name>
                    <Host>
                        <Names>
                            <Name>efoo.example.com</Name>
                        </Names>
                        <TLS>
                            <CertPath>/opt/ovenmediaengine/bin/edge_certs/tls.crt</CertPath>
                            <KeyPath>/opt/ovenmediaengine/bin/edge_certs/tls.key</KeyPath>
                            <ChainCertPath>/opt/ovenmediaengine/bin/edge_certs/tls.crt</ChainCertPath>
                        </TLS>
                    </Host>
                    <Distribution>example.com</Distribution>
                    <OriginMapStore>
                        <RedisServer>
                            <Host>ovenmediaengine-redis-master:6379</Host>
                            <Auth>test</Auth>
                        </RedisServer>
                    </OriginMapStore>
                </VirtualHost>
            </VirtualHosts>
        </Server>

Origin:

<?xml version="1.0" encoding="UTF-8"?>
        <Server version="8">
            <Name>Origin</Name>
            <IP>*</IP>
            <Type>origin</Type>
            <PrivacyProtection>false</PrivacyProtection>
            <StunServer>stun.l.google.com:19302</StunServer>
            <Managers>
                <Host>
                    <Names>
                        <Name>*</Name>
                    </Names>
                </Host>
                <API>
                    <AccessToken>test</AccessToken>
                </API>
            </Managers>
            <Bind>
                <Managers>
                    <API>
                        <Port>8081</Port>
                        <WorkerCount>1</WorkerCount>
                    </API>
                </Managers>
                <Providers>
                    <RTMP>
                        <Port>1935</Port>
                        <WorkerCount>1</WorkerCount>
                    </RTMP>
                    <WebRTC>
                        <Signalling>
                            <Port>3333</Port>
                            <TLSPort>3334</TLSPort>
                            <WorkerCount>1</WorkerCount>
                        </Signalling>
                        <IceCandidates>
                            <IceCandidate>*:10000-10005/udp</IceCandidate>
                            <!-- 
                                If you want to stream WebRTC over TCP, specify IP:Port for TURN server.
                                This uses the TURN protocol, which delivers the stream from the built-in TURN server to the player's TURN client over TCP. 
                                For detailed information, refer https://airensoft.gitbook.io/ovenmediaengine/streaming/webrtc-publishing#webrtc-over-tcp
                            -->
                            <TcpRelay>*:3478</TcpRelay>
                            <!-- TcpForce is an option to force the use of TCP rather than UDP in WebRTC streaming. (You can omit ?transport=tcp accordingly.) If <TcpRelay> is not set, playback may fail. -->
                            <TcpForce>true</TcpForce>
                            <TcpRelayWorkerCount>1</TcpRelayWorkerCount>
                        </IceCandidates>
                    </WebRTC>
                </Providers>
                <Publishers>
                    <OVT>
                        <Port>9000</Port>
                    </OVT>
                </Publishers>
            </Bind>
            <VirtualHosts>
                <VirtualHost>
                    <Name>default</Name>
                    <Host>
                        <Names>
                            <Name>foo.example.com</Name>
                            <Name>ovenmediaengine-origin</Name>
                        </Names>
                        <TLS>
                            <CertPath>/opt/ovenmediaengine/bin/origin_certs/tls.crt</CertPath>
                            <KeyPath>/opt/ovenmediaengine/bin/origin_certs/tls.key</KeyPath>
                            <ChainCertPath>/opt/ovenmediaengine/bin/origin_certs/tls.crt</ChainCertPath>
                        </TLS>
                    </Host>
                    <OriginMapStore>
                        <!-- In order to use OriginMap, you must enable OVT Publisher in Origin and OVT Provider in Edge. -->
                        <RedisServer>
                            <Host>ovenmediaengine-redis-master:6379</Host>
                            <Auth>test</Auth>
                        </RedisServer>

                        <!-- This is only needed for the origin server and used to register the ovt address of the stream.  -->
                        <OriginHostName>${env:POD_IP}</OriginHostName>
                    </OriginMapStore>
                    <Applications>
                        <Application>
                            <Name>app</Name>
                            <Type>live</Type>
                            <OutputProfiles>
                                <HardwareAcceleration>false</HardwareAcceleration>
                                <OutputProfile>
                                    <Name>abr</Name>
                                    <OutputStreamName>${OriginStreamName}</OutputStreamName>
                                    <!--LLHLS URL : https://domain/app/stream/abr.m3u8 --> 
                                    <Playlist>
                                        <Name>For LLHLS</Name>
                                        <FileName>abr</FileName>
                                        <Options> <!-- Optinal -->
                                            <!-- 
                                            Automatically switch rendition in WebRTC ABR 
                                            [Default] : true
                                            -->
                                            <WebRtcAutoAbr>true</WebRtcAutoAbr> 
                                        </Options>
                                        <Rendition>
                                            <Name>Bypass</Name>
                                            <Video>bypass_video</Video>
                                            <Audio>bypass_audio</Audio>
                                        </Rendition>
                                        <Rendition>
                                            <Name>FHD</Name>
                                            <Video>video_1280</Video>
                                            <Audio>bypass_audio</Audio>
                                        </Rendition>
                                        <Rendition>
                                            <Name>HD</Name>
                                            <Video>video_720</Video>
                                            <Audio>bypass_audio</Audio>
                                        </Rendition>
                                        <Rendition>
                                            <Name>SD</Name>
                                            <Video>video_480</Video>
                                            <Audio>bypass_audio</Audio>
                                        </Rendition>
                                    </Playlist>
                                    <!--LLHLS URL : https://domain/app/stream/llhls.m3u8 --> 
                                    <Playlist>
                                        <Name>Change Default</Name>
                                        <FileName>llhls</FileName>
                                        <Rendition>
                                            <Name>HD</Name>
                                            <Video>video_720</Video>
                                            <Audio>bypass_audio</Audio>
                                        </Rendition>
                                    </Playlist> 
                                    <Encodes>
                                        <Audio>
                                            <Name>bypass_audio</Name>
                                            <Bypass>true</Bypass>
                                        </Audio>
                                        <Video>
                                            <Name>bypass_video</Name>
                                            <Bypass>true</Bypass>
                                        </Video>
                                        <Audio>
                                            <Codec>opus</Codec>
                                            <Bitrate>128000</Bitrate>
                                            <Samplerate>48000</Samplerate>
                                            <Channel>2</Channel>
                                        </Audio>
                                        <Video>
                                            <Name>video_1280</Name>
                                            <Codec>h264</Codec>
                                            <Bitrate>5024000</Bitrate>
                                            <Framerate>30</Framerate>
                                            <Height>1280</Height>
                                            <Preset>faster</Preset>
                                        </Video>
                                        <Video>
                                            <Name>video_720</Name>
                                            <Codec>h264</Codec>
                                            <Bitrate>2024000</Bitrate>
                                            <Framerate>30</Framerate>
                                            <Height>720</Height>
                                            <Preset>faster</Preset>
                                        </Video>
                                        <Video>
                                            <Name>video_480</Name>
                                            <Codec>h264</Codec>
                                            <Bitrate>506000</Bitrate>
                                            <Framerate>30</Framerate>
                                            <Height>480</Height>
                                            <Preset>faster</Preset>
                                        </Video>
                                    </Encodes>
                                </OutputProfile>
                            </OutputProfiles>
                            <Providers>
                                <WebRTC />
                                <RTMP />
                            </Providers>
                            <Publishers>
                                <AppWorkerCount>1</AppWorkerCount>
                                <StreamWorkerCount>8</StreamWorkerCount>
                                <OVT />
                            </Publishers>
                        </Application>
                    </Applications>
                </VirtualHost>
            </VirtualHosts>
        </Server>
getroot commented 2 years ago

@guyguy333 Would you like to add Application to Edge and enable OVT Provider? I don't know if OVT is enabled by default because the default application policy is currently unclear. Or post your entire edge log so I can check it out.

guyguy333 commented 2 years ago

Thanks @getroot. I just added section and enabled OVT provider. Result is the same. Here is full log for Edge and full Edge config.

ovenmediaengine-edge-5446998599-xbbd8.log

Edge:

<?xml version="1.0" encoding="UTF-8"?>
        <Server version="8">
            <Name>edge</Name>
            <IP>*</IP>
            <Type>edge</Type>
            <PrivacyProtection>false</PrivacyProtection>
            <StunServer>stun.l.google.com:19302</StunServer>
            <Modules>
                <!-- 
                Currently OME only supports h2 like all browsers do. Therefore, HTTP/2 only works on TLS ports.         
                -->
                <HTTP2>
                    <Enable>true</Enable>
                </HTTP2>
                <!-- P2P works only in WebRTC and is experiment feature -->
                <!-- 
                <P2P>
                    <Enable>false</Enable>
                    <MaxClientPeersPerHostPeer>2</MaxClientPeersPerHostPeer>
                </P2P>
                -->
            </Modules>
            <Bind>
                <Providers />
                <Publishers>
                    <WebRTC>
                        <Signalling>
                            <Port>4333</Port>
                            <TLSPort>4334</TLSPort>
                            <WorkerCount>1</WorkerCount>
                        </Signalling>
                        <IceCandidates>
                            <IceCandidate>*:11000-11005/udp</IceCandidate>
                            <TcpRelay>*:4478</TcpRelay>
                            <TcpForce>true</TcpForce>
                            <TcpRelayWorkerCount>1</TcpRelayWorkerCount>
                        </IceCandidates>
                    </WebRTC>
                </Publishers>
            </Bind>
            <VirtualHosts>
                <VirtualHost>
                    <Name>default</Name>
                    <Host>
                        <Names>
                            <Name>efoo.example.com</Name>
                        </Names>
                        <TLS>
                            <CertPath>/opt/ovenmediaengine/bin/edge_certs/tls.crt</CertPath>
                            <KeyPath>/opt/ovenmediaengine/bin/edge_certs/tls.key</KeyPath>
                            <ChainCertPath>/opt/ovenmediaengine/bin/edge_certs/tls.crt</ChainCertPath>
                        </TLS>
                    </Host>
                    <Distribution>example.com</Distribution>
                    <OriginMapStore>
                        <RedisServer>
                            <Host>ovenmediaengine-redis-master:6379</Host>
                            <Auth>test</Auth>
                        </RedisServer>
                    </OriginMapStore>
                    <Applications>
                      <Application>
                          <Name>app</Name>
                          <Type>live</Type>
                          <OutputProfiles>
                              <OutputProfile>
                                  <Name>passthrough</Name>
                                  <OutputStreamName>${OriginStreamName}</OutputStreamName>
                                  <Encodes>
                                      <Video>
                                          <Bypass>true</Bypass>
                                      </Video>
                                      <Audio>
                                          <Bypass>true</Bypass>
                                      </Audio>
                                  </Encodes>
                              </OutputProfile>
                          </OutputProfiles>
                          <Providers>
                              <OVT />
                          </Providers>
                          <Publishers>
                              <AppWorkerCount>1</AppWorkerCount>
                              <StreamWorkerCount>8</StreamWorkerCount>
                              <WebRTC>
                                  <Timeout>30000</Timeout>
                                  <Rtx>false</Rtx>
                                  <Ulpfec>false</Ulpfec>
                                  <JitterBuffer>false</JitterBuffer>
                              </WebRTC>
                              <LLHLS>
                                  <ChunkDuration>0.5</ChunkDuration>
                                  <SegmentDuration>6</SegmentDuration>
                                  <SegmentCount>10</SegmentCount>
                                  <CrossDomains>
                                      <Url>*</Url>
                                  </CrossDomains>
                              </LLHLS>
                          </Publishers>
                      </Application>
                    </Applications>
                </VirtualHost>
            </VirtualHosts>
        </Server>
chakphanu commented 1 year ago

Thanks @getroot. I just added section and enabled OVT provider. Result is the same. Here is full log for Edge and full Edge config.

ovenmediaengine-edge-5446998599-xbbd8.log

Edge:

<?xml version="1.0" encoding="UTF-8"?>
        <Server version="8">
            <Name>edge</Name>
            <IP>*</IP>
            <Type>edge</Type>
            <PrivacyProtection>false</PrivacyProtection>
            <StunServer>stun.l.google.com:19302</StunServer>
            <Modules>
                <!-- 
                Currently OME only supports h2 like all browsers do. Therefore, HTTP/2 only works on TLS ports.           
                -->
                <HTTP2>
                    <Enable>true</Enable>
                </HTTP2>
                <!-- P2P works only in WebRTC and is experiment feature -->
                <!-- 
                <P2P>
                    <Enable>false</Enable>
                    <MaxClientPeersPerHostPeer>2</MaxClientPeersPerHostPeer>
                </P2P>
                -->
            </Modules>
            <Bind>
                <Providers />
                <Publishers>
                    <WebRTC>
                        <Signalling>
                            <Port>4333</Port>
                            <TLSPort>4334</TLSPort>
                            <WorkerCount>1</WorkerCount>
                        </Signalling>
                        <IceCandidates>
                            <IceCandidate>*:11000-11005/udp</IceCandidate>
                            <TcpRelay>*:4478</TcpRelay>
                            <TcpForce>true</TcpForce>
                            <TcpRelayWorkerCount>1</TcpRelayWorkerCount>
                        </IceCandidates>
                    </WebRTC>
                </Publishers>
            </Bind>
            <VirtualHosts>
                <VirtualHost>
                    <Name>default</Name>
                    <Host>
                        <Names>
                            <Name>efoo.example.com</Name>
                        </Names>
                        <TLS>
                            <CertPath>/opt/ovenmediaengine/bin/edge_certs/tls.crt</CertPath>
                            <KeyPath>/opt/ovenmediaengine/bin/edge_certs/tls.key</KeyPath>
                            <ChainCertPath>/opt/ovenmediaengine/bin/edge_certs/tls.crt</ChainCertPath>
                        </TLS>
                    </Host>
                    <Distribution>example.com</Distribution>
                    <OriginMapStore>
                        <RedisServer>
                            <Host>ovenmediaengine-redis-master:6379</Host>
                            <Auth>test</Auth>
                        </RedisServer>
                    </OriginMapStore>
                    <Applications>
                      <Application>
                          <Name>app</Name>
                          <Type>live</Type>
                          <OutputProfiles>
                              <OutputProfile>
                                  <Name>passthrough</Name>
                                  <OutputStreamName>${OriginStreamName}</OutputStreamName>
                                  <Encodes>
                                      <Video>
                                          <Bypass>true</Bypass>
                                      </Video>
                                      <Audio>
                                          <Bypass>true</Bypass>
                                      </Audio>
                                  </Encodes>
                              </OutputProfile>
                          </OutputProfiles>
                          <Providers>
                              <OVT />
                          </Providers>
                          <Publishers>
                              <AppWorkerCount>1</AppWorkerCount>
                              <StreamWorkerCount>8</StreamWorkerCount>
                              <WebRTC>
                                  <Timeout>30000</Timeout>
                                  <Rtx>false</Rtx>
                                  <Ulpfec>false</Ulpfec>
                                  <JitterBuffer>false</JitterBuffer>
                              </WebRTC>
                              <LLHLS>
                                  <ChunkDuration>0.5</ChunkDuration>
                                  <SegmentDuration>6</SegmentDuration>
                                  <SegmentCount>10</SegmentCount>
                                  <CrossDomains>
                                      <Url>*</Url>
                                  </CrossDomains>
                              </LLHLS>
                          </Publishers>
                      </Application>
                    </Applications>
                </VirtualHost>
            </VirtualHosts>
        </Server>

v0.14.10 == ok v0.14.11 == Could not find Origin for the stream

this commit make edge OriginMapStore fail/crash. commit 5a46220190dceea2840e8844ca33a0c627faf1da (HEAD) Author: getroot getroot@airensoft.com Date: Wed Oct 5 21:13:57 2022 +0900

Added RequestPullStream with url vector

[11-02 03:41:29.883] I [SPRtcSig-T3333:152246] Publisher | publisher.cpp:186 | Try to pull stream from local origin map: [#edge#app/test] [11-02 03:41:29.898] E [SPRtcSig-T3333:152246] Orchestrator | orchestrator.cpp:634 | Could not find Origin for the stream: [#edge#app/test] [11-02 03:41:29.900] I [SPRtcSig-T3333:152246] Publisher | publisher.cpp:190 | Try to pull stream from origin map store: [#edge#app/test] ==152234== Can't extend stack to 0x8593088 during signal delivery for thread 5: ==152234== no stack segment ==152234== ==152234== Process terminating with default action of signal 11 (SIGSEGV) ==152234== Access not within mapped region at address 0x8593088 ==152234== at 0x121B4B8: ocst::Orchestrator::RequestPullStream(std::shared_ptr const&, info::VHostAppName const&, ov::String const&, ov::String const&, long) (orchestrator.cpp:606) ==152234== If you believe this happened as a result of a stack ==152234== overflow in your program's main thread (unlikely but ==152234== possible), you can try to increase the size of the ==152234== main thread stack using the --main-stacksize= flag. ==152234== The main thread stack size used in this run was 8388608. ==152234== Thread 5: ==152234== Invalid write of size 8 ==152234== at 0x4833134: _vgnU_freeres (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_core-amd64-linux.so) ==152234== Address 0x8593f48 is on thread 5's stack ==152234== ==152234== ==152234== Process terminating with default action of signal 11 (SIGSEGV) ==152234== Bad permissions for mapped region at address 0x8593F48 ==152234== at 0x4833134: _vgnU_freeres (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_core-amd64-linux.so) ==152234== ==152234== HEAP SUMMARY: ==152234== in use at exit: 2,671,316 bytes in 31,503 blocks ==152234== total heap usage: 133,868 allocs, 102,365 frees, 7,851,165 bytes allocated ==152234== ==152234== Searching for pointers to 31,503 not-freed blocks ==152234== Checked 181,349,328 bytes ==152234== ==152234== LEAK SUMMARY: ==152234== definitely lost: 64 bytes in 1 blocks ==152234== indirectly lost: 3 bytes in 1 blocks ==152234== possibly lost: 6,818 bytes in 23 blocks ==152234== still reachable: 2,664,431 bytes in 31,478 blocks ==152234== suppressed: 0 bytes in 0 blocks ==152234== Rerun with --leak-check=full to see details of leaked memory ==152234== ==152234== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0) ==152234== ==152234== 1 errors in context 1 of 1: ==152234== Invalid write of size 8 ==152234== at 0x4833134: _vgnU_freeres (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_core-amd64-linux.so) ==152234== Address 0x8593f48 is on thread 5's stack ==152234== ==152234== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0) Segmentation fault (core dumped)

getroot commented 1 year ago

@chakphanu Thanks for the detailed report. Could you please make this a new issue?