search

AirenSoft / OvenMediaEngine

OvenMediaEngine (OME) is a Sub-Second Latency Live Streaming Server with Large-Scale and High-Definition. #WebRTC #LLHLS
https://airensoft.com/ome.html
GNU Affero General Public License v3.0
2.61k stars 1.06k forks source link

startPush before stream start causes unusable state until OME restart #1343

Closed basisbit closed 1 year ago

basisbit commented 1 year ago

Describe the bug The restream feature and the API calls for it seem to not yet be reliable and run into deadlocks / unusable state until OME server restart in a couple of common use cases. This here issue is specifically for: send the restream API call to OME, wait a bit, then start streaming to OME, -> now the restream is unusable and does not restream no matter what until OME restart.

To Reproduce Steps to reproduce the behavior:

  1. Have no stream being sent to the OME instance
  2. Set Server.xml of origin and edge as follows: see https://github.com/basisbit/KitchenCDN/tree/7b71983d2f46c7c56322c1765f500fa57086a976
  3. Send a HTTP POST to /v1/vhosts/default/apps/yourappname:startPush with valid arguments for restreaming to Twitch.
  4. Wait a bit
  5. Only now start streaming with OBS to OME
  6. See that the restream did not automatically start
  7. See that doing the stopPush HTTP POST request runs into an error
  8. See that doing another startPush HTTP POST request always returns HTTP 400 "duplicate id" error

Expected behavior Either don't return HTTP 200 when doing a startPush on a not yet started steam or automatically start restreaming / rtmp pushing once the stream to OME started, also when using a common Origin->Edge setup.

Server (please complete the following information):

Player (please complete the following information):

Keukhan commented 1 year ago

thank you for reporting This is a bug that occurred during the recent push-related refactoring. Thanks to you I solved an important problem. patched in master branch. In addition to this issue, various crash bugs have been patched, and a new version will be released soon.

Thanks. have a good day :)

https://github.com/AirenSoft/OvenMediaEngine/compare/cfe71e3360a7...1fd1d2803650

basisbit commented 1 year ago

Thank you so much for the fast fix! I'll try to verify test it next week.

basisbit commented 1 year ago

@Keukhan for understanding, what is the expected behavior when sending a startPush for a not yet existing stream? Should it ignore the startPush instruction (and thus maybe return a HTTP error code) or should it add the instruction to a collection, which will start the restream once a fitting stream gets available (how would an edge container get to know that the origin now has this stream available)?

max-m commented 1 year ago

@basisbit Thanks for reporting this issue, I ran into part of it as well but I felt like I didn't have enough information yet to make a proper bug report. I knew that I could only start my stream once per "session", so I also discovered the workaround of restarting OME. Sadly I didn't note down the OME version I ran before I updated my Docker container to the latest version a couple of weeks ago, and restarting OME was easy enough so I didn't bother rolling back to an older version yet. :)

My setup is a little different from yours I think: I use an admission hook to automatically start (and stop) a re-stream to YouTube. So as soon as an authenticated stream starts my admission hook calls startPush (then allows the stream to start) and when a stream stops it calls stopPush. But starting a new stream using the same credentials would fail, I'm fairly sure I also saw that "duplicate id" error but I cannot for the life of me find it in my logs. There is also the curious HTTP error occurred: [HTTP] Success (200) after stopping the stream😅 That is probably the call to my admission hook which allows that request to stop the stream. 🤔

@Keukhan thanks for taking a look and also for the quick fix! 👍 I just built OME from commit 1fd1d28 and now I can start and stop my stream again without having to restart OME :)

Keukhan commented 1 year ago

@Keukhan for understanding, what is the expected behavior when sending a startPush for a not yet existing stream? Should it ignore the startPush instruction (and thus maybe return a HTTP error code) or should it add the instruction to a collection, which will start the restream once a fitting stream gets available (how would an edge container get to know that the origin now has this stream available)?

Even if a stream does not exist in OME, it can be registered by requesting it through the startPush API. When a stream is created afterwards, it is automatically pushed to the destination. Even if a stream is deleted and recreated, it is automatically pushed. Until the stopPush API is called... Of course, startPush/stopPush works even when Stream is already created.

And, In the state where startPush is called in Edge while there is no stream in Origin, even if a stream is created in Origin, push does not work in Edge. Like @max-m said, it seems like a good idea to use the Admission Webhook.

basisbit commented 1 year ago

Retested this issue and it seems to be fixed with fb377a2. Thank you again for the quick fix!

basisbit commented 1 year ago

There is also the curious HTTP error occurred: [HTTP] Success (200) after stopping the stream😅

this part is not yet fixed. For reproducing:

  1. start stream
  2. start rtmp restream
  3. stop stream
  4. do the call for stopping restream
Keukhan commented 1 year ago

There is also the curious HTTP error occurred: [HTTP] Success (200) after stopping the stream😅

this part is not yet fixed. For reproducing:

  1. start stream
  2. start rtmp restream
  3. stop stream
  4. do the call for stopping restream

I will check again :)