search

bluenviron / mediamtx

Ready-to-use SRT / WebRTC / RTSP / RTMP / LL-HLS media server and media proxy that allows to read, publish, proxy, record and playback video and audio streams.
MIT License
12.51k stars 1.55k forks source link

runOnDemandCloseAfter blocks the path after teardown #3455

Closed mrlt8 closed 4 months ago

mrlt8 commented 5 months ago

Which version are you using?

v1.8.2

Which operating system are you using?

Describe the issue

Using runOnDemand with runOnDemandCloseAfter seems to create an issue where the path will block and become unresponsive until the runOnDemandCloseAfter times out even if the path is destroyed.

Can we get runOnDemandCloseAfter to reset and run immediately if the path is destroyed or a way to reset it?

Or is there a way to clear out and reset the path with runOnNotReady?

Describe how to replicate the issue

  1. Set runOnDemandCloseAfter: 60s
  2. start the server
  3. read the path
  4. stop the path source
  5. attempt to read the path again before the timeout
  6. attempt to read the path again after the timeout

Did you attach the server logs?

<!-- STOP SOURCE -->

07:07:45 DEB [RTSP] [conn 127.0.0.1:37054] [c->s] TEARDOWN rtsp://0.0.0.0:8554/front-door RTSP/1.0
CSeq: 6
Session: 6e3581c500424bef8d4695af4e4ac9a8
User-Agent: Lavf60.3.100

07:07:45 INF [path front-door] runOnReady command stopped
07:07:45 INF [path front-door] runOnNotReady command launched
07:07:45 INF [RTSP] [session de560115] destroyed: torn down by 127.0.0.1:37054
07:07:45 DEB [RTSP] [conn 127.0.0.1:37054] [s->c] RTSP/1.0 200 OK
CSeq: 6
Server: gortsplib

07:07:45 INF [RTSP] [conn 127.0.0.1:37054] closed: EOF

<!-- ATTEMPT TO OPEN PATH FAILS -->
07:08:15 INF [RTSP] [conn 192.168.65.1:53748] opened
07:08:15 DEB [RTSP] [conn 192.168.65.1:53748] [c->s] OPTIONS rtsp://localhost:8554/front-door RTSP/1.0
CSeq: 2
User-Agent: LibVLC/3.0.20 (LIVE555 Streaming Media v2016.11.28)

07:08:15 DEB [RTSP] [conn 192.168.65.1:53748] [s->c] RTSP/1.0 200 OK
CSeq: 2
Public: DESCRIBE, ANNOUNCE, SETUP, PLAY, RECORD, PAUSE, GET_PARAMETER, TEARDOWN
Server: gortsplib

07:08:15 DEB [RTSP] [conn 192.168.65.1:53748] [c->s] DESCRIBE rtsp://localhost:8554/front-door RTSP/1.0
Accept: application/sdp
CSeq: 3
User-Agent: LibVLC/3.0.20 (LIVE555 Streaming Media v2016.11.28)

07:08:20 INF [RTSP] [conn 192.168.65.1:32700] opened
07:08:20 DEB [RTSP] [conn 192.168.65.1:32700] [c->s] SETUP rtsp://localhost:8554/front-door RTSP/1.0
CSeq: 0
Transport: RTP/AVP;unicast;client_port=9346-9347

07:08:20 INF [RTSP] [session bf9e0231] created by 192.168.65.1:32700
07:08:20 DEB [RTSP] [conn 192.168.65.1:32700] [s->c] RTSP/1.0 461 Unsupported Transport
CSeq: 0
Server: gortsplib
Session: 3abe25442c0b42218fa1d0ae00783bb2

07:08:20 INF [RTSP] [conn 192.168.65.1:32700] closed: invalid URL ((null))
07:08:20 INF [RTSP] [session bf9e0231] destroyed: not in use
07:08:20 INF [RTSP] [conn 192.168.65.1:24446] opened
07:08:20 DEB [RTSP] [conn 192.168.65.1:24446] [c->s] OPTIONS rtsp://localhost:8554 RTSP/1.0
CSeq: 1
Clientchallenge: 9e26d33f2984236010ef6253fb1887f7
Clientid: Linux_2.4_6.0.9.1235_play32_RN01_EN_586
Companyid: KnKV4M4I/B2FjJ1TToLycw==
Guid: 00000000-0000-0000-0000-000000000000
Playerstarttime: [28/03/2003:22:50:23 00:00]
Regiondata: 0
User-Agent: RealMedia Player Version 6.0.9.1235 (linux-2.0-libc6-i386-gcc2.95)

07:08:20 DEB [RTSP] [conn 192.168.65.1:24446] [s->c] RTSP/1.0 200 OK
CSeq: 1
Public: DESCRIBE, ANNOUNCE, SETUP, PLAY, RECORD, PAUSE, GET_PARAMETER, TEARDOWN
Server: gortsplib

07:08:20 INF [RTSP] [conn 192.168.65.1:24446] closed: EOF

<!-- runOnDemand FINALLY TIMES OUT -->
07:08:40 INF [path front-door] runOnDemand command stopped: not needed by anyone
07:08:40 INF [path front-door] runOnUnDemand command launched

<!-- ATTEMPT TO OPEN PATH SUCCEEDS -->
07:08:51 INF [RTSP] [conn 192.168.65.1:60874] opened
07:08:51 DEB [RTSP] [conn 192.168.65.1:60874] [c->s] OPTIONS rtsp://localhost:8554/front-door RTSP/1.0
CSeq: 2
User-Agent: LibVLC/3.0.20 (LIVE555 Streaming Media v2016.11.28)

07:08:51 DEB [RTSP] [conn 192.168.65.1:60874] [s->c] RTSP/1.0 200 OK
CSeq: 2
Public: DESCRIBE, ANNOUNCE, SETUP, PLAY, RECORD, PAUSE, GET_PARAMETER, TEARDOWN
Server: gortsplib

07:08:51 DEB [RTSP] [conn 192.168.65.1:60874] [c->s] DESCRIBE rtsp://localhost:8554/front-door RTSP/1.0
Accept: application/sdp
CSeq: 3
User-Agent: LibVLC/3.0.20 (LIVE555 Streaming Media v2016.11.28)

07:08:51 INF [path front-door] runOnDemand command started

Did you attach a network dump?

no

moenodedev commented 5 months ago

Would this be the cause of the API and path outputs freezing?

EDIT: #3541 is the issue

bybiz98 commented 4 months ago

I meet this similar problem，as runOnDemandCloseAfter is default to 10s, this issue should be a common problem while using OnDemand source/publisher. My log as below: 2024/07/09 09:49:31 INF [RTSP] [conn 127.0.0.1:64348] opened 2024/07/09 09:49:31 INF [RTSP] [session b3be0d4a] created by 127.0.0.1:64348 2024/07/09 09:49:31 INF [RTSP] [session b3be0d4a] is publishing to path 'test', 2 tracks (H265, MPEG-4 Audio) 2024/07/09 09:49:31 INF [HLS] [muxer test] is converting into HLS, 2 tracks (H265, MPEG-4 Audio) 2024/07/09 09:50:00 INF [RTSP] [conn 127.0.0.1:64348] closed: read tcp 127.0.0.1:8554->127.0.0.1:64348: wsarecv: An existing connection was forcibly closed by the remote host. 2024/07/09 09:50:00 INF [path test] runOnDemand command exited: command exited with code 1 2024/07/09 09:50:05 INF [path test] runOnDemand command exited: command exited with code 1 2024/07/09 09:50:10 INF [RTSP] [session b3be0d4a] destroyed: session timed out 2024/07/09 09:50:10 INF [HLS] [muxer test] destroyed: terminated 2024/07/09 09:50:10 INF [path test] runOnDemand command exited: command exited with code 1 2024/07/09 09:50:10 INF [HLS] [muxer test] created (requested by [::1]:64352) 2024/07/09 09:50:20 INF [path test] runOnDemand command stopped: not needed by anyone

fix patch proposal as below: diff --git a/internal/core/path.go b/internal/core/path.go index 2119927c..0f12d8ae 100644 --- a/internal/core/path.go +++ b/internal/core/path.go @@ -531,6 +531,9 @@ func (pa *path) doAddReader(req defs.PathAddReaderReq) { } ` if pa.conf.HasOnDemandStaticSource() { + if pa.onDemandStaticSourceState == pathOnDemandStateClosing { + pa.onDemandStaticSourceStop("error: static source terminated unexceptedly!") + } if pa.onDemandStaticSourceState == pathOnDemandStateInitial { pa.onDemandStaticSourceStart(req.AccessRequest.Query) } @@ -539,6 +542,9 @@ func (pa *path) doAddReader(req defs.PathAddReaderReq) { } if pa.conf.HasOnDemandPublisher() { + if pa.onDemandPublisherState == pathOnDemandStateClosing { + pa.onDemandPublisherStop("error: publisher terminated unexceptedly!") + } if pa.onDemandPublisherState == pathOnDemandStateInitial { pa.onDemandPublisherStart(req.AccessRequest.Query) }`

diff.patch

aler9 commented 4 months ago

Hello, the problem of both proposed solutions (https://github.com/bluenviron/mediamtx/issues/3455#issuecomment-2216296201 and #3507) is that they prevent runOnDemandRestart: yes from working, which is also the correct way to address the issue, although it's slightly slower, since the command is restarted after some seconds and not immediately.

Furthermore, runOnDemandRestart: yes makes the run-on-demand logic to behave exactly live the source-on-demand logic, reducing the number of possible server states and therefore the number of possible bugs.

Therefore, a proper fix consists in making runOnDemandRestart: yes the default option.

mrlt8 commented 4 months ago

Unfortunately, runOnDemandRestart: yes won't work in my situation since I use runOnDemand and runOnUnDemand to start/stop a camera publishing to MTX, and setting runOnDemandRestart: yes will causes runOnDemand to constantly trigger the script to start the camera:

runOnDemand command exited: command exited with code 0
runOnDemand command exited: command exited with code 0
runOnDemand command exited: command exited with code 0
...

Is there a way we can still trigger pa.onDemandPublisherStop("publisher removed") if runOnDemandRestart: no is set?

aler9 commented 4 months ago

@mrlt8 in every operating system or programming language there are two ways to start subprocesses:

The situation you are describing is a mixture between both cases - basically you want to start a background process but keep the monitoring logic into the server. We cannot support this scenario since it's too specific. Adapt your scripts to run either as foreground processes or background processes with a restart logic.

mrlt8 commented 4 months ago

Sorry, @aler9 it seems there might have been a bit of a misunderstanding.

I don't need mediaMTX to monitor or restart the subprocess. I'm just looking for a way to clear out the onDemandPublisherCloseTimer when the on demand publisher closes.

The onDemandPublisherCloseTimer becomes unnecessary once the onDemandPublisher has closed, especially if runOnDemandRestart is set to false.

Thanks for your help!

aler9 commented 4 months ago

The onDemandPublisherCloseTimer becomes unnecessary once the onDemandPublisher has closed

This is not true, the on-demand publisher might be in the process of restarting, as described in the background logic mentioned above. Adding this in the server would cause a service, which is already started, to start again.

Summing up, the logic you are describing is not compatible with the most common use cases for the run-on-demand command and it's also confusing since it mixes up different layers. Switch to one of the two logics written above.

mrlt8 commented 4 months ago

Thanks for the quick reply, @aler9.

If runOnDemandRestart is set to false, the publisher shouldn't be attempting to publish to the path once the runOnNotReady command is launched. However, I suppose a temporary connection issue could cause the path to appear down even if it's still publishing, as you're describing?

Is there an alternate way we might be able to clear the onDemandPublisherCloseTimer, perhaps through the API or as an option with runOnNotReady?

Thanks again for looking into this!