search

sergey-dryabzhinsky / nginx-rtmp-module

NGINX-based Media Streaming Server
http://nginx-rtmp.blogspot.com
BSD 2-Clause "Simplified" License
1.02k stars 216 forks source link

Relay does not reconnect after publisher dies #310

Open mauricioabreu opened 4 years ago

mauricioabreu commented 4 years ago

Context

I use RTMP to receive streaming from OBS, Elemental, and ffmpeg. Recently I created a new service to transcode these streamings, generating more bitrate outputs:

Source (OBS, Elemetal, ffmpeg) ----> RTMP ingest service ----> RTMP transcoding service

RTMP ingest and transcoding service runs on different servers.

if the path for the stream ins live it does not transcode. If it is transcode it is relayed to the transcoding service (there I run ffmpeg to push the stream to another RTMP service to use the segmenting feature (HLS))

Relays are created using Location header

Problem

When my transcoder service dies (network, hardware failures, etc) I see that the RTMP ingest service does not try to relay it again, forcing me to restart the OBS/Elemental/ffmpeg manually.

How can I make it reconnect without human intervention?

kerkradio commented 4 years ago

Good morning

The basic question about reconnecting a strategy is something that I have interest in too.

Many thanks.

From: Mauricio Antunes notifications@github.com Sent: Wednesday, 17 June 2020 23:33 To: sergey-dryabzhinsky/nginx-rtmp-module nginx-rtmp-module@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: [sergey-dryabzhinsky/nginx-rtmp-module] Relay does not reconnect after publisher dies (#310)

Context

I use RTMP to receive streaming from OBS, Elemental, and ffmpeg. Recently I created a new service to transcode these streamings, generating more bitrate outputs:

Source (OBS, Elemetal, ffmpeg) ----> RTMP ingest service ----> RTMP transcoding service

RTMP ingest and transcoding service runs on different servers.

if the path for the stream ins live it does not transcode. If it is transcode it is relayed to the transcoding service (there I run ffmpeg to push the stream to another RTMP service to use the segmenting feature (HLS))

Problem

When my transcoder service dies (network, hardware failures, etc) I see that the RTMP ingest service does not try to relay it again, forcing me to restart the OBS/Elemental/ffmpeg manually.

How can I make it reconnect without human intervention?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/sergey-dryabzhinsky/nginx-rtmp-module/issues/310, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AKNJMY7K5OTFWXMD6J3ILB3RXGDHJANCNFSM4OBFR2JA.

entonbiba commented 3 years ago

why not use the on publish done? You would run a php script to end ffmpeg automatically.

exec_publish_done

mauricioabreu commented 3 years ago

@entonbiba what if my server dies? exec_publish_done would work if my server does not die.

Source (OBS, Elemetal, ffmpeg) ----> RTMP ingest service ----> RTMP transcoding service

Here my RTMP transcoding service host dies and the RTMP ingest service does not reconnect ever.

entonbiba commented 3 years ago

@mauricioabreu is the RTMP ingest service doing a push to the RTMP transcoding service? If so then the transcoding service should be picking up the data feed after it comes online.

The following should work: RTMP INGEST SERVICE SERVER

server { 
    listen 1935;

    chunk_size 4096;
    ping 15s;
    ping_timeout 30s;

    application live { 
        push rtmp://IPHERE/live/live;
    }
}

RTMP TRANSCODING SERVICE SERVER

server { 
    listen 1935;

    chunk_size 4096;
    ping 5s;
    ping_timeout 15s;

    application live { 
        live on; 
        interleave on;

        hls on; 
        hls_path /home/hls/; 
        hls_fragment 5s; 
        hls_playlist_length 30;
        hls_sync 400ms;
        hls_continuous on; #Continuous mode.
        hls_cleanup on;    #Delete the extra slices.
        hls_nested on;     #Nested mode.
        hls_fragment_naming system;
    }
}
mauricioabreu commented 3 years ago

@entonbiba I don't use push, I use relay mode (HTTP location with IP)

Why do I use pushmethod? Because I use lua to authenticate the stream. If the user is able to push, then it will relay the stream to another ingest server

sergey-dryabzhinsky commented 3 years ago

So you need something like: push_reconnect 10s;, push_reconnect_count 60; ?

@mauricioabreu Can you provide your configs?

mauricioabreu commented 3 years ago

@sergey-dryabzhinsky here:

rtmp_auto_push off;

rtmp {
  max_streams <some_number>;
  access_log /opt/nginx/logs/access.log;

  server {
    listen 1935;
    drop_idle_publisher 120s;

    application live {
      live on;
      record off;
      notify_method get;
      notify_update_timeout 5m;
      on_publish http://127.0.0.1:8080/segmenter;
      on_update http://127.0.0.1:8080/verify;
    }

    application transcode {
      live on;
      record off;
      notify_method get;
      notify_update_timeout 5m;
      on_publish http://127.0.0.1:8080/transcoder;
      on_update http://127.0.0.1:8080/verify;
    }
  }
}

push_reconnect has a default value of 3 seconds. I don't know if raising it will fix the problem but I can give it a try.

mauricioabreu commented 3 years ago

@sergey-dryabzhinsky push_reconnect_count option does not seem to exist. Right?

sergey-dryabzhinsky commented 3 years ago

Right

mauricioabreu commented 3 years ago

Logs after disconnecting:

live-ingest-66f7b7cb5d-v55bz nginx-rtmp 2021/08/17 21:28:07 [info] 25#0: *38 recv() failed (104: Connection reset by peer), client: 10.231.192.52/live_5/stream1_2316_0, server: ngx-relay
live-ingest-66f7b7cb5d-v55bz nginx-rtmp 2021/08/17 21:28:07 [info] 25#0: *38 disconnect, client: 10.231.192.52/live_5/stream1_2316_0, server: ngx-relay
live-ingest-66f7b7cb5d-v55bz nginx-rtmp 2021/08/17 21:28:07 [info] 25#0: *38 deleteStream, client: 10.231.192.52/live_5/stream1_2316_0, server: ngx-relay
live-ingest-66f7b7cb5d-v55bz nginx-rtmp 2021/08/17 21:28:07 [info] 25#0: *30 recv() failed (104: Connection reset by peer), client: 10.231.192.52/live_5/stream1_1485_0, server: ngx-relay
live-ingest-66f7b7cb5d-v55bz nginx-rtmp 2021/08/17 21:28:07 [info] 25#0: *21 disconnect, client: 10.231.192.52/live_5/stream1_964_0, server: ngx-relay
live-ingest-66f7b7cb5d-v55bz nginx-rtmp 2021/08/17 21:28:07 [info] 25#0: *21 deleteStream, client: 10.231.192.52/live_5/stream1_964_0, server: ngx-relay
live-ingest-66f7b7cb5d-v55bz nginx-rtmp 2021/08/17 21:28:07 [info] 25#0: *30 disconnect, client: 10.231.192.52/live_5/stream1_1485_0, server: ngx-relay
live-ingest-66f7b7cb5d-v55bz nginx-rtmp 2021/08/17 21:28:07 [info] 25#0: *30 deleteStream, client: 10.231.192.52/live_5/stream1_1485_0, server: ngx-relay
live-ingest-66f7b7cb5d-v55bz nginx-rtmp 2021/08/17 21:28:07 [info] 25#0: *34 disconnect, client: 10.231.192.52/live_5/stream1_264_0, server: ngx-relay
live-ingest-66f7b7cb5d-v55bz nginx-rtmp 2021/08/17 21:28:07 [info] 25#0: *34 deleteStream, client: 10.231.192.52/live_5/stream1_264_0, server: ngx-relay
live-ingest-66f7b7cb5d-v55bz nginx-rtmp 2021/08/17 21:28:07 [info] 25#0: *36 disconnect, client: 10.231.192.52/live_5/stream1_407_0, server: ngx-relay
live-ingest-66f7b7cb5d-v55bz nginx-rtmp 2021/08/17 21:28:07 [info] 25#0: *36 deleteStream, client: 10.231.192.52/live_5/stream1_407_0, server: ngx-relay
mauricioabreu commented 3 years ago

After some debugging, I discovered that publishing with 302 redirect does not behave the same as a simple push. When the upstream server dies/has network issues, it just does not reconnect.

Here is a demo: https://github.com/mauricioabreu/redirect-nginx-rtmp-reconnect-error

OBS.: I tried the same project without docker and NGINX 1.16

mauricioabreu commented 3 years ago

Just pushed a workaround: https://github.com/mauricioabreu/redirect-nginx-rtmp-reconnect-error/commit/58b0a7e80ea0e15a4b3c6c742d0f7f4ba8595739

mauricioabreu commented 3 years ago

I've just discovered where the bug is:

https://github.com/sergey-dryabzhinsky/nginx-rtmp-module/blob/8e344d799483145666fa875344bddf67a324e561/ngx_rtmp_relay_module.c#L1550-L1568

For some reason, it does not reconnect.