Origin server suddenly disconnected from Edge servers

[mulder0103]

Hello,

Currently, the SRS cluster occasionally experiences a situation where the Edge Server cannot fetch the stream from the Origin Server, but it recovers after a while (about 2 minutes). It has been confirmed that the broadcaster is continuously streaming to the Origin Server, so we can exclude any issues on the broadcaster side. Upon further investigation of the logs, it was found that when the resize iovs 256 => 512, max_msgs=128 occurs, the Origin Server starts to disconnect from all Edge Servers one by one.

The SRS versions used are all the latest version 2.0.243.

The session log of the Origin Server and one of the Edges.

4260:[2018-04-04 14:34:12.217][trace][24880][20933] RTMP client ip=13.113.152.108 4262:[2018-04-04 14:34:12.390][trace][24880][20933] complex handshake success 4265:[2018-04-04 14:34:12.562][trace][24880][20933] connect app, tcUrl=rtmp://34.199.141.251/live, pageUrl=, swfUrl=, schema=rtmp, vhost=defaultVhost, port=1935, app=live, args=(obj) 4266:[2018-04-04 14:34:12.562][trace][24880][20933] edge-srs ip=172.31.31.45, version=2.0.243, pid=22770, id=3483 4275:[2018-04-04 14:34:13.115][trace][24880][20933] client identified, type=Play, stream_name=990143, duration=-1.00 4276:[2018-04-04 14:34:13.115][trace][24880][20933] source url=/live/990143, ip=13.113.152.108, cache=1, is_edge=0, source_id=20885[20885] 4277:[2018-04-04 14:34:13.115][trace][24880][20933] dispatch cached gop success. count=115, duration=1903 4278:[2018-04-04 14:34:13.115][trace][24880][20933] create consumer, queue_size=30.00, jitter=1 4279:[2018-04-04 14:34:13.125][trace][24880][20933] mw changed sleep 350=>350, max_msgs=128, esbuf=218750, sbuf 87040=>218750, realtime=0 4280:[2018-04-04 14:34:13.125][trace][24880][20933] start play smi=0.00, mw_sleep=350, mw_enabled=1, realtime=0, tcp_nodelay=0 4286:[2018-04-04 14:34:13.640][trace][24880][20933] protocol in.buffer=1000, in.ack=2500000, out.ack=2500000, in.chunk=128, out.chunk=60000 4560:[2018-04-04 14:35:12.705][trace][24880][20933] -> PLA time=59001, msgs=23, okbps=1013,1020,0, ikbps=0,1,0, mw=350 4819:[2018-04-04 14:36:43.279][trace][24880][20933] -> PLA time=149005, msgs=24, okbps=1001,1038,0, ikbps=0,0,0, mw=350 4911:[2018-04-04 14:37:15.272][trace][24880][20933] -> PLA time=181005, msgs=25, okbps=1001,942,0, ikbps=0,0,0, mw=350 4996:[2018-04-04 14:37:43.597][trace][24880][20933] -> PLA time=210006, msgs=23, okbps=1006,1007,0, ikbps=0,0,0, mw=350 5156:[2018-04-04 14:38:31.844][trace][24880][20933] -> PLA time=258007, msgs=23, okbps=1009,1026,0, ikbps=0,0,0, mw=350 5218:[2018-04-04 14:38:51.474][trace][24880][20933] -> PLA time=278007, msgs=24, okbps=1010,1026,0, ikbps=0,0,0, mw=350 5545:[2018-04-04 14:40:30.515][trace][24880][20933] -> PLA time=377009, msgs=23, okbps=1007,1005,1010, ikbps=0,0,0, mw=350 5624:[2018-04-04 14:40:55.955][warn][24880][20933][104] resize iovs 256 => 512, max_msgs=128 5631:[2018-04-04 14:40:56.126][warn][24880][20933][32] client disconnect peer. ret=1004

The session log corresponding to one of the Edge Servers.

2055:[2018-04-04 14:38:13.687][trace][22770][3483] <- EIG time=241004, okbps=0,0,0, ikbps=933,1006,0 2064:[2018-04-04 14:38:23.535][trace][22770][3483] <- EIG time=251004, okbps=0,0,0, ikbps=937,1006,0 2073:[2018-04-04 14:38:33.607][trace][22770][3483] <- EIG time=261004, okbps=0,0,0, ikbps=939,1006,0 2091:[2018-04-04 14:38:43.604][trace][22770][3483] <- EIG time=271004, okbps=0,0,0, ikbps=944,1040,0 2100:[2018-04-04 14:38:53.434][trace][22770][3483] <- EIG time=281004, okbps=0,0,0, ikbps=945,1040,945 2109:[2018-04-04 14:39:03.534][trace][22770][3483] <- EIG time=291004, okbps=0,0,0, ikbps=948,1040,945 2118:[2018-04-04 14:39:13.578][trace][22770][3483] <- EIG time=301004, okbps=0,0,0, ikbps=952,1040,945 2127:[2018-04-04 14:39:23.490][trace][22770][3483] <- EIG time=311004, okbps=0,0,0, ikbps=951,1005,945 2143:[2018-04-04 14:39:33.542][trace][22770][3483] <- EIG time=321004, okbps=0,0,0, ikbps=950,1005,945 2152:[2018-04-04 14:39:43.732][trace][22770][3483] <- EIG time=331004, okbps=0,0,0, ikbps=953,1005,945 2161:[2018-04-04 14:39:53.645][trace][22770][3483] <- EIG time=341005, okbps=0,0,0, ikbps=955,992,945 2162:[2018-04-04 14:39:58.485][warn][22770][3483][11] fast vector incrase 1024=>2048 2171:[2018-04-04 14:40:03.701][trace][22770][3483] <- EIG time=351005, okbps=0,0,0, ikbps=958,992,945 2180:[2018-04-04 14:40:12.435][trace][22770][3483] shrink the cache queue, size=0, removed=1892, max=30.00 2181:[2018-04-04 14:40:13.535][trace][22770][3483] <- EIG time=361005, okbps=0,0,0, ikbps=956,992,945 2191:[2018-04-04 14:40:23.735][trace][22770][3483] <- EIG time=371005, okbps=0,0,0, ikbps=959,992,945 2201:[2018-04-04 14:40:33.586][trace][22770][3483] <- EIG time=381005, okbps=0,0,0, ikbps=959,996,945 2210:[2018-04-04 14:40:43.475][trace][22770][3483] <- EIG time=391005, okbps=0,0,0, ikbps=960,996,945 2219:[2018-04-04 14:40:54.863][warn][22770][3483][62] origin disconnected, retry. ret=1011

TRANS_BY_GPT3

[frankcgit]

When the edge server is disconnected, the message "[32] write flv header failed. ret=1009(Broken pipe)" will appear in the log.

TRANS_BY_GPT3

[winlinvip]

shrink the cache queue, size=0, removed=1892, max=30.00 This indicates that the queue is very long, and the client's network may have issues, causing accumulation.

If the network is normal, then it means that the player is not reading the data. You need to investigate why the player is not receiving the data.

TRANS_BY_GPT3