search

AirenSoft / OvenMediaEngine

OvenMediaEngine (OME) is a Sub-Second Latency Live Streaming Server with Large-Scale and High-Definition. #WebRTC #LLHLS
https://airensoft.com/ome.html
GNU Affero General Public License v3.0
2.6k stars 1.06k forks source link

Edge Server Stops with timeout error - DQHTTPTimer http_connection.cpp:100 has timed out #1629

Closed sourabhgrepsale closed 3 months ago

sourabhgrepsale commented 6 months ago

Describe the bug I have created the cluster with one origin and 3 edge servers, getting timeout on the edge sometimes, when the stream starts

To Reproduce

  1. Start the cluster using ome_docker_launcher.sh. keep it idle for 12 hours
  2. Start stream on origin
  3. One of the edge starts getting DQHTTPTimer timeout

Expected behavior Edge should start pulling the stream and start serving to clients

Logs [2024-05-29 13:30:26.900] I [SPRtcSig-t3333:10] Publisher | publisher.cpp:187 | Try to pull stream from local origin map: [#default#app/spomlive] [2024-05-29 13:30:26.900] I [DQOrchestrator:23] Orchestrator | orchestrator.cpp:68 | There are no streams in the dynamic application for 60 seconds. Delete the application: #default#app [2024-05-29 13:30:57.948] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100 | Client(HTTP/1.1 - <ClientSocket: 0x794eb2427c10, #26, Connected, TCP, Nonblocking, 10.122.0.15:17452>) has timed out [2024-05-29 13:31:02.948] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100 | Client(HTTP/1.1 - <ClientSocket: 0x794eb2427c10, #26, Connected, TCP, Nonblocking, 10.122.0.15:17452>) has timed out [2024-05-29 13:31:07.949] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100 | Client(HTTP/1.1 - <ClientSocket: 0x794eb2427c10, #26, Connected, TCP, Nonblocking, 10.122.0.15:17452>) has timed out [2024-05-29 13:31:12.949] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100 | Client(HTTP/1.1 - <ClientSocket: 0x794eb2427c10, #26, Connected, TCP, Nonblocking, 10.122.0.15:17452>) has timed out [2024-05-29 13:31:17.949] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100 | Client(HTTP/1.1 - <ClientSocket: 0x794eb2427c10, #26, Connected, TCP, Nonblocking, 10.122.0.15:17452>) has timed out [2024-05-29 13:31:22.950] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100 | Client(HTTP/1.1 - <ClientSocket: 0x794eb2434610, #36, Connected, TCP, Nonblocking, 10.122.0.14:13992>) has timed out [2024-05-29 13:31:22.950] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100 | Client(HTTP/1.1 - <ClientSocket: 0x794eb2427210, #32, Connected, TCP, Nonblocking, 10.122.0.47:41640>) has timed out [2024-05-29 13:31:22.950] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100 | Client(HTTP/1.1 - <ClientSocket: 0x794eb2427c10, #26, Connected, TCP, Nonblocking, 10.122.0.15:17452>) has timed out [2024-05-29 13:31:27.950] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100 | Client(HTTP/1.1 - <ClientSocket: 0x794eb2434610, #36, Connected, TCP, Nonblocking, 10.122.0.14:13992>) has timed out [2024-05-29 13:31:27.950] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100 | Client(HTTP/1.1 - <ClientSocket: 0x794eb2432e10, #34, Connected, TCP, Nonblocking, 10.122.0.15:46828>) has timed out [2024-05-29 13:31:27.950] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100 | Client(HTTP/1.1 - <ClientSocket: 0x794eb2432a10, #33, Connected, TCP, Nonblocking, 10.122.0.47:41648>) has timed out

Server (please complete the following information):

Player (please complete the following information): Not related to player

Additional context Happens when a stream is started on the idle cluster

getroot commented 6 months ago

I need a few more explanations to make sure I understand.

  1. Have you tried playing the stream in Edge?
  2. Is this a situation where the playback attempted in (1) fails and only HTTP Timeout occurs?
  3. Dynamic Application (<Name>*</Name>) may be the cause. Instead, could you please check if the issue is reproduced after specifying the app name in <Name>?
sourabhgrepsale commented 5 months ago
  1. PORT 3333 was down, could not connect to the edge. I had 3 edge servers, 2 of them were working fine, I was able to play stream via them, but one of them was down.
  2. Yes
  3. Okay let me try that, this happens once in 4-5 days, it can be hard for me to confirm, if this fixes the issue.
stale[bot] commented 3 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.