search

ant-media / Ant-Media-Server

Ant Media Server is a live streaming engine software that provides adaptive, ultra low latency streaming by using WebRTC technology with ~0.5 seconds latency. Ant Media Server is auto-scalable and it can run on-premise or on-cloud.
https://antmedia.io
Other
4.29k stars 632 forks source link

[SWR @ 0x7fe01c03da40] Context has not been initialized #3186

Open SelimEmre opened 3 years ago

SelimEmre commented 3 years ago

Short description

One of our users (@k0nr4d) reported this issue. Thanks for reporting this issue to us @k0nr4d He getting an error [SWR @ 0x7fe01c03da40] Context has not been initialized thousands of lines.

We are still experimenting with this. Thus far, we have seen one person be fairly consistently able to crash the daemon by broadcasting from a very old MacBook with Chrome 67. Not every time, but often. We have also seen this happen from some phones but also not always. Seems to be related to older apple devices we think. Right now we are playing with resolutions to see if this is maybe causing issue

More detail: https://groups.google.com/g/ant-media-server/c/4Vbq-WAO-8Q/m/F_B3AoExAQAJ

Steps to reproduce

  1. Use Chrome 67 version on the old MacBook.
  2. Publish stream

Expected behavior

It shouldn't give an error.

Actual behavior

Getting errors with thousands of lines.

Logs

thousands of below lines:

[SWR @ 0x7fe01c08d6c0] Context has not been initialized
[SWR @ 0x7fe01c03da40] Context has not been initialized
[SWR @ 0x7fe01c078700] Context has not been initialized
k0nr4d commented 3 years ago

Just to be clear, we don't know how old the macbook was in question but it was single core. This also manages to cause the REST API to timeout. I'm not 100% sure if it's this error specifically but it seems to happen every time that the API locks up.

k0nr4d commented 3 years ago

Another error, maybe related maybe not. We just had a performer who's stream kept freezing and she kept having to reconnect (webrtc ingress)

(rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (stun_port.cc:96): Binding request timed out from 10.106.16.x:39111 (eno2) (stun_port.cc:96): Binding request timed out from 10.106.16.x:55254 (eno2) (stun_port.cc:96): Binding request timed out from 10.106.16.x:48001 (eno2) (stun_port.cc:96): Binding request timed out from 10.106.16.x:45313 (eno2) (stun_port.cc:96): Binding request timed out from 10.106.16.x:47373 (eno2) (stun_port.cc:96): Binding request timed out from 10.106.16.x:42957 (eno2) (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. Apr 21, 2021 2:00:27 PM org.apache.coyote.AbstractProtocol$ConnectionHandler process SEVERE: Error reading request, ignored java.lang.IllegalArgumentException: Reason Phrase cannot exceed 123 UTF-8 encoded bytes: Unexpected error [20,014] reading data from the APR/native socket [139,803,131,797,664] with wrapper [org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper@1f2a9721:139803131797664]. at javax.websocket.CloseReason.(CloseReason.java:70) at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler.upgradeDispatch(WsHttpUpgradeHandler.java:153) at org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:59) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:818) at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2075) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Thread.java:834) Apr 21, 2021 2:01:10 PM org.apache.coyote.AbstractProtocol$ConnectionHandler process SEVERE: Error reading request, ignored java.lang.IllegalArgumentException: Reason Phrase cannot exceed 123 UTF-8 encoded bytes: Unexpected error [20,014] reading data from the APR/native socket [139,803,131,904,160] with wrapper [org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper@2c3f29db:139803131904160]. at javax.websocket.CloseReason.(CloseReason.java:70) at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler.upgradeDispatch(WsHttpUpgradeHandler.java:153) at org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:59) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:818) at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2075) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Thread.java:834) (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (stun_port.cc:96): Binding request timed out from 10.106.16.x:54492 (eno2) (stun_port.cc:96): Binding request timed out from 10.106.16.x:55502 (eno2) (stun_port.cc:96): Binding request timed out from 10.106.16.x:36472 (eno2) (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (stun_port.cc:96): Binding request timed out from 10.106.16.x:33953 (eno2) Apr 21, 2021 2:12:32 PM org.apache.coyote.AbstractProtocol$ConnectionHandler process SEVERE: Error reading request, ignored java.lang.IllegalArgumentException: Reason Phrase cannot exceed 123 UTF-8 encoded bytes: Unexpected error [20,014] reading data from the APR/native socket [139,803,131,855,008] with wrapper [org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper@15dbd426:139803131855008]. at javax.websocket.CloseReason.(CloseReason.java:70) at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler.upgradeDispatch(WsHttpUpgradeHandler.java:153) at org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:59) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:818) at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2075) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Thread.java:834) (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender.

k0nr4d commented 3 years ago

I am trying to think logically, what we are doing differently then everyone else is. We are using the same webrtc adapter, the same everything and the only difference I can think of is that we are passing resolution in mediaConstraints in order to reduce outgoing bandwidth from the user. This is the only thing that the examples don't do that we do. Is it possible that this is what is causing this issue?

k0nr4d commented 3 years ago

Going off the above, we changed our width/height mediaConstraints. We have not had an API hang in 24 hours across any out of 7 servers - something that is effectively unheard of for us.

This seems to have something to do with width and height in mediaConstraints when doing WebRTC ingress and may have something to do with older apple devices (so old iphone, ipad, and older macbooks). It might not be the browser/OS itself but the cameras on them. It doesn't always happen, it's intermittent.

Below is code changes we made, maybe it will point you guys in the right direction. https://pastebin.com/edvmxwSz

So ultimately, the issue might be resolved for us, I will report back further in the next days but this would still need diagnosis and patch, as there is a potential attack vector in here somewhere.

chai2 commented 2 years ago

@k0nr4d - Thanks for the snippet. We are getting into similar issue with latest 2.4.2.1 for only one specific user. He was able to stream from Wifi but on mobile data.

@SelimEmre - Is there a config or change we can do on the server side to fix this?

k0nr4d commented 2 years ago

@k0nr4d - Thanks for the snippet. We are getting into similar issue with latest 2.4.2.1 for only one specific user. He was able to stream from Wifi but on mobile data.

@SelimEmre - Is there a config or change we can do on the server side to fix this?

That sounds more like an issue with mobile ISP blocking UDP. Are you running coturn alongside Antmedia?