search

ant-media / Ant-Media-Server

Ant Media Server is a live streaming engine software that provides adaptive, ultra low latency streaming by using WebRTC technology with ~0.5 seconds latency. Ant Media Server is auto-scalable and it can run on-premise or on-cloud.
https://antmedia.io
Other
4.3k stars 636 forks source link

Mongo socket exception causes crash #6733

Open lastpeony opened 1 month ago

lastpeony commented 1 month ago

2.11.3 EE reproduce scenario is not clear. occured while user trying to login through web panel after switching mongo. bunch of mongo cluster socket exception are printed in logs

in the end server crash occured. Might be related to insufficient server resources of mongodb. user was using shared mongodb atlas.

AMS should not crash in any case.

2024-10-23 03:01:30,338 [cluster-ClusterId{value='6705caf6953e005638069828', description='null'}-cluster0-shard-00-01.pd5nw.mongodb.net:27017] INFO  org.mongodb.driver.cluster - Exception in monitor thread while connecting to server cluster0-shard-00-01.pd5nw.mongodb.net:27017
com.mongodb.MongoNodeIsRecoveringException: Command failed with error 91 (ShutdownInProgress): 'The server is in quiesce mode and will shut down' on server cluster0-shard-00-01.pd5nw.mongodb.net:27017. The full response is {"topologyVersion": {"processId": {"$oid": "671844b6262bc058b934959f"}, "counter": 10}, "ok": 0.0, "errmsg": "The server is in quiesce mode and will shut down", "code": 91, "codeName": "ShutdownInProgress", "remainingQuiesceTimeMillis": 59000, "$clusterTime": {"clusterTime": {"$timestamp": {"t": 1729648890, "i": 31}}, "signature": {"hash": {"$binary": {"base64": "xgy9kVM4iE1ey2KUDgaH6yXaBts=", "subType": "00"}}, "keyId": 7389539697178968068}}, "operationTime": {"$timestamp": {"t": 1729648890, "i": 31}}}
    at com.mongodb.internal.connection.ProtocolHelper.createSpecialException(ProtocolHelper.java:251)
    at com.mongodb.internal.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:201)
    at com.mongodb.internal.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:454)
    at com.mongodb.internal.connection.InternalStreamConnection.receive(InternalStreamConnection.java:404)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.lookupServerDescription(DefaultServerMonitor.java:224)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:156)
    at java.base/java.lang.Thread.run(Thread.java:840)
2024-10-23 03:01:30,339 [cluster-ClusterId{value='6705cb0e953e00563806982a', description='null'}-cluster0-shard-00-01.pd5nw.mongodb.net:27017] INFO  org.mongodb.driver.cluster - Exception in monitor thread while connecting to server cluster0-shard-00-01.pd5nw.mongodb.net:27017
com.mongodb.MongoNodeIsRecoveringException: Command failed with error 91 (ShutdownInProgress): 'The server is in quiesce mode and will shut down' on server cluster0-shard-00-01.pd5nw.mongodb.net:27017. The full response is {"topologyVersion": {"processId": {"$oid": "671844b6262bc058b934959f"}, "counter": 10}, "ok": 0.0, "errmsg": "The server is in quiesce mode and will shut down", "code": 91, "codeName": "ShutdownInProgress", "remainingQuiesceTimeMillis": 59000, "$clusterTime": {"clusterTime": {"$timestamp": {"t": 1729648890, "i": 31}}, "signature": {"hash": {"$binary": {"base64": "xgy9kVM4iE1ey2KUDgaH6yXaBts=", "subType": "00"}}, "keyId": 7389539697178968068}}, "operationTime": {"$timestamp": {"t": 1729648890, "i": 31}}}
    at com.mongodb.internal.connection.ProtocolHelper.createSpecialException(ProtocolHelper.java:251)
    at com.mongodb.internal.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:201)
    at com.mongodb.internal.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:454)
    at com.mongodb.internal.connection.InternalStreamConnection.receive(InternalStreamConnection.java:404)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.lookupServerDescription(DefaultServerMonitor.java:224)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:156)
    at java.base/java.lang.Thread.run(Thread.java:840)
2024-10-23 03:01:53,084 [cluster-ClusterId{value='6705cb23953e00563806982c', description='null'}-cluster0-shard-00-01.pd5nw.mongodb.net:27017] INFO  org.mongodb.driver.cluster - Exception in monitor thread while connecting to server cluster0-shard-00-01.pd5nw.mongodb.net:27017
com.mongodb.MongoSocketReadException: Prematurely reached end of stream
    at com.mongodb.internal.connection.SocketStream.read(SocketStream.java:177)
    at com.mongodb.internal.connection.SocketStream.read(SocketStream.java:200)
    at com.mongodb.internal.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:739)
    at com.mongodb.internal.connection.InternalStreamConnection.receiveMessageWithAdditionalTimeout(InternalStreamConnection.java:603)
    at com.mongodb.internal.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:451)
    at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:372)
    at com.mongodb.internal.connection.CommandHelper.sendAndReceive(CommandHelper.java:102)
    at com.mongodb.internal.connection.CommandHelper.executeCommand(CommandHelper.java:49)
    at com.mongodb.internal.connection.InternalStreamConnectionInitializer.initializeConnectionDescription(InternalStreamConnectionInitializer.java:130)
    at com.mongodb.internal.connection.InternalStreamConnectionInitializer.startHandshake(InternalStreamConnectionInitializer.java:78)
    at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:213)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.lookupServerDescription(DefaultServerMonitor.java:196)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:156)
    at java.base/java.lang.Thread.run(Thread.java:840)
2024-10-23 03:01:53,088 [cluster-ClusterId{value='6705cb03953e005638069829', description='null'}-cluster0-shard-00-01.pd5nw.mongodb.net:27017] INFO  org.mongodb.driver.cluster - Exception in monitor thread while connecting to server cluster0-shard-00-01.pd5nw.mongodb.net:27017
com.mongodb.MongoSocketReadTimeoutException: Timeout while receiving message
    at com.mongodb.internal.connection.InternalStreamConnection.translateReadException(InternalStreamConnection.java:728)
    at com.mongodb.internal.connection.InternalStreamConnection.receiveMessageWithAdditionalTimeout(InternalStreamConnection.java:606)
    at com.mongodb.internal.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:451)
    at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:372)
    at com.mongodb.internal.connection.CommandHelper.sendAndReceive(CommandHelper.java:102)
    at com.mongodb.internal.connection.CommandHelper.executeCommand(CommandHelper.java:49)
    at com.mongodb.internal.connection.InternalStreamConnectionInitializer.initializeConnectionDescription(InternalStreamConnectionInitializer.java:130)
    at com.mongodb.internal.connection.InternalStreamConnectionInitializer.startHandshake(InternalStreamConnectionInitializer.java:78)
    at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:213)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.lookupServerDescription(DefaultServerMonitor.java:196)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:156)
    at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.net.SocketTimeoutException: Read timed out

ant-media-server.log (59).zip

lexy10 commented 1 month ago

Yes I think it is related to the insufficient server resources, I will migrate to a self hosted EC2 instance and install mongoDb on the server. Hopefully it should work fine.

I was using Atlas MongoDB shared Server with the following spec: STORAGE 512 MB RAM Shared vCPU Shared.

You can kindly suggest the minimum but cost effective spec for EC2 to use for the mongodb server, because AWS is quite expensive.

Thank you.

muratugureminoglu commented 3 weeks ago

Hi @lastpeony

I’m observing these logs when using a shared server on MongoDB Atlas, which, as noted by @lexy10 , appears to be insufficient server resources.

MongoSocketReadTimeoutException: Timeout while receiving message — This indicates that Ant Media Server couldn’t get a response from MongoDB Atlas. High traffic and unstable network connections could be causing this issue.

To address this, I recommend using at least a 2vCore (CPU-Optimized) instance.