search

turms-im / turms

🕊️ The world's most advanced open source instant messaging engine for 100K~10M concurrent users https://turms-im.github.io/docs
Apache License 2.0
1.77k stars 276 forks source link

API to turms on the demo site become extremely slow due to the timeout of the operations with MongoDB #821

Closed JamesChenX closed 2 years ago

JamesChenX commented 3 years ago

It occurs because the bitnami/mongodb-sharded MongoDB router becomes out of service intermittently and we will find out the reasons for it.

JamesChenX commented 3 years ago

After research, I found it may be a bug of mongo-java-driver, calling watch() for changestream will send redundant getMore forever, which makes a lot of slow operations to the router and make it out of service (but why only the router becomes out of service?). I just submitted a ticket to confirm and wait for https://jira.mongodb.org/projects/JAVA/issues/JAVA-4283?filter=allopenissues to be handled right now.

JamesChenX commented 3 years ago

Seems only me and the workers can view the ticket, so I copy it here. image

JamesChenX commented 3 years ago

The same docker-compose.standalone.yml works fine on my local machine and mongos won't be stuck, but mongos in the remote machine always hang, it takes minutes to update one document in one collection while consuming only a few system resources no matter when it's available or stuck.

Just as the following logs show, there is a big gap between {"t":{"$date":"2021-09-05T04:27:37.045+00:00"} and {"t":{"$date":"2021-09-05T04:31:28.524+00:00"}, mongos was just frozen, while other services work fine. I have tried 4.4.8 and 5.0.2, but the bug still exists. I think what we can do is just upgrade the hardware in the future to test it again because we don't want to dive into a bug of MongoDB while users can deploy MongoDB servers by the cloud service provider and they will just provide available servers without suffering such kind of bug, even if this bug happens just call them to fix.

And this kind of bug exists before with different causes: https://jira.mongodb.org/browse/SERVER-53337 https://jira.mongodb.org/browse/SERVER-49001 https://jira.mongodb.org/browse/SERVER-34190

{"t":{"$date":"2021-09-05T04:26:46.398+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn30","msg":"Slow query","attr":{"type":"command","ns":"turms-standalone.groupType","command":{"getMore":9029233272629332758,"collection":"groupType","batchSize":2147483647,"$db":"turms-standalone","$clusterTime":{"clusterTime":{"$timestamp":{"t":1630816005,"i":1}},"signature":{"hash":{"$binary":{"base64":"UEtxMSkCgmATKG/CN3iLBL/IXds=","subType":"0"}},"keyId":7004269001740124171}},"lsid":{"id":{"$uuid":"d78eb551-5cc9-49d8-bb83-e95490a9cf73"}}},"originatingCommand":{"aggregate":"groupType","pipeline":[{"$changeStream":{"fullDocument":"updateLookup"}}],"cursor":{"batchSize":2147483647},"$db":"turms-standalone","$clusterTime":{"clusterTime":{"$timestamp":{"t":1630808497,"i":1}},"signature":{"hash":{"$binary":{"base64":"lvh8xBERqptwK30tdlcOu3Jhe0s=","subType":"0"}},"keyId":7004269001740124171}},"lsid":{"id":{"$uuid":"d78eb551-5cc9-49d8-bb83-e95490a9cf73"}}},"nShards":2,"cursorid":9029233272629332758,"numYields":0,"nreturned":0,"reslen":308,"readConcern":{"level":"local","provenance":"implicitDefault"},"remote":"172.19.0.14:59952","protocol":"op_msg","remoteOpWaitMillis":999,"durationMillis":999}}
{"t":{"$date":"2021-09-05T04:26:46.398+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn32","msg":"Slow query","attr":{"type":"command","ns":"turms-standalone.admin","command":{"getMore":890552081760502487,"collection":"admin","batchSize":2147483647,"$db":"turms-standalone","$clusterTime":{"clusterTime":{"$timestamp":{"t":1630816005,"i":1}},"signature":{"hash":{"$binary":{"base64":"UEtxMSkCgmATKG/CN3iLBL/IXds=","subType":"0"}},"keyId":7004269001740124171}},"lsid":{"id":{"$uuid":"08979159-f7d7-4db9-8e74-80981b2fbcaa"}}},"originatingCommand":{"aggregate":"admin","pipeline":[{"$changeStream":{"fullDocument":"updateLookup"}}],"cursor":{"batchSize":2147483647},"$db":"turms-standalone","$clusterTime":{"clusterTime":{"$timestamp":{"t":1630808496,"i":10}},"signature":{"hash":{"$binary":{"base64":"4VuDdyFTZ0C7WrfMkI4hJYkTIAw=","subType":"0"}},"keyId":7004269001740124171}},"lsid":{"id":{"$uuid":"08979159-f7d7-4db9-8e74-80981b2fbcaa"}}},"nShards":2,"cursorid":890552081760502487,"numYields":0,"nreturned":0,"reslen":304,"readConcern":{"level":"local","provenance":"implicitDefault"},"remote":"172.19.0.14:59958","protocol":"op_msg","remoteOpWaitMillis":1000,"durationMillis":1000}}
{"t":{"$date":"2021-09-05T04:26:46.402+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn41","msg":"Slow query","attr":{"type":"command","ns":"turms-standalone.userPermissionGroup","command":{"getMore":4450117709628191852,"collection":"userPermissionGroup","batchSize":2147483647,"$db":"turms-standalone","$clusterTime":{"clusterTime":{"$timestamp":{"t":1630816005,"i":1}},"signature":{"hash":{"$binary":{"base64":"UEtxMSkCgmATKG/CN3iLBL/IXds=","subType":"0"}},"keyId":7004269001740124171}},"lsid":{"id":{"$uuid":"412f8fa2-438e-47e3-b4be-e88ed0d2f1c6"}}},"originatingCommand":{"aggregate":"userPermissionGroup","pipeline":[{"$changeStream":{"fullDocument":"updateLookup"}}],"cursor":{"batchSize":2147483647},"$db":"turms-standalone","$clusterTime":{"clusterTime":{"$timestamp":{"t":1630808497,"i":2}},"signature":{"hash":{"$binary":{"base64":"lvh8xBERqptwK30tdlcOu3Jhe0s=","subType":"0"}},"keyId":7004269001740124171}},"lsid":{"id":{"$uuid":"412f8fa2-438e-47e3-b4be-e88ed0d2f1c6"}}},"nShards":2,"cursorid":4450117709628191852,"numYields":0,"nreturned":0,"reslen":318,"readConcern":{"level":"local","provenance":"implicitDefault"},"remote":"172.19.0.14:59976","protocol":"op_msg","remoteOpWaitMillis":1000,"durationMillis":1000}}
{"t":{"$date":"2021-09-05T04:26:48.475+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"127.0.0.1:45968","uuid":"b4c16eac-33f3-4838-9a7b-fc52547962d9","connectionId":5333,"connectionCount":1270}}
{"t":{"$date":"2021-09-05T04:26:56.925+00:00"},"s":"I",  "c":"SH_REFR",  "id":4619901, "ctx":"CatalogCache-44","msg":"Refreshed cached collection","attr":{"namespace":"config.system.sessions","lookupSinceVersion":{"0":{"$timestamp":{"t":1,"i":1024}},"1":{"$oid":"61342ac2dd486fb7a26147af"},"2":{"$timestamp":{"t":1630808770,"i":4}}},"newVersion":{"chunkVersion":{"0":{"$timestamp":{"t":1,"i":1024}},"1":{"$oid":"61342ac2dd486fb7a26147af"},"2":{"$timestamp":{"t":1630808770,"i":4}}},"forcedRefreshSequenceNum":103,"epochDisambiguatingSequenceNum":179},"timeInStore":{"chunkVersion":"None","forcedRefreshSequenceNum":102,"epochDisambiguatingSequenceNum":178},"durationMillis":3}}
{"t":{"$date":"2021-09-05T04:27:37.045+00:00"},"s":"I",  "c":"CONNPOOL", "id":22567,   "ctx":"establishCursors cleanup","msg":"Ending idle connection because the pool meets constraints","attr":{"hostAndPort":"mongodb-shard:27017","numOpenConns":8}}
{"t":{"$date":"2021-09-05T04:31:28.524+00:00"},"s":"I",  "c":"CONNPOOL", "id":22567,   "ctx":"conn5309","msg":"Ending idle connection because the pool meets constraints","attr":{"hostAndPort":"mongodb-shard:27017","numOpenConns":7}}
{"t":{"$date":"2021-09-05T04:31:28.530+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn4458","msg":"Slow query","attr":{"type":"command","ns":"turms-standalone.member","command":{"update":"member","ordered":true,"writeConcern":{"w":"majority"},"txnNumber":5,"$db":"turms-standalone","$clusterTime":{"clusterTime":{"$timestamp":{"t":1630816057,"i":1}},"signature":{"hash":{"$binary":{"base64":"9WQ6auz8Map+iqwYzE+Q/BNUGSE=","subType":"0"}},"keyId":7004269001740124171}},"lsid":{"id":{"$uuid":"e4774df3-9f4e-4ecf-ba42-0ff34ce978de"}}},"nShards":1,"nMatched":0,"nModified":0,"numYields":0,"reslen":185,"readConcern":{"level":"local","provenance":"implicitDefault"},"remote":"172.19.0.14:51404","protocol":"op_msg","durationMillis":211485}}
{"t":{"$date":"2021-09-05T04:31:28.531+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn5333","msg":"client metadata","attr":{"remote":"127.0.0.1:45968","client":"conn5333","doc":{"application":{"name":"MongoDB Shell"},"driver":{"name":"MongoDB Internal Client","version":"5.0.2"},"os":{"type":"Linux","name":"PRETTY_NAME=\"Debian GNU/Linux 10 (buster)\"","architecture":"x86_64","version":"Kernel 5.4.0-67-generic"},"mongos":{"host":"3541b56f5063:27017","client":"127.0.0.1:45968","version":"5.0.2"}}}}
{"t":{"$date":"2021-09-05T04:31:28.532+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn5309","msg":"Slow query","attr":{"type":"command","ns":"turms-standalone.member","command":{"update":"member","ordered":true,"writeConcern":{"w":"majority"},"txnNumber":16,"$db":"turms-standalone","$clusterTime":{"clusterTime":{"$timestamp":{"t":1630816053,"i":1}},"signature":{"hash":{"$binary":{"base64":"6Nr2RHIjFR3GIhOqAnKtS/H8UPE=","subType":"0"}},"keyId":7004269001740124171}},"lsid":{"id":{"$uuid":"4e540894-d4ba-42e3-b75c-b22a3f60134d"}}},"nShards":1,"nMatched":1,"nModified":1,"numYields":0,"reslen":185,"readConcern":{"level":"local","provenance":"implicitDefault"},"remote":"172.19.0.14:57700","protocol":"op_msg","durationMillis":231488}}
{"t":{"$date":"2021-09-05T04:31:28.532+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.7:38358","uuid":"e8da6603-28a6-4f4f-854f-ed0cbcfb5e61","connectionId":5334,"connectionCount":1271}}
{"t":{"$date":"2021-09-05T04:31:28.533+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn5294","msg":"Slow query","attr":{"type":"command","ns":"turms-standalone.member","command":{"update":"member","ordered":true,"writeConcern":{"w":"majority"},"txnNumber":21,"$db":"turms-standalone","$clusterTime":{"clusterTime":{"$timestamp":{"t":1630816057,"i":1}},"signature":{"hash":{"$binary":{"base64":"9WQ6auz8Map+iqwYzE+Q/BNUGSE=","subType":"0"}},"keyId":7004269001740124171}},"lsid":{"id":{"$uuid":"905e9710-7398-4081-9257-d434d1eb88f1"}}},"nShards":1,"nMatched":0,"nModified":0,"numYields":0,"reslen":185,"readConcern":{"level":"local","provenance":"implicitDefault"},"remote":"172.19.0.14:57942","protocol":"op_msg","durationMillis":221488}}
{"t":{"$date":"2021-09-05T04:31:28.536+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.7:38360","uuid":"a08f8c98-d7b4-48ab-8f15-c203c9f7e6d4","connectionId":5335,"connectionCount":1272}}
{"t":{"$date":"2021-09-05T04:31:28.537+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.7:38378","uuid":"22395fb3-684d-4eaa-b565-45fec1afddda","connectionId":5336,"connectionCount":1273}}
{"t":{"$date":"2021-09-05T04:31:28.537+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.7:38380","uuid":"aeff1d2b-7c54-4514-8ed5-b7374ea66071","connectionId":5337,"connectionCount":1274}}
{"t":{"$date":"2021-09-05T04:31:28.538+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.7:38384","uuid":"1a948d0a-219d-462c-b966-bda7c3aa18a0","connectionId":5338,"connectionCount":1275}}
{"t":{"$date":"2021-09-05T04:31:28.538+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.7:38386","uuid":"d9cfb7ef-b166-4afa-aad8-e640c082aa54","connectionId":5339,"connectionCount":1276}}
{"t":{"$date":"2021-09-05T04:31:28.539+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"127.0.0.1:46012","uuid":"a98559d7-16e4-47b8-80b0-74e306d4f3fa","connectionId":5340,"connectionCount":1277}}
{"t":{"$date":"2021-09-05T04:31:28.539+00:00"},"s":"I",  "c":"CONNPOOL", "id":22576,   "ctx":"establishCursors cleanup","msg":"Connecting","attr":{"hostAndPort":"mongodb-shard:27017"}}
{"t":{"$date":"2021-09-05T04:31:28.540+00:00"},"s":"I",  "c":"CONNPOOL", "id":22567,   "ctx":"establishCursors cleanup","msg":"Ending idle connection because the pool meets constraints","attr":{"hostAndPort":"mongodb-cfg:27017","numOpenConns":8}}
{"t":{"$date":"2021-09-05T04:31:28.540+00:00"},"s":"I",  "c":"CONNPOOL", "id":22567,   "ctx":"establishCursors cleanup","msg":"Ending idle connection because the pool meets constraints","attr":{"hostAndPort":"mongodb-cfg:27017","numOpenConns":7}}
{"t":{"$date":"2021-09-05T04:31:28.540+00:00"},"s":"I",  "c":"CONNPOOL", "id":22567,   "ctx":"establishCursors cleanup","msg":"Ending idle connection because the pool meets constraints","attr":{"hostAndPort":"mongodb-cfg:27017","numOpenConns":6}}
{"t":{"$date":"2021-09-05T04:31:28.540+00:00"},"s":"I",  "c":"CONNPOOL", "id":22567,   "ctx":"establishCursors cleanup","msg":"Ending idle connection because the pool meets constraints","attr":{"hostAndPort":"mongodb-cfg:27017","numOpenConns":5}}
{"t":{"$date":"2021-09-05T04:31:28.540+00:00"},"s":"I",  "c":"CONNPOOL", "id":22567,   "ctx":"establishCursors cleanup","msg":"Ending idle connection because the pool meets constraints","attr":{"hostAndPort":"mongodb-cfg:27017","numOpenConns":4}}
{"t":{"$date":"2021-09-05T04:31:28.544+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn4279","msg":"Slow query","attr":{"type":"command","ns":"turms-standalone.member","command":{"update":"member","ordered":true,"writeConcern":{"w":"majority"},"txnNumber":4,"$db":"turms-standalone","$clusterTime":{"clusterTime":{"$timestamp":{"t":1630816083,"i":1}},"signature":{"hash":{"$binary":{"base64":"3NxpjldSXBhDcqYjde7eNhqFEGY=","subType":"0"}},"keyId":7004269001740124171}},"lsid":{"id":{"$uuid":"feb3d36f-eb51-4359-87e2-691b22448c33"}}},"nShards":1,"nMatched":0,"nModified":0,"numYields":0,"reslen":185,"readConcern":{"level":"local","provenance":"implicitDefault"},"remote":"172.19.0.14:48806","protocol":"op_msg","durationMillis":181498}}
{"t":{"$date":"2021-09-05T04:31:28.545+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn4378","msg":"Slow query","attr":{"type":"command","ns":"turms-standalone.member","command":{"update":"member","ordered":true,"writeConcern":{"w":"majority"},"txnNumber":6,"$db":"turms-standalone","$clusterTime":{"clusterTime":{"$timestamp":{"t":1630816113,"i":1}},"signature":{"hash":{"$binary":{"base64":"2RG1ITRhCfLOOTQSd2LBcXGGQeA=","subType":"0"}},"keyId":7004269001740124171}},"lsid":{"id":{"$uuid":"929f9c55-ba12-4ac1-972c-b0d8d67128cd"}}},"nShards":1,"nMatched":0,"nModified":0,"numYields":0,"reslen":185,"readConcern":{"level":"local","provenance":"implicitDefault"},"remote":"172.19.0.14:49344","protocol":"op_msg","durationMillis":171498}}
{"t":{"$date":"2021-09-05T04:31:28.545+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn4447","msg":"Slow query","attr":{"type":"command","ns":"turms-standalone.member","command":{"update":"member","ordered":true,"writeConcern":{"w":"majority"},"txnNumber":9,"$db":"turms-standalone","$clusterTime":{"clusterTime":{"$timestamp":{"t":1630816143,"i":1}},"signature":{"hash":{"$binary":{"base64":"LAh+lb9+0bkDJQ/fNYdjQch8ANs=","subType":"0"}},"keyId":7004269001740124171}},"lsid":{"id":{"$uuid":"1c6b9af7-f4af-4089-b3b2-e2fd04d582e2"}}},"nShards":1,"nMatched":0,"nModified":0,"numYields":0,"reslen":185,"readConcern":{"level":"local","provenance":"implicitDefault"},"remote":"172.19.0.14:51382","protocol":"op_msg","durationMillis":131496}}
{"t":{"$date":"2021-09-05T04:31:28.545+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn4786","msg":"Slow query","attr":{"type":"command","ns":"turms-standalone.member","command":{"update":"member","ordered":true,"writeConcern":{"w":"majority"},"txnNumber":23,"$db":"turms-standalone","$clusterTime":{"clusterTime":{"$timestamp":{"t":1630816083,"i":1}},"signature":{"hash":{"$binary":{"base64":"3NxpjldSXBhDcqYjde7eNhqFEGY=","subType":"0"}},"keyId":7004269001740124171}},"lsid":{"id":{"$uuid":"bddb5ced-bee2-4619-a90e-70fabde0c9bc"}}},"nShards":1,"nMatched":0,"nModified":0,"numYields":0,"reslen":185,"readConcern":{"level":"local","provenance":"implicitDefault"},"remote":"172.19.0.14:52932","protocol":"op_msg","durationMillis":191499}}
{"t":{"$date":"2021-09-05T04:31:28.545+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn4453","msg":"Slow query","attr":{"type":"command","ns":"turms-standalone.member","command":{"update":"member","ordered":true,"writeConcern":{"w":"majority"},"txnNumber":5,"$db":"turms-standalone","$clusterTime":{"clusterTime":{"$timestamp":{"t":1630816083,"i":1}},"signature":{"hash":{"$binary":{"base64":"3NxpjldSXBhDcqYjde7eNhqFEGY=","subType":"0"}},"keyId":7004269001740124171}},"lsid":{"id":{"$uuid":"ce0588b0-969a-42db-8d8f-acd8cd143e3e"}}},"nShards":1,"nMatched":0,"nModified":0,"numYields":0,"reslen":185,"readConcern":{"level":"local","provenance":"implicitDefault"},"remote":"172.19.0.14:51396","protocol":"op_msg","durationMillis":201500}}
{"t":{"$date":"2021-09-05T04:31:28.548+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.7:38406","uuid":"20fc86a3-d041-4b98-8d2d-f68a1faa5a86","connectionId":5341,"connectionCount":1278}}
{"t":{"$date":"2021-09-05T04:31:28.548+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.7:38410","uuid":"7ffd336d-1058-4f29-ab45-64a005e14c40","connectionId":5342,"connectionCount":1279}}
{"t":{"$date":"2021-09-05T04:31:28.549+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.7:38412","uuid":"c9312373-376f-4fe6-8fe8-abc90a39f0f4","connectionId":5343,"connectionCount":1280}}
{"t":{"$date":"2021-09-05T04:31:28.549+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.7:38414","uuid":"e9391de0-880c-4f1f-b672-d262b1bf3ecb","connectionId":5344,"connectionCount":1281}}
{"t":{"$date":"2021-09-05T04:31:28.550+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn4286","msg":"Slow query","attr":{"type":"command","ns":"turms-standalone.member","command":{"update":"member","ordered":true,"writeConcern":{"w":"majority"},"txnNumber":9,"$db":"turms-standalone","$clusterTime":{"clusterTime":{"$timestamp":{"t":1630816113,"i":1}},"signature":{"hash":{"$binary":{"base64":"2RG1ITRhCfLOOTQSd2LBcXGGQeA=","subType":"0"}},"keyId":7004269001740124171}},"lsid":{"id":{"$uuid":"5c898333-410a-4e17-bd41-4604396e71aa"}}},"nShards":1,"nMatched":0,"nModified":0,"numYields":0,"reslen":185,"readConcern":{"level":"local","provenance":"implicitDefault"},"remote":"172.19.0.14:48834","protocol":"op_msg","durationMillis":161503}}
{"t":{"$date":"2021-09-05T04:31:28.550+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.7:38416","uuid":"bba3a2af-e434-497f-a099-1a54f1efbb1f","connectionId":5345,"connectionCount":1282}}
{"t":{"$date":"2021-09-05T04:31:28.551+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.7:38418","uuid":"3c2b1df7-c1fe-4f5e-a4e3-4e1c006ad6c9","connectionId":5346,"connectionCount":1283}}
{"t":{"$date":"2021-09-05T04:31:28.551+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"127.0.0.1:46044","uuid":"5209626c-66c4-4c2f-b405-afe47edf8d33","connectionId":5347,"connectionCount":1284}}
{"t":{"$date":"2021-09-05T04:31:28.552+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.7:38436","uuid":"2adcbc33-2085-4cef-89e6-8c542edd9b3c","connectionId":5348,"connectionCount":1285}}
{"t":{"$date":"2021-09-05T04:31:28.552+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.7:38438","uuid":"4b9f3e12-3e8c-4269-b480-35f626552004","connectionId":5349,"connectionCount":1286}}
{"t":{"$date":"2021-09-05T04:31:28.553+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.7:38440","uuid":"d871bd5a-1b75-4bcb-b11e-018b6b09c133","connectionId":5350,"connectionCount":1287}}
{"t":{"$date":"2021-09-05T04:31:28.553+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn4363","msg":"Slow query","attr":{"type":"command","ns":"turms-standalone.member","command":{"update":"member","ordered":true,"writeConcern":{"w":"majority"},"txnNumber":6,"$db":"turms-standalone","$clusterTime":{"clusterTime":{"$timestamp":{"t":1630816113,"i":1}},"signature":{"hash":{"$binary":{"base64":"2RG1ITRhCfLOOTQSd2LBcXGGQeA=","subType":"0"}},"keyId":7004269001740124171}},"lsid":{"id":{"$uuid":"e8fe0c7f-319b-49bf-8486-632273e9102f"}}},"nShards":1,"nMatched":0,"nModified":0,"numYields":0,"reslen":185,"readConcern":{"level":"local","provenance":"implicitDefault"},"remote":"172.19.0.14:49298","protocol":"op_msg","durationMillis":151506}}
{"t":{"$date":"2021-09-05T04:31:28.554+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.7:38442","uuid":"d311c673-bc92-4a78-a418-af2ea24e0c63","connectionId":5351,"connectionCount":1288}}
{"t":{"$date":"2021-09-05T04:31:28.555+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn5338","msg":"client metadata","attr":{"remote":"172.19.0.7:38384","client":"conn5338","doc":{"driver":{"name":"mongo-go-driver","version":"v1.5.3"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.16.6","application":{"name":"mongodb_exporter"},"mongos":{"host":"3541b56f5063:27017","client":"172.19.0.7:38384","version":"5.0.2"}}}}
{"t":{"$date":"2021-09-05T04:31:28.554+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn5334","msg":"client metadata","attr":{"remote":"172.19.0.7:38358","client":"conn5334","doc":{"driver":{"name":"mongo-go-driver","version":"v1.5.3"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.16.6","application":{"name":"mongodb_exporter"},"mongos":{"host":"3541b56f5063:27017","client":"172.19.0.7:38358","version":"5.0.2"}}}}
{"t":{"$date":"2021-09-05T04:31:28.556+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.19.0.7:38444","uuid":"30ec1a1c-5f60-41da-a6f4-8ccd2615c4b6","connectionId":5352,"connectionCount":1289}}
JamesChenX commented 2 years ago

I am not sure whether it's a protection mechanism of MongoDB (like Turms, Turms server rejects to serve if running out of memory). But I can reproduce the bug when running in a low memory environment, and it's the problem of the environment so we can just close this ticket.