search

mmmaxwwwell / space-engineers-dedicated-docker-linux

Space Engineers Dedicated Server running in Docker for Linux
MIT License
175 stars 41 forks source link

Out Of Memory Exception #34

Closed Kaffee81 closed 2 years ago

Kaffee81 commented 2 years ago

Hi,

I'm running your docker image on a fresh server and after about 2 hours of gameplay (vanilla, just a little base), the server keeps crashing. Found this in my logs:

2022-02-13 14:50:18.426 - Thread: 27 -> Exception occurred: Sandbox.MyInitializer+MyOutOfMemoryException: Game is at critically low memory 2022-02-13 14:50:18.427 - Thread: 27 -> Exception source: 2022-02-13 14:50:18.429 - Thread: 27 -> U3lzdGVtLk91dE9mTWVtb3J5RXhjZXB0aW9uOiBBcnJheSBkaW1lbnNpb25zIGV4Y2VlZGVkIHN1cHBvcnRlZCByYW5nZS4NCiAgIGF0IFN5c3RlbS5OZXQuU2VjdXJpdHkuU3NsU3RhdGUuQ2hlY2tUaHJvdyhCb29sZWFuIGF1dGhTdWNjZXNzQ2hlY2ssIEJvb2xlYW4gc2h1dGRvd25DaGVjaykNCiAgIGF0IFN5c3RlbS5OZXQuU2VjdXJpdHkuU3NsU3RhdGUuZ2V0X1NlY3VyZVN0cmVhbSgpDQogICBhdCBTeXN0ZW0uTmV0LlRsc1N0cmVhbS5SZXN1bWVJTyhCdWZmZXJBc3luY1Jlc3VsdCBidWZmZXJSZXN1bHQpDQogICBhdCBTeXN0ZW0uTmV0LlRsc1N0cmVhbS5SZXN1bWVJT1dvcmtlcihPYmplY3QgcmVzdWx0KQ0KICAgYXQgU3lzdGVtLk5ldC5UbHNTdHJlYW0uV2FrZXVwUGVuZGluZ0lPKElBc3luY1Jlc3VsdCBhcikNCiAgIGF0IFN5c3RlbS5OZXQuTGF6eUFzeW5jUmVzdWx0LkNvbXBsZXRlKEludFB0ciB1c2VyVG9rZW4pDQogICBhdCBTeXN0ZW0uTmV0LkxhenlBc3luY1Jlc3VsdC5Qcm90ZWN0ZWRJbnZva2VDYWxsYmFjayhPYmplY3QgcmVzdWx0LCBJbnRQdHIgdXNlclRva2VuKQ0KICAgYXQgU3lzdGVtLk5ldC5TZWN1cml0eS5Tc2xTdGF0ZS5GaW5pc2hIYW5kc2hha2UoRXhjZXB0aW9uIGUsIEFzeW5jUHJvdG9jb2xSZXF1ZXN0IGFzeW5jUmVxdWVzdCkNCiAgIGF0IFN5c3RlbS5OZXQuU2VjdXJpdHkuU3NsU3RhdGUuUGFydGlhbEZyYW1lQ2FsbGJhY2soQXN5bmNQcm90b2NvbFJlcXVlc3QgYXN5bmNSZXF1ZXN0KQ0KICAgYXQgU3lzdGVtLk5ldC5Bc3luY1Byb3RvY29sUmVxdWVzdC5Db21wbGV0ZVJlcXVlc3QoSW50MzIgcmVzdWx0KQ0KICAgYXQgU3lzdGVtLk5ldC5GaXhlZFNpemVSZWFkZXIuQ2hlY2tDb21wbGV0aW9uQmVmb3JlTmV4dFJlYWQoSW50MzIgYnl0ZXMpDQogICBhdCBTeXN0ZW0uTmV0LkZpeGVkU2l6ZVJlYWRlci5SZWFkQ2FsbGJhY2soSUFzeW5jUmVzdWx0IHRyYW5zcG9ydFJlc3VsdCkNCiAgIGF0IFN5c3RlbS5OZXQuTGF6eUFzeW5jUmVzdWx0LkNvbXBsZXRlKEludFB0ciB1c2VyVG9rZW4pDQogICBhdCBTeXN0ZW0uVGhyZWFkaW5nLkV4ZWN1dGlvbkNvbnRleHQuUnVuSW50ZXJuYWwoRXhlY3V0aW9uQ29udGV4dCBleGVjdXRpb25Db250ZXh0LCBDb250ZXh0Q2FsbGJhY2sgY2FsbGJhY2ssIE9iamVjdCBzdGF0ZSwgQm9vbGVhbiBwcmVzZXJ2ZVN5bmNDdHgpDQogICBhdCBTeXN0ZW0uVGhyZWFkaW5nLkV4ZWN1dGlvbkNvbnRleHQuUnVuKEV4ZWN1dGlvbkNvbnRleHQgZXhlY3V0aW9uQ29udGV4dCwgQ29udGV4dENhbGxiYWNrIGNhbGxiYWNrLCBPYmplY3Qgc3RhdGUsIEJvb2xlYW4gcHJlc2VydmVTeW5jQ3R4KQ0KICAgYXQgU3lzdGVtLlRocmVhZGluZy5FeGVjdXRpb25Db250ZXh0LlJ1bihFeGVjdXRpb25Db250ZXh0IGV4ZWN1dGlvbkNvbnRleHQsIENvbnRleHRDYWxsYmFjayBjYWxsYmFjaywgT2JqZWN0IHN0YXRlKQ0KICAgYXQgU3lzdGVtLk5ldC5Db250ZXh0QXdhcmVSZXN1bHQuQ29tcGxldGUoSW50UHRyIHVzZXJUb2tlbikNCiAgIGF0IFN5c3RlbS5OZXQuTGF6eUFzeW5jUmVzdWx0LlByb3RlY3RlZEludm9rZUNhbGxiYWNrKE9iamVjdCByZXN1bHQsIEludFB0ciB1c2VyVG9rZW4pDQogICBhdCBTeXN0ZW0uTmV0LlNvY2tldHMuQmFzZU92ZXJsYXBwZWRBc3luY1Jlc3VsdC5Db21wbGV0aW9uUG9ydENhbGxiYWNrKFVJbnQzMiBlcnJvckNvZGUsIFVJbnQzMiBudW1CeXRlcywgTmF0aXZlT3ZlcmxhcHBlZCogbmF0aXZlT3ZlcmxhcHBlZCkNCiAgIGF0IFN5c3RlbS5UaHJlYWRpbmcuX0lPQ29tcGxldGlvbkNhbGxiYWNrLlBlcmZvcm1JT0NvbXBsZXRpb25DYWxsYmFjayhVSW50MzIgZXJyb3JDb2RlLCBVSW50MzIgbnVtQnl0ZXMsIE5hdGl2ZU92ZXJsYXBwZWQqIHBPVkVSTEFQKQ== 2022-02-13 14:50:18.429 - Thread: 27 -> Showing message 2022-02-13 14:50:20.090 - Thread: 1 -> GC Memory: 4396.962 / 4396.962 MB 2022-02-13 14:50:27.278 - Thread: 27 -> ================================== CRASH INFO ================================== AppVersion: 01_200_027 GameName: Space Engineers IsOutOfMemory: True IsGPU: False IsNative: False IsTask: False IsExperimental: True ProcessRunTime: 692 PCUCount: 11676 IsHang: False GCMemory: 4397 GCMemoryAllocated: 4397 HWAvailableMemory: 0 ProcessPrivateMemory: 6155 AnalyticId: SEDS

How to increase memory allocation? I have 17GB of free memory available when the server crashes. Had this with vanilla and non-experimental, did not find the errorlog, so I tried to switch to experimental, but same crash. Finally found the error above.

Thank you for your effort on this image!

mmmaxwwwell commented 2 years ago

Never seen this before. Not sure what's up.

That super long string of letters and numbers that ends with a == is base64 encoded data. I ran it through a decoder on a whim and this is the result in UTF8:

System.OutOfMemoryException: Array dimensions exceeded supported range.
   at System.Net.Security.SslState.CheckThrow(Boolean authSuccessCheck, Boolean shutdownCheck)
   at System.Net.Security.SslState.get_SecureStream()
   at System.Net.TlsStream.ResumeIO(BufferAsyncResult bufferResult)
   at System.Net.TlsStream.ResumeIOWorker(Object result)
   at System.Net.TlsStream.WakeupPendingIO(IAsyncResult ar)
   at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
   at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
   at System.Net.Security.SslState.FinishHandshake(Exception e, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.PartialFrameCallback(AsyncProtocolRequest asyncRequest)
   at System.Net.AsyncProtocolRequest.CompleteRequest(Int32 result)
   at System.Net.FixedSizeReader.CheckCompletionBeforeNextRead(Int32 bytes)
   at System.Net.FixedSizeReader.ReadCallback(IAsyncResult transportResult)
   at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Net.ContextAwareResult.Complete(IntPtr userToken)
   at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
   at System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)

Interesting, but I still don't have any answers.

mmmaxwwwell commented 2 years ago

Reading the discord, some of the others found out that a keenswh ssl cert is expired. Wonder if this is related? Reading through the decoded error, it does have something to do with network requests not completing and then piling up until it reaches some threshold and then crashes.

If you really wanted to find out what's going on, you could hook up something to capture the http requests, they might give us some insight into what it keeps doing before crashing.

Example here https://askubuntu.com/questions/252179/how-to-inspect-outgoing-http-requests-of-a-single-application

Or, assuming it's the ssl cert causing the error, just set it to reboot every hour and a half (i know it sucks) until they fix the cert. If this is the case, it could very well stop happening in a few days.

Kaffee81 commented 2 years ago

Ah, yes. I changed /etc/hosts and redirected the host: 127.0.0.1 gate195.keenswh.com

since then, the server keeps running.