search

Exide / top-down-arena-shooter

MIT License
0 stars 0 forks source link

Intermittent issues with the main server loop #48

Closed Exide closed 5 years ago

Exide commented 6 years ago

Every once in a while i get this intermittent error that stops the server.

/usr/bin/node[1]: ../src/env-inl.h:188:node::Environment::AsyncHooks::DefaultTriggerAsyncIdScope::DefaultTriggerAsyncIdScope(node::Environment*, double): Assertion `(default_trigger_async_id) >= (0)' failed.
 1: node::Abort() [/usr/bin/node]
 2: 0x8cf02b [/usr/bin/node]
 3: 0x958908 [/usr/bin/node]
 4: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) [/usr/bin/node]
 5: 0xb1532c [/usr/bin/node]
 6: v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/bin/node]
 7: 0x5da133042fd

This was the last message logged prior to the error in this instance. Need to figure out if this is the same each time.

2018-04-13T22:02:52.598Z | NetworkService | ws | broadcast: remove|e814ee79-5b58-4561-8d72-e5692aa66902
Exide commented 6 years ago

I've seen four more of these crashes and each time the last message logged was a remove command broadcasted via websockets. Interesting...

Exide commented 6 years ago

The server crashed after about 9 hours of runtime, this time with a new error:

2018-04-23T05:37:44.053Z | NetworkService | ws | broadcast: update|fb2349a7-a534-485f-a0a3-f030a7b3e8f9,Ship,2790.163302532698,2006.056840598672,320.3999999999984,50,50|f8d62837-9440-41c6-9632-cca04e58cc1d,Ship,689.1081779653063,1533.4077753870067,168.3000000000008,50,50|ed456a4e-d18e-4744-b633-ab77aae08151,Ship,845.9602219029114,1455.0999396169104,45.90000000000001,50,50
RangeError [ERR_INVALID_ASYNC_ID]: Invalid asyncId value: -2147483641
    at validateAsyncId (internal/async_hooks.js:116:16)
    at emitBeforeScript (internal/async_hooks.js:318:3)
    at process._tickCallback (internal/process/next_tick.js:97:9)
Exide commented 6 years ago

Looks like it ran for ~9 hours this time before crashing with the original error again :(

2018-04-23T14:56:28.377Z | index | heartbeat
/usr/bin/node[1]: ../src/env-inl.h:188:node::Environment::AsyncHooks::DefaultTriggerAsyncIdScope::DefaultTriggerAsyncIdScope(node::Environment*, double): Assertion `(default_trigger_async_id) >= (0)' failed.
 1: node::Abort() [/usr/bin/node]
 2: 0x8cf02b [/usr/bin/node]
 3: 0x958908 [/usr/bin/node]
 4: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) [/usr/bin/node]
 5: 0xb1532c [/usr/bin/node]
 6: v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/bin/node]
 7: 0x574051842fd
Exide commented 6 years ago

Based on the symptoms, some reading on the internals of Node.js, and the fact that it's been running for a week without issues since the last change, I believe I was starving the main thread.

Now that we run at 65-70% cpu utilization at idle (on the test machine) we haven't had the issue in 7 days.

I plan to leave this open for a little bit longer in case it crops back up.

Exide commented 5 years ago

The server has been running for months without issue. Closing this.