tardis-dev / serum-vial

Real-time WebSocket market data API for Serum
Mozilla Public License 2.0
173 stars 60 forks source link

Serum producer worker 31 error occurred #37

Closed timnugent closed 2 years ago

timnugent commented 2 years ago

Getting this crash with version 1.4.10, node/v16.0.0, using an RPCPool endpoint. Was running fine for a few hours before this:

2021-12-21T18:54:28.116Z info: Established new RPC WebSocket connection... {"market":"HGET/USDC"} 2021-12-21T18:54:28.213Z info: Established new RPC WebSocket connection... {"market":"DXL/USDC"} 2021-12-21T18:54:28.228Z info: Established new RPC WebSocket connection... {"market":"MNGO/USDC"} 2021-12-21T18:54:28.262Z info: Established new RPC WebSocket connection... {"market":"KIN/USDC"} 2021-12-21T18:54:28.263Z info: Established new RPC WebSocket connection... {"market":"ALEPH/USDC"} 2021-12-21T18:54:28.469Z info: Established new RPC WebSocket connection... {"market":"MSOL/USDC"} 2021-12-21T18:54:28.536Z error: Serum producer worker 31 error occurred: _subscribeToAccountsNotifications WS closed Error: _subscribeToAccountsNotifications WS closed at AccountsChangeNotifications._subscribeToAccountsNotifications (/Users/tnugent/.nvm/versions/node/v16.0.0/lib/node_modules/serum-vial/dist/rpc_client.js:331:23) at WebSocket.ws.onopen (/Users/tnugent/.nvm/versions/node/v16.0.0/lib/node_modules/serum-vial/dist/rpc_client.js:191:18) at runMicrotasks () at processTicksAndRejections (node:internal/process/task_queues:96:5) Uncaught Exception thrown Error: _subscribeToAccountsNotifications WS closed at AccountsChangeNotifications._subscribeToAccountsNotifications (/Users/tnugent/.nvm/versions/node/v16.0.0/lib/node_modules/serum-vial/dist/rpc_client.js:331:23) at WebSocket.ws.onopen (/Users/tnugent/.nvm/versions/node/v16.0.0/lib/node_modules/serum-vial/dist/rpc_client.js:191:18) at runMicrotasks () at processTicksAndRejections (node:internal/process/task_queues:96:5) FATAL ERROR: v8::HandleScope::CreateHandle() Cannot create a handle without a HandleScope 1: 0x10d988ee5 node::Abort() (.cold.1) [/Users/tnugent/.nvm/versions/node/v16.0.0/bin/node] 2: 0x10c63edc9 node::Abort() [/Users/tnugent/.nvm/versions/node/v16.0.0/bin/node] 3: 0x10c63ef2f node::OnFatalError(char const, char const) [/Users/tnugent/.nvm/versions/node/v16.0.0/bin/node] 4: 0x10c7bb1be v8::Utils::ReportApiFailure(char const, char const) [/Users/tnugent/.nvm/versions/node/v16.0.0/bin/node] 5: 0x10c94f9b2 v8::internal::HandleScope::Extend(v8::internal::Isolate) [/Users/tnugent/.nvm/versions/node/v16.0.0/bin/node] 6: 0x10c9674d8 v8::internal::FactoryBase::NewRawOneByteString(int, v8::internal::AllocationType) [/Users/tnugent/.nvm/versions/node/v16.0.0/bin/node] 7: 0x10c96e10c v8::internal::Factory::NewStringFromOneByte(v8::internal::Vector const&, v8::internal::AllocationType) [/Users/tnugent/.nvm/versions/node/v16.0.0/bin/node] 8: 0x10c7df0ef v8::String::NewFromOneByte(v8::Isolate, unsigned char const, v8::NewStringType, int) [/Users/tnugent/.nvm/versions/node/v16.0.0/bin/node] 9: 0x10c783dc4 node::crypto::TLSWrap::InvokeQueued(int, char const) [/Users/tnugent/.nvm/versions/node/v16.0.0/bin/node] 10: 0x10c783a6f node::crypto::TLSWrap::Destroy() [/Users/tnugent/.nvm/versions/node/v16.0.0/bin/node] 11: 0x10c783902 node::crypto::TLSWrap::~TLSWrap() [/Users/tnugent/.nvm/versions/node/v16.0.0/bin/node] 12: 0x10c783b8e node::crypto::TLSWrap::~TLSWrap() [/Users/tnugent/.nvm/versions/node/v16.0.0/bin/node] 13: 0x10c5e021f node::Environment::RunCleanup() [/Users/tnugent/.nvm/versions/node/v16.0.0/bin/node] 14: 0x10c573ea4 node::FreeEnvironment(node::Environment) [/Users/tnugent/.nvm/versions/node/v16.0.0/bin/node] 15: 0x10c6e3a55 node::worker::Worker::Run() [/Users/tnugent/.nvm/versions/node/v16.0.0/bin/node] 16: 0x10c6e76b2 node::worker::Worker::StartThread(v8::FunctionCallbackInfo const&)::$_3::__invoke(void) [/Users/tnugent/.nvm/versions/node/v16.0.0/bin/node] 17: 0x7fff2052f8fc _pthread_start [/usr/lib/system/libsystem_pthread.dylib] 18: 0x7fff2052b443 thread_start [/usr/lib/system/libsystem_pthread.dylib] Abort trap: 6

thaaddeus commented 2 years ago

Sorry, it should be fixed in v1.4.11.

timnugent commented 2 years ago

Another one with 1.4.11:

2021-12-22T15:36:32.145Z info: Established new RPC WebSocket connection... {"market":"LINK/USDC"} 2021-12-22T15:36:32.335Z info: Established new RPC WebSocket connection... {"market":"SWAG/USDT"} 2021-12-22T15:36:32.838Z warn: Asks orders count do not match {"market":"MER/USDC","currentAsksCount":147,"localAsksCount":148,"slot":112762102} 2021-12-22T15:36:32.838Z warn: Invalid l3diff {"market":"MER/USDC","asksAccountExists":true,"bidsAccountExists":true,"eventQueueAccountExists":false,"slot":112762102,"l3Diff":[]} 2021-12-22T15:36:32.838Z warn: Publishing full l3 snapshot as requested... {"market":"MER/USDC","slot":112762102} 2021-12-22T15:36:32.839Z warn: L2 diff without corresponding L3 diff {"market":"MER/USDC","asksAccountExists":true,"bidsAccountExists":true,"eventQueueAccountExists":false,"slot":112762102,"asksDiff":[["0.249","122158.2"]],"bidsDiff":[]} 2021-12-22T15:36:33.033Z error: Serum producer worker 45 error occurred: Received second update for asks account for slot 112762123, market AKRO/USDT Error: Received second update for asks account for slot 112762123, market AKRO/USDT at AccountsChangeNotifications._update (/Users/tnugent/.nvm/versions/node/v16.0.0/lib/node_modules/serum-vial/dist/rpc_client.js:488:27) at WebSocket.ws.onmessage (/Users/tnugent/.nvm/versions/node/v16.0.0/lib/node_modules/serum-vial/dist/rpc_client.js:219:26) at WebSocket.onMessage (/Users/tnugent/.nvm/versions/node/v16.0.0/lib/node_modules/serum-vial/node_modules/ws/lib/event-target.js:199:18) at WebSocket.emit (node:events:365:28) at Receiver.receiverOnMessage (/Users/tnugent/.nvm/versions/node/v16.0.0/lib/node_modules/serum-vial/node_modules/ws/lib/websocket.js:1098:20) at Receiver.emit (node:events:365:28) at Receiver.dataMessage (/Users/tnugent/.nvm/versions/node/v16.0.0/lib/node_modules/serum-vial/node_modules/ws/lib/receiver.js:528:14) at Receiver.getData (/Users/tnugent/.nvm/versions/node/v16.0.0/lib/node_modules/serum-vial/node_modules/ws/lib/receiver.js:446:17) at Receiver.startLoop (/Users/tnugent/.nvm/versions/node/v16.0.0/lib/node_modules/serum-vial/node_modules/ws/lib/receiver.js:148:22) at Receiver._write (/Users/tnugent/.nvm/versions/node/v16.0.0/lib/node_modules/serum-vial/node_modules/ws/lib/receiver.js:83:10) Uncaught Exception thrown Error: Received second update for asks account for slot 112762123, market AKRO/USDT at AccountsChangeNotifications._update (/Users/tnugent/.nvm/versions/node/v16.0.0/lib/node_modules/serum-vial/dist/rpc_client.js:488:27) at WebSocket.ws.onmessage (/Users/tnugent/.nvm/versions/node/v16.0.0/lib/node_modules/serum-vial/dist/rpc_client.js:219:26) at WebSocket.onMessage (/Users/tnugent/.nvm/versions/node/v16.0.0/lib/node_modules/serum-vial/node_modules/ws/lib/event-target.js:199:18) at WebSocket.emit (node:events:365:28) at Receiver.receiverOnMessage (/Users/tnugent/.nvm/versions/node/v16.0.0/lib/node_modules/serum-vial/node_modules/ws/lib/websocket.js:1098:20) at Receiver.emit (node:events:365:28) at Receiver.dataMessage (/Users/tnugent/.nvm/versions/node/v16.0.0/lib/node_modules/serum-vial/node_modules/ws/lib/receiver.js:528:14) at Receiver.getData (/Users/tnugent/.nvm/versions/node/v16.0.0/lib/node_modules/serum-vial/node_modules/ws/lib/receiver.js:446:17) at Receiver.startLoop (/Users/tnugent/.nvm/versions/node/v16.0.0/lib/node_modules/serum-vial/node_modules/ws/lib/receiver.js:148:22) at Receiver._write (/Users/tnugent/.nvm/versions/node/v16.0.0/lib/node_modules/serum-vial/node_modules/ws/lib/receiver.js:83:10) FATAL ERROR: v8::HandleScope::CreateHandle() Cannot create a handle without a HandleScope 1: 0x103cffee5 node::Abort() (.cold.1) [/Users/tnugent/.nvm/versions/node/v16.0.0/bin/node] 2: 0x1029b5dc9 node::Abort() [/Users/tnugent/.nvm/versions/node/v16.0.0/bin/node] 3: 0x1029b5f2f node::OnFatalError(char const, char const) [/Users/tnugent/.nvm/versions/node/v16.0.0/bin/node] 4: 0x102b321be v8::Utils::ReportApiFailure(char const, char const) [/Users/tnugent/.nvm/versions/node/v16.0.0/bin/node] 5: 0x102cc69b2 v8::internal::HandleScope::Extend(v8::internal::Isolate) [/Users/tnugent/.nvm/versions/node/v16.0.0/bin/node] 6: 0x102cde4d8 v8::internal::FactoryBase::NewRawOneByteString(int, v8::internal::AllocationType) [/Users/tnugent/.nvm/versions/node/v16.0.0/bin/node] 7: 0x102ce510c v8::internal::Factory::NewStringFromOneByte(v8::internal::Vector const&, v8::internal::AllocationType) [/Users/tnugent/.nvm/versions/node/v16.0.0/bin/node] 8: 0x102b560ef v8::String::NewFromOneByte(v8::Isolate, unsigned char const, v8::NewStringType, int) [/Users/tnugent/.nvm/versions/node/v16.0.0/bin/node] 9: 0x102afadc4 node::crypto::TLSWrap::InvokeQueued(int, char const) [/Users/tnugent/.nvm/versions/node/v16.0.0/bin/node] 10: 0x102afaa6f node::crypto::TLSWrap::Destroy() [/Users/tnugent/.nvm/versions/node/v16.0.0/bin/node] 11: 0x102afa902 node::crypto::TLSWrap::~TLSWrap() [/Users/tnugent/.nvm/versions/node/v16.0.0/bin/node] 12: 0x102afab8e node::crypto::TLSWrap::~TLSWrap() [/Users/tnugent/.nvm/versions/node/v16.0.0/bin/node] 13: 0x10295721f node::Environment::RunCleanup() [/Users/tnugent/.nvm/versions/node/v16.0.0/bin/node] 14: 0x1028eaea4 node::FreeEnvironment(node::Environment) [/Users/tnugent/.nvm/versions/node/v16.0.0/bin/node] 15: 0x102a5aa55 node::worker::Worker::Run() [/Users/tnugent/.nvm/versions/node/v16.0.0/bin/node] 16: 0x102a5e6b2 node::worker::Worker::StartThread(v8::FunctionCallbackInfo const&)::$_3::__invoke(void) [/Users/tnugent/.nvm/versions/node/v16.0.0/bin/node] 17: 0x7fff2052f8fc _pthread_start [/usr/lib/system/libsystem_pthread.dylib] 18: 0x7fff2052b443 thread_start [/usr/lib/system/libsystem_pthread.dylib] Abort trap: 6 MacBook-Pro:serum-vial tnugent$ serum-vial --version 1.4.11

thaaddeus commented 2 years ago

This error is handled more gracefully in v1.4.12, but I was running serum-vial for more than 6 months and it never happened to me - any of those error never occurred. It seems like you're connecting to some very unreliable Solana RPC node and that is the cause of those issues.

timnugent commented 2 years ago

Thanks for the update. Will try 1.4.12 with RPCPool and QuickNode. I'm on a VPN so could be related to that

timnugent commented 2 years ago

Yes, does coincide with getting kicked off my VPN - although strangely the mango-bowl connection survives this