search

eosrio / hyperion-sequential-reader

MIT License
2 stars 1 forks source link

Reconnect websocket on random connection failures #3

Open guilledk opened 1 year ago

guilledk commented 1 year ago

It's really common for websockets to drop the connection specially after long sessions, we should port over hyperion-history-api's reconnect machinery https://github.com/eosrio/hyperion-history-api/blob/v3.3.7/connections/state-history.ts

guilledk commented 9 months ago

This was already addressed in #4 and #6 but JT just reported this new one:

[READER] 2 block/s | Blocks: 0 (0.0%) | Actions: 0 | Deltas: 0
2023-11-16T14:07:08.625Z [PID:523520] [info] : 310360890 pushed, at 2 blocks/sec
2023-11-16T14:07:08.828Z [PID:523520] [info] : drained 2 operations.
2023-11-16T14:07:08.828Z [PID:523520] [info] : broadcasting 1 blocks...
2023-11-16T14:07:08.828Z [PID:523520] [info] : done.
2023-11-16T14:07:09.326Z [PID:523520] [info] : drained 2 operations.
2023-11-16T14:07:09.326Z [PID:523520] [info] : broadcasting 1 blocks...
2023-11-16T14:07:09.327Z [PID:523520] [info] : done.
2023-11-16T14:07:09.625Z [PID:523520] [info] : 310360892 pushed, at 2 blocks/sec
2023-11-16T14:07:09.827Z [PID:523520] [info] : drained 2 operations.
2023-11-16T14:07:09.827Z [PID:523520] [info] : broadcasting 1 blocks...
2023-11-16T14:07:09.827Z [PID:523520] [info] : done.
2023-11-16T14:07:10.324Z [PID:523520] [info] : drained 2 operations.
2023-11-16T14:07:10.324Z [PID:523520] [info] : broadcasting 1 blocks...
2023-11-16T14:07:10.324Z [PID:523520] [info] : done.
[READER] 2 block/s | Blocks: 0 (0.0%) | Actions: 0 | Deltas: 0
2023-11-16T14:07:10.625Z [PID:523520] [info] : 310360894 pushed, at 2 blocks/sec
2023-11-16T14:07:10.825Z [PID:523520] [info] : drained 2 operations.
2023-11-16T14:07:10.825Z [PID:523520] [info] : broadcasting 1 blocks...
2023-11-16T14:07:10.825Z [PID:523520] [info] : done.
2023-11-16T14:07:11.122Z [PID:523520] [info] : drained 2 operations.
2023-11-16T14:07:11.122Z [PID:523520] [info] : broadcasting 1 blocks...
2023-11-16T14:07:11.122Z [PID:523520] [info] : done.
2023-11-16T14:07:11.625Z [PID:523520] [info] : 310360896 pushed, at 2 blocks/sec
2023-11-16T14:07:11.791Z [PID:523520] [info] : drained 2 operations.
2023-11-16T14:07:11.791Z [PID:523520] [info] : broadcasting 1 blocks...
2023-11-16T14:07:11.791Z [PID:523520] [info] : done.
[READER] 1.5 block/s | Blocks: 0 (0.0%) | Actions: 0 | Deltas: 0
2023-11-16T14:07:12.625Z [PID:523520] [info] : 310360897 pushed, at 1 blocks/sec
2023-11-16T14:07:13.626Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:14.626Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:15.626Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:16.626Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:17.626Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:18.626Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:19.627Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:20.627Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:21.627Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:22.627Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:23.627Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:24.628Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:25.628Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:26.628Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:27.628Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:28.629Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:29.629Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:30.629Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:31.629Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:32.629Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:33.629Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:34.629Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:35.630Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:36.631Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:37.632Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:38.633Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:39.633Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:40.633Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:41.633Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:42.633Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:43.633Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:44.633Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:45.634Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:46.634Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:47.634Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:48.635Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:49.635Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:50.636Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:51.636Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:52.638Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:53.638Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:54.639Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:55.639Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:56.639Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:57.640Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:58.641Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:07:59.641Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:00.642Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:01.642Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:02.642Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:03.642Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:04.643Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:05.644Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:06.644Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:07.644Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:08.644Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:09.645Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:10.645Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:11.645Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:12.645Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:13.645Z [PID:523520] [info] : stall detected... restarting ship reader.
2023-11-16T14:08:13.645Z [PID:523520] [warn] : restarting SHIP reader!...
2023-11-16T14:08:13.646Z [PID:523520] [warn] : reader stopped, waiting 2 seconds to restart.
2023-11-16T14:08:13.646Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:13.647Z [PID:523520] [warn] : SHIP Reader disconnected.
2023-11-16T14:08:14.645Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:15.645Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
[READER] Restarting...
2023-11-16T14:08:16.645Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:17.645Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:18.645Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
[READER] Connecting to ws://127.0.0.1:19000...
Connecting to ws://127.0.0.1:19000...
2023-11-16T14:08:18.650Z [PID:523520] [info] : SHIP Reader connected.
[READER] Head block: 310361029
[READER] Requesting blocks from 309853135 to 4294967295
2023-11-16T14:08:19.644Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:20.645Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
[READER] Connecting to ws://127.0.0.1:19000...
Connecting to ws://127.0.0.1:19000...
2023-11-16T14:08:20.652Z [PID:523520] [info] : SHIP Reader connected.
DecodingError: Unknown variant idx: 123
    at decodeInner (/home/jbuice/telosevm-translator-mainnet/node_modules/@greymass/eosio/src/serializer/decoder.ts:215:27)
    at decodeBinary (/home/jbuice/telosevm-translator-mainnet/node_modules/@greymass/eosio/src/serializer/decoder.ts:181:16)
    at Object.abiDecode [as decode] (/home/jbuice/telosevm-translator-mainnet/node_modules/@greymass/eosio/src/serializer/decoder.ts:131:20)
    at HyperionSequentialReader.handleShipMessage (file:///home/jbuice/telosevm-translator-mainnet/node_modules/@eosrio/hyperion-sequential-reader/lib/esm/reader.js:184:35)
    at file:///home/jbuice/telosevm-translator-mainnet/node_modules/@eosrio/hyperion-sequential-reader/lib/esm/reader.js:140:18
    at WebSocket.<anonymous> (file:///home/jbuice/telosevm-translator-mainnet/node_modules/@eosrio/hyperion-sequential-reader/lib/esm/state-history.js:27:41)
    at WebSocket.emit (node:events:514:28)
    at Receiver.receiverOnMessage (/home/jbuice/telosevm-translator-mainnet/node_modules/@eosrio/hyperion-sequential-reader/node_modules/ws/lib/websocket.js:1192:20)
    at Receiver.emit (node:events:514:28)
    at Receiver.dataMessage (/home/jbuice/telosevm-translator-mainnet/node_modules/@eosrio/hyperion-sequential-reader/node_modules/ws/lib/receiver.js:560:14) {
  ctx: {
    types: {
      bool: [Object],
      string: [Object],
      asset: [Function],
      symbol: [Function],
      symbol_code: [Function],
      block_timestamp_type: [Function],
      bytes: [Function],
      checksum160: [Function],
      checksum256: [Function],
      checksum512: [Function],
      extended_asset: [Function],
      float128: [Function],
      float32: [Function],
      float64: [Function],
      int128: [Function],
      int16: [Function],
      int32: [Function],
      int64: [Function],
      int8: [Function],
      name: [Function],
      public_key: [Function],
      signature: [Function],
      time_point: [Function],
      time_point_sec: [Function],
      uint128: [Function],
      uint16: [Function],
      uint32: [Function],
      uint64: [Function],
      uint8: [Function],
      varint32: [Function],
      varuint32: [Function]
    },
    strictExtensions: false,
    codingPath: [ [Object] ]
  },
  underlyingError: Error: Unknown variant idx: 123
      at decodeInner (/home/jbuice/telosevm-translator-mainnet/node_modules/@greymass/eosio/src/serializer/decoder.ts:215:27)
      at decodeBinary (/home/jbuice/telosevm-translator-mainnet/node_modules/@greymass/eosio/src/serializer/decoder.ts:181:16)
      at Object.abiDecode [as decode] (/home/jbuice/telosevm-translator-mainnet/node_modules/@greymass/eosio/src/serializer/decoder.ts:131:20)
      at HyperionSequentialReader.handleShipMessage (file:///home/jbuice/telosevm-translator-mainnet/node_modules/@eosrio/hyperion-sequential-reader/lib/esm/reader.js:184:35)
      at file:///home/jbuice/telosevm-translator-mainnet/node_modules/@eosrio/hyperion-sequential-reader/lib/esm/reader.js:140:18
      at WebSocket.<anonymous> (file:///home/jbuice/telosevm-translator-mainnet/node_modules/@eosrio/hyperion-sequential-reader/lib/esm/state-history.js:27:41)
      at WebSocket.emit (node:events:514:28)
      at Receiver.receiverOnMessage (/home/jbuice/telosevm-translator-mainnet/node_modules/@eosrio/hyperion-sequential-reader/node_modules/ws/lib/websocket.js:1192:20)
      at Receiver.emit (node:events:514:28)
      at Receiver.dataMessage (/home/jbuice/telosevm-translator-mainnet/node_modules/@eosrio/hyperion-sequential-reader/node_modules/ws/lib/receiver.js:560:14)
}
2023-11-16T14:08:21.645Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:22.645Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:23.645Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:24.645Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
2023-11-16T14:08:25.646Z [PID:523520] [info] : 310360897 pushed, at 0 blocks/sec
guilledk commented 9 months ago

https://pastebin.com/j9Xu6KzU