search

helium / blockchain-etl

Blockchain follower that follows and stores the Helium blockchain
Apache License 2.0
64 stars 37 forks source link

ETL rage quits updating gateway status #245

Closed densone closed 3 years ago

densone commented 3 years ago

Issue

About once every few hours ETL will crash on updating gateway status which looks like a timeout. I am no expert by far. So here lies the questions.

DB Info

ETL Server.

Error

2021-09-25 22:18:15.562 [info] <0.16953.76>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"11zSqov6EX5Bm55vEbqCydHVsrBTEE2whvLsqxSXbJ2cUXwiCdo">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.4137770983.699662341.183541>,true},5000]}}}
2021-09-25 22:18:16.382 [error] <0.1674.0> gen_server be_db_gateway_status terminated with reason: {timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.4137770983.702545928.225606>,true},5000]}} in gen_server:call/3 line 223
2021-09-25 22:18:16.383 [error] <0.1674.0> CRASH REPORT Process be_db_gateway_status with 15 neighbours exited with reason: {timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.4137770983.702545928.225606>,true},5000]}} in gen_server:call/3 line 223
2021-09-25 22:18:16.384 [error] <0.1251.0> Supervisor be_sup had child be_db_gateway_status started with be_db_gateway_status:start_link() at <0.1674.0> exit with reason {timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.4137770983.702545928.225606>,true},5000]}} in context child_terminated
2021-09-25 22:18:16.384 [warning] <0.1318.0>@blockchain_event:terminate:108 terminating remove_handler
2021-09-25 22:18:16.384 [error] <0.1251.0> Supervisor be_sup had child be_db_gateway_status started with be_db_gateway_status:start_link() at <0.1674.0> exit with reason reached_max_restart_intensity in context shutdown
2021-09-25 22:18:17.476 [info] <0.15787.75>@blockchain_txn:absorb:558 took 32 ms to absorb blockchain_txn_assert_location_v2
2021-09-25 22:18:17.487 [info] <0.15787.75>@blockchain_txn:validate:290 valid: [{blockchain_txn_assert_location_v2,4},{blockchain_txn_payment_v1,3},{blockchain_txn_poc_receipts_v1,217},{blockchain_txn_poc_request_v1,375},{blockchain_txn_validator_heartbeat_v1,8}], invalid: []
2021-09-25 22:18:17.675 [info] <0.15787.75>@blockchain_txn:absorb:558 took 32 ms to absorb blockchain_txn_assert_location_v2
2021-09-25 22:18:17.719 [error] emulator Error in process <0.15234.76> on node 'blockchain_etl@127.0.0.1' with exit value:
{badarg,[{dets,insert,[be_db_gateway,[{<<0,241,112,112,31,94,80,252,248,109,121,75,60,5,83,217,233,186,7,91,13,49,58,59,130,165,66,218,103,235,220,140,10>>},{<<0,110,64,123,146,185,15,226,177,118,161,164,88,208,127,98,191,200,162,172,63,216,191,49,35,122,74,85,149,2,63,189,31>>},{<<0,199,78,37,44,99,57,73,78,3,242,0,137,40,123,124,192,141,200,134,102,126,79,250,39,90,121,209,55,63,173,84,213>>},{<<0,176,156,232,159,196,100,132,72,192,226,194,24,160,212,190,118,12,173,132,0,143,231,91,1,12,214,205,158,255,178,87,195>>},{<<0,255,76,54,134,201,27,200,165,37,164,22,78,5,46,209,132,123,184,101,149,109,179,251,137,94,129,226,128,61,123,144,221>>},{<<0,80,230,204,199,30,104,190,185,105,240,68,117,249,84,5,143,253,204,110,20,155,243,56,139,221,159,188,87,194,106,21,176>>},{<<0,69,134,31,147,138,148,177,143,118,111,223,117,207,223,76,209,254,199,143,6,156,171,111,166,126,41,81,208,115,12,37,55>>},{<<0,55,80,45,49,174,55,130,27,235,201,157,108,101,202,29,171,158,53,100,102,46,56,115,102,102,75,113,220,216,22,174,214>>},{<<0,75,102,179,112,252,206,131,162,231,179,181,241,3,160,238,221,188,180,69,139,180,154,54,159,102,237,139,152,143,122,220,64>>},{<<0,224,68,48,117,152,132,176,54,209,254,199,2,189,250,203,21,142,253,37,177,49,227,197,222,180,82,164,202,144,75,247,40>>},{<<0,49,42,37,251,60,112,49,222,7,119,21,85,93,9,145,77,194,93,69,76,54,187,220,71,211,107,171,86,195,0,120,233>>},{<<0,225,174,137,237,178,223,14,224,131,85,183,183,91,213,153,42,195,163,146,9,18,109,146,188,158,36,198,86,94,134,136,10>>},{<<0,244,102,222,250,149,75,219,186,34,117,179,203,23,101,67,10,129,239,177,190,105,79,100,255,39,140,218,247,106,254,211,177>>},{<<0,79,38,178,148,245,209,88,180,169,103,64,202,170,253,226,60,90,246,215,60,214,52,133,11,206,42,136,109,138,103,120,201>>},{<<0,240,169,212,217,137,61,127,215,64,241,234,158,199,253,225,45,88,120,168,134,171,172,130,78,228,192,12,19,206,186,96,30>>},{<<0,228,200,246,18,7,30,51,183,92,223,87,215,253,123,183,111,223,99,184,57,206,99,116,27,87,175,117,123,173,166,24,5>>},{<<0,1,18,116,56,120,102,46,122,142,31,166,139,146,99,96,231,233,146,221,33,59,175,135,180,129,218,171,27,141,178,168,187>>},{<<0,35,251,237,213,125,63,25,206,72,78,159,215,187,76,128,244,93,16,181,117,124,79,246,147,118,166,220,37,120,35,76,109>>},{<<0,43,80,224,79,142,145,8,137,131,238,242,91,220,236,9,213,236,98,242,235,141,184,253,68,28,154,39,97,96,8,147,9>>},{<<0,195,114,201,181,31,204,129,203,55,111,40,236,174,54,13,31,169,102,46,161,73,252,82,1,101,106,209,29,253,58,232,186>>},{<<0,149,205,45,105,78,34,55,241,111,91,40,151,92,238,197,218,170,209,201,17,11,251,241,135,116,205,53,31,45,136,53,19>>},{<<0,125,174,41,248,71,188,80,26,204,8,208,39,12,222,105,18,202,92,40,242,131,193,169,9,241,47,221,3,191,74,46,153>>},{<<0,29,47,193,115,3,242,2,128,97,117,83,227,101,185,65,102,214,87,16,197,80,72,230,155,231,173,13,204,164,228,29,121>>},{<<0,164,156,139,110,66,201,111,198,109,201,207,216,3,239,94,151,0,244,87,38,17,16,208,84,203,209,208,26,158,218,38,22>>},{<<0,13,184,74,172,189,18,31,42,200,249,135,5,98,250,10,224,245,236,254,177,163,53,144,252,95,192,97,183,101,59,88,242>>},{<<0,138,200,234,202,173,156,109,231,35,221,9,176,244,113,108,172,244,195,80,208,92,45,170,180,238,27,226,209,125,114,170,167>>},{<<0,15,17,253,103,183,245,146,21,102,144,48,178,149,59,235,195,194,1,83,178,49,1,199,189,27,153,80,190,243,205,51,154>>},{<<0,191,126,90,56,27,201,208,138,255,6,178,225,180,84,99,84,228,76,145,31,11,22,123,128,24,46,138,22,237,20,240,94>>},{<<0,3,76,109,34,72,56,154,23,70,31,37,188,154,233,95,138,175,147,205,79,209,210,52,17,38,106,221,197,201,197,216,85>>},{<<0,103,22,10,1,54,98,29,151,17,67,7,230,48,90,205,202,72,203,50,39,222,101,216,50,53,129,226,47,248,110,228,142>>},{<<0,233,247,136,188,37,89,64,103,183,102,40,136,131,38,255,145,90,29,137,112,145,156,164,173,81,194,71,135,44,12,156,6>>},{<<0,49,16,68,111,191,252,159,160,14,23,239,136,174,66,214,177,37,240,32,128,97,248,133,16,179,128,90,82,206,65,8,0>>},...]],...},...]}
2021-09-25 22:18:17.859 [info] <0.15787.75>@blockchain_txn:absorb_and_commit:452 validation took 971 absorb took 231 post took 141 ms for block height 1026137
2021-09-25 22:18:17.974 [info] <0.1320.0>@blockchain_worker:handle_info:617 block sync down: normal
2021-09-25 22:18:17.975 [info] <0.1320.0>@blockchain_worker:start_sync:814 new block sync starting with Pid: <0.16856.76>, Ref: #Ref<0.4137770983.701759489.90171>, Peer: "/p2p/112rMiatDEFtijATwPWzQthnLRmV2aDZoKYNN9TvBh5fUR6mPyH8"
2021-09-25 22:18:19.409 [warning] <0.1318.0>@blockchain_event:terminate:108 terminating remove_handler
2021-09-25 22:18:19.409 [warning] <0.1318.0>@blockchain_event:terminate:108 terminating remove_handler
2021-09-25 22:18:19.411 [warning] <0.1318.0>@blockchain_event:terminate:108 terminating remove_handler
2021-09-25 22:18:19.411 [warning] <0.1318.0>@blockchain_event:terminate:108 terminating remove_handler
2021-09-25 22:18:19.412 [warning] <0.32548.44>@libp2p_yamux_session:handle_info:229 Unhandled message: {'EXIT',<0.365.45>,killed}
2021-09-25 22:18:19.412 [error] <0.1258.0> Supervisor libp2p_swarm_session_sup_blockchain_swarm had child #Ref<0.4137770983.698351617.173399> started with {libp2p_yamux_session,start_client,undefined} at <0.843.64> exit with reason killed in context child_terminated
2021-09-25 22:18:19.412 [error] <0.1945.27> Supervisor {<0.1945.27>,libp2p_simple_sup} had child 1 started with {libp2p_yamux_stream,open_stream,undefined} at <0.1236.27> exit with reason killed in context child_terminated
2021-09-25 22:18:19.412 [error] <0.3180.74> Supervisor {<0.3180.74>,libp2p_simple_sup} had child 5 started with {libp2p_yamux_stream,open_stream,undefined} at <0.1699.76> exit with reason noproc in context shutdown_error
2021-09-25 22:18:19.412 [error] <0.1258.0> Supervisor libp2p_swarm_session_sup_blockchain_swarm had child #Ref<0.4137770983.702283779.27303> started with {libp2p_yamux_session,start_client,undefined} at <0.914.72> exit with reason killed in context child_terminated
2021-09-25 22:18:19.412 [error] <0.3180.74> gen_server <0.3180.74> terminated with reason: killed
2021-09-25 22:18:19.413 [error] <0.32230.63> gen_statem <0.32230.63> in state established terminated with reason: {killed,handle_event_function}
2021-09-25 22:18:19.413 [error] <0.3180.74> CRASH REPORT Process <0.3180.74> with 0 neighbours exited with reason: killed in gen_server:decode_msg/9 line 432
2021-09-25 22:18:19.413 [error] <0.1258.0> Supervisor libp2p_swarm_session_sup_blockchain_swarm had child #Ref<0.4137770983.702808067.51548> started with {libp2p_yamux_session,start_client,undefined} at <0.993.74> exit with reason killed in context child_terminated
2021-09-25 22:18:19.413 [error] <0.32230.63> CRASH REPORT Process <0.32230.63> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-25 22:18:19.413 [error] <0.1258.0> Supervisor libp2p_swarm_session_sup_blockchain_swarm had child #Ref<0.4137770983.698875905.232386> started with {libp2p_yamux_session,start_client,undefined} at <0.389.66> exit with reason killed in context child_terminated
2021-09-25 22:18:19.413 [error] <0.3004.66> Supervisor {<0.3004.66>,libp2p_simple_sup} had child 1 started with {libp2p_yamux_stream,open_stream,undefined} at <0.2315.66> exit with reason noproc in context shutdown_error
2021-09-25 22:18:19.414 [error] <0.1261.0> gen_server libp2p_swarm_group_sup_blockchain_swarm terminated with reason: killed
2021-09-25 22:18:19.414 [error] <0.3004.66> gen_server <0.3004.66> terminated with reason: killed
2021-09-25 22:18:19.414 [error] <0.1261.0> CRASH REPORT Process libp2p_swarm_group_sup_blockchain_swarm with 0 neighbours exited with reason: killed in gen_server:decode_msg/9 line 432
2021-09-25 22:18:19.414 [error] <0.3004.66> CRASH REPORT Process <0.3004.66> with 0 neighbours exited with reason: killed in gen_server:decode_msg/9 line 432
2021-09-25 22:18:19.414 [error] <0.31875.63> gen_server <0.31875.63> terminated with reason: {killed,{gen_statem,call,[<0.32230.63>,{recv,4,120000},infinity]}} in gen:do_call/4 line 177
2021-09-25 22:18:19.414 [error] <0.2187.76> gen_statem <0.2187.76> in state established terminated with reason: {killed,handle_event_function}
2021-09-25 22:18:19.414 [error] <0.31875.63> CRASH REPORT Process <0.31875.63> with 0 neighbours exited with reason: {killed,{gen_statem,call,[<0.32230.63>,{recv,4,120000},infinity]}} in gen:do_call/4 line 177
2021-09-25 22:18:19.414 [error] <0.2187.76> CRASH REPORT Process <0.2187.76> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-25 22:18:19.414 [error] <0.1259.0> gen_server libp2p_swarm_transport_sup_blockchain_swarm terminated with reason: killed
2021-09-25 22:18:19.414 [error] <0.1259.0> CRASH REPORT Process libp2p_swarm_transport_sup_blockchain_swarm with 0 neighbours exited with reason: killed in gen_server:decode_msg/9 line 432
2021-09-25 22:18:19.415 [error] <0.1262.0>@libp2p_config:lookup_pids:106 gen_server libp2p_swarm_server_blockchain_swarm terminated with reason: bad argument in call to ets:match(blockchain_swarm, {{session,'$1'},'$2'}) in libp2p_config:lookup_pids/2 line 106
2021-09-25 22:18:19.415 [error] <0.1262.0>@libp2p_config:lookup_pids:106 CRASH REPORT Process libp2p_swarm_server_blockchain_swarm with 0 neighbours crashed with reason: bad argument in call to ets:match(blockchain_swarm, {{session,'$1'},'$2'}) in libp2p_config:lookup_pids/2 line 106
2021-09-25 22:18:19.415 [error] <0.2255.27> gen_server <0.2255.27> terminated with reason: killed
2021-09-25 22:18:19.415 [error] <0.1286.0> gen_statem <0.1286.0> in state connecting terminated with reason: {killed,state_functions}
2021-09-25 22:18:19.415 [error] <0.1267.0>@libp2p_config:remove_stream_handler:408 gen_server libp2p_group_gossip_server_blockchain_swarm terminated with reason: bad argument in call to ets:delete(blockchain_swarm, {stream_handler,"gossip/1.0.2"}) in libp2p_config:remove_stream_handler/2 line 408
2021-09-25 22:18:19.415 [error] <0.1286.0> CRASH REPORT Process <0.1286.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-25 22:18:19.416 [error] <0.1285.0> gen_statem <0.1285.0> in state connected terminated with reason: {killed,state_functions}
2021-09-25 22:18:19.416 [error] <0.1281.0> gen_statem <0.1281.0> in state connected terminated with reason: {killed,state_functions}
2021-09-25 22:18:19.416 [error] <0.1267.0>@libp2p_config:remove_stream_handler:408 CRASH REPORT Process libp2p_group_gossip_server_blockchain_swarm with 0 neighbours crashed with reason: bad argument in call to ets:delete(blockchain_swarm, {stream_handler,"gossip/1.0.2"}) in libp2p_config:remove_stream_handler/2 line 408
2021-09-25 22:18:19.416 [error] <0.1302.0> gen_statem <0.1302.0> in state connected terminated with reason: {killed,state_functions}
2021-09-25 22:18:19.416 [error] <0.1285.0> CRASH REPORT Process <0.1285.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-25 22:18:19.416 [error] <0.1270.0> gen_statem <0.1270.0> in state connected terminated with reason: {killed,state_functions}
2021-09-25 22:18:19.416 [error] <0.1281.0> CRASH REPORT Process <0.1281.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-25 22:18:19.416 [error] <0.1302.0> CRASH REPORT Process <0.1302.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-25 22:18:19.417 [error] <0.1270.0> CRASH REPORT Process <0.1270.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-25 22:18:19.417 [error] <0.1271.0> gen_statem <0.1271.0> in state connected terminated with reason: {{killed,{gen_server,call,[<0.955.76>,{send,<<10,15,112,101,101,114,95,114,101,115,111,108,117,116,105,111,110,18,48,120,156,1,37,0,218,255,10,35,10,33,0,228,93,131,106,155,215,130,87,161,17,113,159,213,223,35,206,162,128,160,241,254,232,147,217,49,105,113,148,236,33,71,90,62,66,18,139>>,120000},240000]}},state_functions}
2021-09-25 22:18:19.417 [error] <0.1292.0> gen_statem <0.1292.0> in state connected terminated with reason: {killed,state_functions}
2021-09-25 22:18:19.417 [error] <0.1287.0> gen_statem <0.1287.0> in state connected terminated with reason: {killed,state_functions}
2021-09-25 22:18:19.417 [error] <0.1275.0> gen_statem <0.1275.0> in state connected terminated with reason: {killed,state_functions}
2021-09-25 22:18:19.417 [error] <0.1292.0> CRASH REPORT Process <0.1292.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-25 22:18:19.418 [error] <0.1278.0> gen_statem <0.1278.0> in state connected terminated with reason: {killed,state_functions}
2021-09-25 22:18:19.418 [error] <0.1287.0> CRASH REPORT Process <0.1287.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-25 22:18:19.418 [error] <0.1275.0> CRASH REPORT Process <0.1275.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-25 22:18:19.418 [error] <0.1278.0> CRASH REPORT Process <0.1278.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-25 22:18:19.418 [error] <0.2255.27> CRASH REPORT Process <0.2255.27> with 0 neighbours exited with reason: killed in gen_server:decode_msg/9 line 432
2021-09-25 22:18:19.418 [error] <0.1271.0> CRASH REPORT Process <0.1271.0> with 0 neighbours exited with reason: {killed,{gen_server,call,[<0.955.76>,{send,<<10,15,112,101,101,114,95,114,101,115,111,108,117,116,105,111,110,18,48,120,156,1,37,0,218,255,10,35,10,33,0,228,93,131,106,155,215,130,87,161,17,113,159,213,223,35,206,162,128,160,241,254,232,147,217,49,105,113,148,236,33,71,90,62,66,18,139>>,120000},240000]}} in gen_server:call/3 line 223
2021-09-25 22:18:19.418 [error] <0.1294.0> gen_statem <0.1294.0> in state connected terminated with reason: {killed,state_functions}
2021-09-25 22:18:19.418 [error] <0.1273.0> gen_statem <0.1273.0> in state connecting terminated with reason: {killed,state_functions}
2021-09-25 22:18:19.418 [error] <0.1277.0> gen_statem <0.1277.0> in state connected terminated with reason: {killed,state_functions}
madninja commented 3 years ago

Are you using the writer db as a reader too?

madninja commented 3 years ago

Actually it looks like your db pool isn’t big enough there

densone commented 3 years ago

This client has definitely used the writer as the reader eventhough they have a reader too. That has stopped and the issue still arises.

Here's an updated more detailed error:

2021-09-26 22:25:38.922 [info] <0.4550.32>@be_db_follower:maybe_log_duration:121 be_db_stats took 690 ms
2021-09-26 22:25:52.233 [info] <0.26692.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"11Hka2Afr4oHLBjEhxFmxfi3QSbozu3EsSecFcCpQERZVYkgTRX">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3347578882.30427>,true},5000]}}}
2021-09-26 22:25:52.233 [info] <0.27670.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"11AJiE9QXeCsYidmYCmSjBANBcCqc349WiGTMG3aefApFBdkGQD">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3347578882.30429>,true},5000]}}}
2021-09-26 22:25:52.233 [info] <0.25693.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"11w5Nrd2wKVzq5926YaYfCQcjBLjV7tdtD1T1jY6seQbbSGqWYs">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3348627464.11781>,true},5000]}}}
2021-09-26 22:25:52.233 [info] <0.27352.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"11DU7j1o4tuiaeukkvSa2zAWs3cQAhAST5ZS7W7s5qQwe6uayPc">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3348627464.11783>,true},5000]}}}
2021-09-26 22:25:52.234 [info] <0.27625.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"112EzZu9oV47Me4sA7rTaDBnHi7obBuJ2cd31PdmcyhLW27xY3kN">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3347578882.30443>,true},5000]}}}
2021-09-26 22:25:52.235 [info] <0.27561.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"11q3e7PMPJoHQWKuUCxKkudm8YAGLvAMCfRgMnHRtRza7SP6X5M">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3347054597.60111>,true},5000]}}}
2021-09-26 22:25:52.235 [info] <0.27265.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"11Hyhv2c1SYAWH233VFehotKiezgCi5tzEk5Vx1RzGadFY7Ccf5">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3348103174.144121>,true},5000]}}}
2021-09-26 22:25:52.235 [info] <0.26450.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"113y5RvzfC2ME3XG7vJihC44Hn3Fi9oDSzB7haSeZx5qQkxagzQ">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3347054597.60113>,true},5000]}}}
2021-09-26 22:25:52.235 [info] <0.26203.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"11SLAvBuLezMm87tnKok3dLoPaP2KmeMYD7DFZRhSn1cWziBdnQ">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3348103174.144124>,true},5000]}}}
2021-09-26 22:25:52.236 [info] <0.27644.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"112QS1vutqez9NkKe8gMXrTdBsJCuWYEGbBNgmJBjYGxwFXAVkTB">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3348103174.144126>,true},5000]}}}
2021-09-26 22:25:52.236 [info] <0.26918.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"112ph3NkAPsACFp6urd7vPqfEUafJc4HMYLUHfzYQjEHf8NDpz4P">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3348889607.636>,true},5000]}}}
2021-09-26 22:25:52.236 [info] <0.27600.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"112e9vnnVTHnSTiKiEpj3yZZQxS9XkH9ufSUPz5GdYDVAyDxqotR">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3347578882.30466>,true},5000]}}}
2021-09-26 22:25:52.236 [info] <0.27514.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"112CJ9xe9zMQSPvrhwvKJgjPgkEKUA2h3y42UWPN7ZXTBFuknCgf">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3347054597.60115>,true},5000]}}}
2021-09-26 22:25:52.236 [info] <0.27554.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"11yfmnLpzZNFFbPxsag8jhn8qfJ9VU7qWw8NnGkJu6h9mYya2F4">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3347054597.60117>,true},5000]}}}
2021-09-26 22:25:52.237 [info] <0.27154.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"11PWi6Hqpe5L6sxvUSn23DVYsL3JxQJF16p3EbfbdaK9mHLwMoT">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3347578882.30472>,true},5000]}}}
2021-09-26 22:25:52.237 [info] <0.27497.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"112NvDy89bTnsfDunwSV6X2iy1FQhfx2vMLfEqM5MKjs7VUMyVSd">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3348627460.148039>,true},5000]}}}
2021-09-26 22:25:52.237 [info] <0.25140.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"118Ri8Zij6yWFbTjqF7VcFN6Q1yAgfVJM2TSYLdD1omD1Y83qr6">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3348889603.114571>,true},5000]}}}
2021-09-26 22:25:52.238 [info] <0.26729.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"112CoofbNG4QXmMpTonzgJg7a5AM8XAZeRAUFnycGbEZtv3rR3Kj">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3348889603.114573>,true},5000]}}}
2021-09-26 22:25:52.238 [info] <0.27648.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"11t4VAhdaW7hx26TmKZjVvuB8ziDGvLnupz9LKyAkAbzEz4dUQu">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3347054597.60121>,true},5000]}}}
2021-09-26 22:25:52.238 [info] <0.27542.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"112HRmgBKb4bJCuhcVEAqAB3sEPBZHfVYkMr2dj8orcjCFeMtrWn">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3348103174.144138>,true},5000]}}}
2021-09-26 22:25:52.238 [info] <0.27635.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"11w4FV4CTefovXYPY7VbTgAoCuihhtPEdoCBopxtadqJ8DkJ19B">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3348627460.148043>,true},5000]}}}
2021-09-26 22:25:52.238 [info] <0.27666.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"112Rg9mzsirrem9AoGENKtch2m1wUbWHKdHQWpaNXoowD3WqjiB">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3348627460.148041>,true},5000]}}}
2021-09-26 22:25:52.238 [info] <0.26784.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"1124kArbjAPKQYU1u95UP2sDqfLTwEzNHHh6y8ZuAHAPkCXJTCBX">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3348627460.148049>,true},5000]}}}
2021-09-26 22:25:52.238 [info] <0.27718.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"117WdeJevXpHsGVprzxpexdB6WcLRyGajX8DhUAMPHZYKz4mn9L">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3348627464.11829>,true},5000]}}}
2021-09-26 22:25:52.238 [info] <0.27584.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"112Abc94S1AzsVj8Qyy6NX7FypdMaSMgqTnLcdS5U8hthb65pd29">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3348627460.148047>,true},5000]}}}
2021-09-26 22:25:52.239 [info] <0.27660.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"11hzFGHNfqZfzRsNNmo2UDwCW4HcBM8KcGEYzmrHR7DCjZsEddX">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3348627464.11834>,true},5000]}}}
2021-09-26 22:25:52.239 [info] <0.27737.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"112NpL4YtjxYicvWWmuqK8Wq1RzwPyRV2Kz215mb6762cUY7A6Xf">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3348627464.11832>,true},5000]}}}
2021-09-26 22:25:52.239 [info] <0.26355.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"112TAN4mez98z3oqHvzJAsQYkSvJv457bbJ41iJKX749Ngxxkj6E">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3347054597.60126>,true},5000]}}}
2021-09-26 22:25:52.239 [info] <0.27153.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"112v62iwyaSeQLjHpo2Z4KNGUvBMasvd4Z2tZpi447qQHDKWS8iA">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3348103169.228667>,true},5000]}}}
2021-09-26 22:25:52.240 [info] <0.26627.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"112Em57LE4ZXhfnUEPRL4x9NDMFNtZtNvoV5XMPNgKG7xbVRFCXT">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3347578882.30476>,true},5000]}}}
2021-09-26 22:25:52.310 [info] <0.26681.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"112FELstc7gQErJJFrKp7Q6VRm23b3s1rRXd38REciWH9Di4KKbd">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3348627460.148057>,true},5000]}}}
2021-09-26 22:25:52.310 [info] <0.26735.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"116Rwj7ab3RNedpTFVVqjjGAdvpkGPYeL83Pf6FALHypLsobTcZ">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3348627460.148060>,true},5000]}}}
2021-09-26 22:25:52.310 [info] <0.26107.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"11NLcDiLYVaemu6g5if7bAr1wbK75UMLTqNbgvLxiHf4QGp8rce">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3348889603.114575>,true},5000]}}}
2021-09-26 22:25:52.311 [info] <0.27357.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"13GbPBAz1ZS2ZuoPQ7heqA1ZB1zB8EU7bj6nDmTXRGqxrkHHphU">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3347578882.31262>,true},5000]}}}
2021-09-26 22:25:52.315 [info] <0.27678.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"112Q1Vk98T1dqja7aGqHUWNGGcM64VhFBK4M75Zo526fysWpKoKY">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3348103174.144142>,true},5000]}}}
2021-09-26 22:25:52.320 [info] <0.27541.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"112wRji8oMviVovPrDw81CZmiBbQsv8czkbsYQcmrywjn3fp6oeQ">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3347578882.31265>,true},5000]}}}
2021-09-26 22:25:52.320 [info] <0.27640.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"112tKWZw8qU28hffmJG3dUBNgjoG7zqBJUWgHKQBMBMwmrKvQUAw">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3347578882.31268>,true},5000]}}}
2021-09-26 22:25:52.324 [info] <0.27182.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"112uJ7JNyRzX8u3FqYCNqjPhQrrSE1iLjikReG8tXZsjMBCkSnwY">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3348103174.144147>,true},5000]}}}
2021-09-26 22:25:52.324 [info] <0.27543.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"112hyNP9wPfXD5E31xTAtE8AQyhLgqfpDoSiKg12MhL8bT4WYAf6">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3348103174.144149>,true},5000]}}}
2021-09-26 22:25:52.326 [info] <0.27181.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"115cvCfKrKgvsPUiTdZR7hewxSYQkY5doYGCc3SKLFSj1tFcxkB">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3348627460.148065>,true},5000]}}}
2021-09-26 22:25:52.327 [info] <0.27653.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"112ZRTSaW3DEGcnjaui5XKEThC2b39REsWnENf8sgp4MkhwYDi8R">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3348889603.114577>,true},5000]}}}
2021-09-26 22:25:52.331 [info] <0.27571.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"11644H6TRHAJhwtnhNvvE5uPD2nmEHbX7YqkkY2NL3EpXWogJdt">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3347054597.60130>,true},5000]}}}
2021-09-26 22:25:52.331 [info] <0.26182.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"11mYGD9pYXu857YEk3GwevXN2qiC65iyHNymRzxZ5FqdKWXwFoE">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3347054597.60133>,true},5000]}}}
2021-09-26 22:25:52.335 [info] <0.27520.32>@be_db_gateway_status:request_status:234 Failed to update gateway status for <<"112KcoSfJtjFqxMi5wS59UZXRLtChNcQJhBNarHJMHVtu54WT8na">>: {exit,{timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3347578882.31272>,true},5000]}}}
2021-09-26 22:25:52.998 [error] <0.1641.0> gen_server be_db_validator_status terminated with reason: {timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3348627460.148071>,true},5000]}} in gen_server:call/3 line 223
2021-09-26 22:25:52.998 [error] <0.1641.0> CRASH REPORT Process be_db_validator_status with 0 neighbours exited with reason: {timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3348627460.148071>,true},5000]}} in gen_server:call/3 line 223
2021-09-26 22:25:52.998 [error] <0.1251.0> Supervisor be_sup had child be_db_validator_status started with be_db_validator_status:start_link() at <0.1641.0> exit with reason {timeout,{gen_server,call,[db_pool,{checkout,#Ref<0.1891530541.3348627460.148071>,true},5000]}} in context child_terminated
2021-09-26 22:25:52.998 [warning] <0.1318.0>@blockchain_event:terminate:108 terminating remove_handler
2021-09-26 22:25:52.998 [error] <0.1251.0> Supervisor be_sup had child be_db_validator_status started with be_db_validator_status:start_link() at <0.1641.0> exit with reason reached_max_restart_intensity in context shutdown
2021-09-26 22:25:54.105 [info] <0.28272.31>@blockchain_txn:validate:290 valid: [{blockchain_txn_add_gateway_v1,1},{blockchain_txn_assert_location_v2,2},{blockchain_txn_payment_v1,1},{blockchain_txn_payment_v2,5},{blockchain_txn_poc_receipts_v1,179},{blockchain_txn_poc_request_v1,407},{blockchain_txn_validator_heartbeat_v1,6}], invalid: []
2021-09-26 22:25:54.267 [error] emulator Error in process <0.27751.32> on node 'blockchain_etl@127.0.0.1' with exit value:
{badarg,[{dets,insert,[be_db_gateway,[{<<0,194,223,249,99,117,177,228,253,106,26,231,207,253,135,128,242,64,173,155,243,79,95,60,164,10,120,76,169,224,63,225,63>>},{<<0,235,175,213,112,130,250,132,51,52,183,114,142,161,222,6,201,14,9,165,253,120,213,174,197,66,114,75,74,236,4,150,241>>},{<<0,118,192,189,10,42,245,184,206,27,14,247,216,76,21,19,160,66,160,211,122,113,241,82,13,72,53,210,17,95,88,12,162>>},{<<0,3,7,182,125,7,98,113,78,117,141,83,23,12,13,227,25,221,39,238,38,146,129,235,35,132,99,180,215,34,7,28,168>>},{<<0,54,102,90,228,48,96,198,90,226,134,1,161,148,236,192,30,67,241,225,13,168,142,189,237,52,80,119,19,123,40,205,106>>},{<<0,252,245,127,196,110,107,108,201,198,139,204,7,221,53,5,63,172,185,45,149,76,10,19,244,154,58,82,31,203,231,105,113>>},{<<0,209,188,107,166,255,255,104,238,143,91,178,66,165,32,152,113,113,147,97,199,193,187,211,69,11,31,89,188,65,87,247,189>>},{<<0,38,124,20,62,225,70,109,227,106,114,117,165,84,145,200,58,162,53,183,223,44,250,136,164,110,244,166,168,235,235,127,138>>},{<<0,164,183,173,150,232,115,38,160,214,99,240,72,246,10,21,220,178,213,188,23,210,162,205,40,15,166,73,145,95,162,75,3>>},{<<0,182,252,123,13,151,51,236,76,132,47,14,0,93,52,163,220,4,36,31,249,140,198,166,167,245,27,44,220,27,124,202,26>>},{<<0,204,118,8,4,117,72,43,241,195,129,215,17,136,198,184,48,99,223,246,201,143,64,44,214,62,62,195,0,230,71,169,231>>},{<<0,54,89,126,254,110,185,184,110,31,166,107,13,6,47,12,55,135,0,234,190,165,15,28,122,8,210,45,249,172,53,173,240>>},{<<0,80,83,231,212,89,92,46,27,147,14,151,96,107,254,104,189,178,157,18,130,162,171,212,47,101,204,196,165,44,222,176,18>>},{<<0,173,153,16,170,79,59,182,9,6,110,110,148,81,113,52,198,75,75,86,246,88,61,178,252,74,216,41,233,205,47,108,210>>},{<<0,140,252,90,172,210,221,82,132,209,112,159,13,39,31,96,92,210,52,130,10,194,195,248,158,162,155,72,248,172,123,82,133>>},{<<0,19,41,106,244,204,170,37,16,181,30,236,113,232,113,218,172,221,138,212,19,63,172,252,63,171,255,103,19,183,140,236,136>>},{<<0,170,130,57,162,222,147,144,216,196,164,132,151,198,86,50,212,171,128,9,46,154,180,105,7,61,48,126,62,241,179,92,31>>},{<<0,224,219,2,241,129,94,167,70,141,195,207,194,126,214,15,61,159,55,27,130,136,21,181,246,116,232,249,240,122,16,179,81>>},{<<0,128,114,65,12,210,125,163,180,122,213,168,174,161,209,8,233,61,107,70,245,76,115,115,176,110,139,177,139,4,67,10,72>>},{<<0,196,24,32,73,210,168,86,241,238,156,65,141,103,110,164,182,211,118,30,38,158,158,39,135,143,230,165,84,13,115,54,150>>},{<<0,101,44,162,9,157,43,84,31,230,201,142,183,87,63,207,240,212,11,225,41,103,9,16,7,108,239,30,184,170,217,145,25>>},{<<0,206,119,28,223,173,221,69,106,174,208,3,215,0,254,232,18,141,76,252,82,243,146,165,209,93,100,233,94,0,208,156,204>>},{<<0,3,186,112,86,206,79,190,8,97,152,176,152,242,83,82,162,177,30,140,72,177,87,91,218,85,116,143,192,143,143,228,31>>},{<<0,45,68,179,119,25,249,95,101,216,132,193,214,47,255,124,140,23,166,104,97,191,14,143,169,26,243,114,67,148,30,147,6>>},{<<0,33,127,57,67,85,102,196,230,119,224,78,35,71,54,1,55,45,226,197,135,212,179,168,125,233,35,116,163,22,41,195,138>>},{<<0,142,248,24,109,163,143,141,218,14,82,167,78,206,4,73,32,197,239,173,87,198,101,234,9,171,56,104,188,12,136,30,29>>},{<<0,8,151,21,215,70,24,183,99,98,252,242,235,155,54,90,12,80,171,106,92,211,83,27,167,240,222,16,163,250,98,66,14>>},{<<0,119,219,149,88,187,102,67,17,154,91,154,195,72,129,13,57,230,93,49,43,218,42,124,160,157,228,194,147,85,171,96,147>>},{<<0,96,100,24,19,248,52,32,252,34,252,41,222,64,169,168,126,151,86,189,204,248,205,30,168,44,134,171,210,188,247,158,22>>},{<<0,141,98,136,226,38,231,129,222,105,228,38,30,252,219,193,74,186,145,150,40,172,79,244,43,50,8,161,202,215,56,193,190>>},{<<0,87,26,86,173,224,21,24,49,74,222,228,209,91,150,0,55,194,67,101,178,138,20,118,203,154,227,41,221,1,177,141,21>>},{<<0,206,198,212,199,67,175,69,28,212,240,27,226,148,123,137,150,204,44,166,115,178,142,32,177,165,116,22,244,218,53,121,138>>},{<<0,...>>},...]],...},...]}
2021-09-26 22:25:54.513 [info] <0.28272.31>@blockchain_txn:absorb_and_commit:452 validation took 931 absorb took 161 post took 247 ms for block height 1027736
2021-09-26 22:25:55.810 [info] <0.28272.31>@blockchain_txn:validate:290 valid: [{blockchain_txn_add_gateway_v1,1},{blockchain_txn_assert_location_v2,4},{blockchain_txn_payment_v2,1},{blockchain_txn_poc_receipts_v1,226},{blockchain_txn_poc_request_v1,494},{blockchain_txn_validator_heartbeat_v1,9}], invalid: []
2021-09-26 22:25:55.998 [error] emulator Error in process <0.27673.32> on node 'blockchain_etl@127.0.0.1' with exit value:
{badarg,[{dets,insert,[be_db_gateway,[{<<0,218,192,71,60,4,76,143,22,174,136,170,198,103,80,159,134,107,182,201,236,231,36,49,135,152,23,99,143,119,205,205,76>>},{<<0,173,38,194,235,46,112,167,144,247,231,166,61,209,101,183,174,145,109,94,243,116,41,17,100,223,124,91,32,121,131,75,13>>},{<<0,193,121,215,61,149,180,66,21,235,233,222,146,181,43,2,174,220,46,85,32,202,80,62,229,64,96,24,235,190,143,207,121>>},{<<0,196,1,242,214,252,209,223,153,254,242,74,33,71,40,0,82,16,26,182,36,204,85,88,177,2,193,215,117,38,126,248,87>>},{<<0,66,252,104,195,253,154,166,78,94,184,182,56,55,67,210,65,10,202,191,120,173,0,7,189,6,211,212,70,101,196,205,31>>},{<<0,188,26,97,214,155,187,91,54,35,189,66,148,225,235,153,99,250,243,253,3,49,132,215,41,82,31,9,95,131,62,251,77>>},{<<0,187,92,80,128,207,164,158,10,248,62,105,37,212,80,251,53,189,186,163,175,247,201,198,251,213,169,35,123,42,250,119,235>>},{<<0,179,88,96,242,61,3,71,158,61,172,243,159,24,138,178,136,115,255,118,75,110,196,156,80,49,230,3,149,69,159,207,79>>},{<<0,64,76,64,181,69,54,190,132,104,254,133,196,72,111,204,111,147,42,149,142,2,168,173,77,11,55,196,67,123,60,246,51>>},{<<0,195,93,164,89,249,210,108,101,2,210,234,154,255,159,64,46,89,128,127,215,193,69,180,133,48,109,204,237,192,220,231,69>>},{<<0,143,13,97,27,186,51,139,8,222,192,177,185,66,48,255,69,86,11,125,163,155,250,214,165,43,60,79,60,78,168,181,170>>},{<<0,210,194,112,56,128,10,203,66,80,73,71,175,191,183,224,143,240,34,71,8,131,157,145,30,191,33,161,27,153,47,45,29>>},{<<0,229,206,58,213,188,167,201,158,200,50,12,103,46,148,89,20,29,197,94,1,60,47,247,133,88,40,31,54,32,71,30,16>>},{<<0,26,171,229,151,100,47,75,195,94,240,80,144,37,21,21,137,71,127,98,82,204,122,162,199,3,25,183,248,237,7,217,224>>},{<<0,191,207,147,41,181,132,89,75,53,249,9,171,205,225,208,111,121,246,134,181,233,45,0,244,106,189,222,163,63,90,106,16>>},{<<0,6,158,200,103,221,168,10,229,101,127,109,5,196,229,100,71,240,141,12,48,16,214,211,14,119,18,43,206,53,85,193,80>>},{<<0,51,87,143,149,80,47,110,7,228,117,72,25,54,162,163,59,252,137,156,229,105,48,73,65,65,76,99,132,122,234,94,150>>},{<<0,245,50,0,95,247,13,153,144,76,25,106,227,155,74,197,243,72,20,209,157,107,38,84,140,247,132,111,180,209,90,38,87>>},{<<0,226,57,188,64,137,19,232,191,29,241,220,220,100,249,122,41,72,160,248,164,131,80,29,219,242,23,125,100,159,212,62,208>>},{<<0,17,119,110,141,61,252,72,63,224,113,104,211,112,90,207,248,143,223,135,206,248,69,11,65,227,18,255,57,101,211,160,52>>},{<<0,131,146,14,127,125,140,23,177,164,115,179,83,144,129,64,214,17,248,47,172,66,67,92,182,108,90,213,158,79,3,148,97>>},{<<0,65,150,116,40,67,227,137,14,175,244,0,193,89,174,63,194,120,213,76,115,178,6,6,171,190,120,147,145,194,26,213,28>>},{<<0,138,239,246,173,236,194,76,163,130,210,64,82,98,55,33,183,18,124,146,226,164,156,50,199,113,89,239,94,76,8,189,127>>},{<<0,117,66,192,39,31,225,151,167,143,234,22,46,143,173,200,98,75,98,22,253,224,66,106,147,97,171,209,145,31,2,239,30>>},{<<0,228,123,249,44,175,110,100,151,237,3,14,89,226,92,231,203,223,73,189,193,166,40,198,115,10,155,141,29,39,36,242,89>>},{<<0,20,11,248,9,140,214,211,173,184,245,198,232,193,102,6,105,235,189,253,168,182,62,71,164,109,76,187,75,139,127,174,167>>},{<<0,195,85,240,12,118,203,21,151,231,252,53,147,179,162,196,188,178,123,184,61,125,153,221,59,0,137,142,172,149,159,175,147>>},{<<0,187,235,10,193,14,87,27,97,89,21,20,127,204,3,200,19,125,93,24,15,137,101,217,64,43,92,109,209,240,24,187,224>>},{<<0,27,61,182,133,41,166,89,244,200,204,217,177,85,196,145,154,244,87,26,5,128,206,27,146,129,181,146,246,202,26,6,212>>},{<<0,219,30,51,54,26,87,252,6,178,185,24,150,0,48,99,103,39,126,150,196,12,224,161,152,89,153,11,159,217,71,147,217>>},{<<0,166,188,129,66,40,215,68,223,159,234,46,224,64,143,207,236,33,246,56,166,98,206,181,60,22,146,81,74,208,160,32,80>>},{<<0,183,22,151,174,27,222,105,144,12,23,140,66,136,232,100,22,248,222,217,90,77,214,187,178,47,34,90,5,142,60,229,178>>},{<<0,94,178,241,233,29,...>>},...]],...},...]}
2021-09-26 22:25:56.202 [info] <0.28272.31>@blockchain_txn:absorb_and_commit:452 validation took 1118 absorb took 187 post took 205 ms for block height 1027737
2021-09-26 22:25:56.318 [info] <0.1320.0>@blockchain_worker:handle_info:617 block sync down: normal
2021-09-26 22:25:56.319 [info] <0.1320.0>@blockchain_worker:start_sync:814 new block sync starting with Pid: <0.27412.32>, Ref: #Ref<0.1891530541.3348103169.232834>, Peer: "/p2p/11SS34W643WJiTrEQhCUFajVXUWVpCLY1jLC6nR19y8EhdkTzbc"
2021-09-26 22:25:56.678 [warning] <0.1318.0>@blockchain_event:terminate:108 terminating remove_handler
2021-09-26 22:25:56.678 [warning] <0.1318.0>@blockchain_event:terminate:108 terminating remove_handler
2021-09-26 22:25:56.680 [warning] <0.1318.0>@blockchain_event:terminate:108 terminating remove_handler
2021-09-26 22:25:56.680 [warning] <0.1318.0>@blockchain_event:terminate:108 terminating remove_handler
2021-09-26 22:25:56.681 [error] <0.1269.0> gen_statem <0.1269.0> in state connected terminated with reason: {killed,state_functions}
2021-09-26 22:25:56.681 [error] <0.1275.0> gen_statem <0.1275.0> in state connected terminated with reason: {killed,state_functions}
2021-09-26 22:25:56.681 [error] <0.1270.0> gen_statem <0.1270.0> in state connected terminated with reason: {killed,state_functions}
2021-09-26 22:25:56.681 [error] <0.1281.0> gen_statem <0.1281.0> in state connected terminated with reason: {killed,state_functions}
2021-09-26 22:25:56.681 [error] <0.1279.0> gen_statem <0.1279.0> in state connected terminated with reason: {killed,state_functions}
2021-09-26 22:25:56.682 [error] <0.1274.0> gen_statem <0.1274.0> in state connected terminated with reason: {killed,state_functions}
2021-09-26 22:25:56.682 [error] <0.1269.0> CRASH REPORT Process <0.1269.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-26 22:25:56.682 [error] <0.1270.0> CRASH REPORT Process <0.1270.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-26 22:25:56.682 [error] <0.1275.0> CRASH REPORT Process <0.1275.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-26 22:25:56.682 [error] <0.1281.0> CRASH REPORT Process <0.1281.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-26 22:25:56.682 [error] <0.1274.0> CRASH REPORT Process <0.1274.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-26 22:25:56.683 [error] <0.1279.0> CRASH REPORT Process <0.1279.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-26 22:25:56.683 [error] <0.1280.0> gen_statem <0.1280.0> in state connected terminated with reason: {killed,state_functions}
2021-09-26 22:25:56.683 [error] <0.1282.0> gen_statem <0.1282.0> in state connected terminated with reason: {killed,state_functions}
2021-09-26 22:25:56.683 [error] <0.1283.0> gen_statem <0.1283.0> in state connected terminated with reason: {killed,state_functions}
2021-09-26 22:25:56.684 [error] <0.1285.0> gen_statem <0.1285.0> in state connected terminated with reason: {killed,state_functions}
2021-09-26 22:25:56.684 [error] <0.1289.0> gen_statem <0.1289.0> in state connected terminated with reason: {killed,state_functions}
2021-09-26 22:25:56.684 [error] <0.1293.0> gen_statem <0.1293.0> in state connected terminated with reason: {killed,state_functions}
2021-09-26 22:25:56.684 [error] <0.1280.0> CRASH REPORT Process <0.1280.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-26 22:25:56.684 [error] <0.1283.0> CRASH REPORT Process <0.1283.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-26 22:25:56.684 [error] <0.1287.0> gen_statem <0.1287.0> in state connected terminated with reason: {killed,state_functions}
2021-09-26 22:25:56.684 [error] <0.1282.0> CRASH REPORT Process <0.1282.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-26 22:25:56.685 [error] <0.1285.0> CRASH REPORT Process <0.1285.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-26 22:25:56.685 [error] <0.1293.0> CRASH REPORT Process <0.1293.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-26 22:25:56.685 [error] <0.1289.0> CRASH REPORT Process <0.1289.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-26 22:25:56.685 [error] <0.1287.0> CRASH REPORT Process <0.1287.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-26 22:25:56.685 [error] <0.1290.0> gen_statem <0.1290.0> in state connected terminated with reason: {killed,state_functions}
2021-09-26 22:25:56.685 [error] <0.1297.0> gen_statem <0.1297.0> in state connected terminated with reason: {killed,state_functions}
2021-09-26 22:25:56.685 [error] <0.1303.0> gen_statem <0.1303.0> in state connected terminated with reason: {killed,state_functions}
2021-09-26 22:25:56.685 [error] <0.1290.0> CRASH REPORT Process <0.1290.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-26 22:25:56.685 [error] <0.1294.0> gen_statem <0.1294.0> in state connected terminated with reason: {killed,state_functions}
2021-09-26 22:25:56.685 [error] <0.1297.0> CRASH REPORT Process <0.1297.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-26 22:25:56.685 [error] <0.1303.0> CRASH REPORT Process <0.1303.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-26 22:25:56.685 [error] <0.1288.0> gen_statem <0.1288.0> in state connected terminated with reason: {killed,state_functions}
2021-09-26 22:25:56.685 [error] <0.1296.0> gen_statem <0.1296.0> in state connected terminated with reason: {killed,state_functions}
2021-09-26 22:25:56.685 [error] <0.1294.0> CRASH REPORT Process <0.1294.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-26 22:25:56.686 [error] <0.1295.0> gen_statem <0.1295.0> in state connected terminated with reason: {killed,state_functions}
2021-09-26 22:25:56.686 [error] <0.1288.0> CRASH REPORT Process <0.1288.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-26 22:25:56.686 [error] <0.1296.0> CRASH REPORT Process <0.1296.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-26 22:25:56.686 [error] <0.1299.0> gen_statem <0.1299.0> in state connected terminated with reason: {killed,state_functions}
2021-09-26 22:25:56.686 [error] <0.1292.0> gen_statem <0.1292.0> in state connecting terminated with reason: {killed,state_functions}
2021-09-26 22:25:56.686 [error] <0.1295.0> CRASH REPORT Process <0.1295.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-26 22:25:56.686 [error] <0.1292.0> CRASH REPORT Process <0.1292.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-26 22:25:56.687 [error] <0.1284.0> gen_statem <0.1284.0> in state connected terminated with reason: {killed,state_functions}
2021-09-26 22:25:56.687 [error] <0.1299.0> CRASH REPORT Process <0.1299.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-26 22:25:56.687 [error] <0.1298.0> gen_statem <0.1298.0> in state connected terminated with reason: {killed,state_functions}
2021-09-26 22:25:56.687 [error] <0.1302.0> gen_statem <0.1302.0> in state connected terminated with reason: {killed,state_functions}
2021-09-26 22:25:56.687 [error] <0.1276.0> gen_statem <0.1276.0> in state connected terminated with reason: {killed,state_functions}
2021-09-26 22:25:56.687 [error] <0.1298.0> CRASH REPORT Process <0.1298.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993
2021-09-26 22:25:56.687 [error] <0.1284.0> CRASH REPORT Process <0.1284.0> with 0 neighbours exited with reason: killed in gen_statem:loop_receive/3 line 993

thanks @madninja

densone commented 3 years ago

I increased the db pool to 100. Not entirely sure if that's helpful? Maybe they are just long running connections and the pool saturates. Maybe we increase the timeout threshold a little for being able to catch a connection?

madninja commented 3 years ago

I’m not sure why you’re running into this and I’m not. Are you sure you have enough file descriptors available?

densone commented 3 years ago

I’m not sure why you’re running into this and I’m not. Are you sure you have enough file descriptors available?

Here's my hypothesis This is a 64GB / 24Thread machine. Albeit, it's expensive, but not that powerful. So we are timing out trying to get a connection from poolboy because the requests are taking longer than what they would take to complete on your systems. This crash only occurs during update gateway status, so the presumption here is that that will use up a bunch of connections really fast, update some records, and move on to the next thing.

Upon changing the db pool to 100 connections it has not crashed since, which it should have crashed several times by now.

One recommendation that I'd like to make is potentially increasing the timeout value instead of the open DB connections?

Vagabond commented 3 years ago

You can also do blocking checkouts from the pool.

densone commented 3 years ago

@Vagabond Ill look at the poolboy docs for that. Would a blocking checkout signal differently to the worker requesting the connection?

After almost 24 hours we have not crashed on this ETL. Fix was increasing DB pool from 50-100 to avoid crashing on gateway status updates.

madninja commented 3 years ago

That sounds about right.. I'd close this and leave it be

densone commented 3 years ago

Closing. Maybe this will come back to life as the hotspots continue to grow?