search

helium / miner

Miner for the helium blockchain
Apache License 2.0
609 stars 266 forks source link

Pktfwd receiving too much packets may crash the miner #1018

Closed jerryyip closed 2 years ago

jerryyip commented 3 years ago

We found a situation on serval devices: The pktfwd of these devices are receiving too much lorawan packets, almost 1 packet per second. These packets are pushed to the miner, and crash the miner every 30 seconds. Here are the logs from the miner:

2021-08-28 15:38:31.720 160 [info] <0.1565.0>@miner_poc_statem:find_request:884 got block <<94,195,171,214,53,144,174,218,84,226,190,127,101,180,49,54,106,215,20,207,166,171,171,34,159,170,3,19,135,204,245,87>> checking content
2021-08-28 15:38:31.743 160 [info] <0.1539.0>@miner:handle_info:461 non-consensus block 986005
2021-08-28 15:38:31.747 160 [info] <0.3836.0>@blockchain_gossip_handler:add_block:88 got gossipped block 986005
2021-08-28 15:38:31.818 160 [info] <0.1565.0>@miner_poc_statem:find_request:904 request not found in block <<94,195,171,214,53,144,174,218,84,226,190,127,101,180,49,54,106,215,20,207,166,171,171,34,159,170,3,19,135,204,245,87>>
2021-08-28 15:38:31.818 160 [info] <0.1517.0>@blockchain_txn:validate:290 valid: [{blockchain_txn_poc_request_v1,1}], invalid: []
2021-08-28 15:38:51.996 160 [warning] <0.1562.0>@miner_ebus:handle_message:96 Unhandled dbus message "org.freedesktop.DBus.Introspectable.Introspect"
2021-08-28 15:38:56.323 160 [info] <0.3854.0>@blockchain_txn:validate:290 valid: [], invalid: []
2021-08-28 15:38:59.580 160 [info] <0.1565.0>@miner_poc_statem:find_request:884 got block <<41,242,127,234,113,14,241,66,83,157,115,224,133,137,156,246,91,22,158,237,104,164,45,55,92,175,193,65,156,35,8,111>> checking content
2021-08-28 15:38:59.624 160 [info] <0.1539.0>@miner:handle_info:461 non-consensus block 986006
2021-08-28 15:38:59.627 160 [info] <0.3854.0>@blockchain_gossip_handler:add_block:88 got gossipped block 986006
2021-08-28 15:38:59.689 160 [info] <0.1517.0>@blockchain_txn:validate:290 valid: [{blockchain_txn_poc_request_v1,1}], invalid: []
2021-08-28 15:38:59.718 160 [info] <0.1565.0>@miner_poc_statem:find_request:904 request not found in block <<41,242,127,234,113,14,241,66,83,157,115,224,133,137,156,246,91,22,158,237,104,164,45,55,92,175,193,65,156,35,8,111>>
2021-08-28 15:39:24.150 160 [warning] <0.1562.0>@miner_ebus:handle_message:96 Unhandled dbus message "org.freedesktop.DBus.Introspectable.Introspect"
2021-08-28 15:39:34.239 160 [info] <0.1491.0>@blockchain_worker:start_sync:758 new block sync starting with Pid: <0.3864.0>, Ref: #Ref<0.3948368610.302252033.40403>, Peer: "/p2p/116nxAvg7EG2Bj786gpJNCoRZfFxuLRpPhikmcWnF3h6gqfWEAv"
2021-08-28 15:39:49.556 160 [info] <0.3907.0>@blockchain_txn:validate:290 valid: [], invalid: []
2021-08-28 15:39:51.990 160 [info] <0.3907.0>@blockchain_txn_state_channel_close_v1:absorb:348 Closing with conflict false
2021-08-28 15:39:55.006 160 [info] <0.1565.0>@miner_poc_statem:find_request:884 got block <<245,134,245,243,38,5,253,27,147,133,224,132,167,88,47,180,95,222,252,200,229,217,185,167,23,234,139,193,123,170,243,84>> checking content
2021-08-28 15:39:55.035 160 [info] <0.3907.0>@blockchain_gossip_handler:add_block:88 got gossipped block 986007
2021-08-28 15:39:55.081 160 [info] <0.1539.0>@miner:handle_info:461 non-consensus block 986007
2021-08-28 15:39:55.125 160 [info] <0.1517.0>@blockchain_txn:validate:290 valid: [{blockchain_txn_poc_request_v1,1}], invalid: []
2021-08-28 15:39:55.169 160 [info] <0.1565.0>@miner_poc_statem:find_request:904 request not found in block <<245,134,245,243,38,5,253,27,147,133,224,132,167,88,47,180,95,222,252,200,229,217,185,167,23,234,139,193,123,170,243,84>>
2021-08-28 15:39:56.233 160 [warning] <0.1562.0>@miner_ebus:handle_message:96 Unhandled dbus message "org.freedesktop.DBus.Introspectable.Introspect"
2021-08-28 15:40:28.315 160 [warning] <0.1562.0>@miner_ebus:handle_message:96 Unhandled dbus message "org.freedesktop.DBus.Introspectable.Introspect"
2021-08-28 15:40:45.030 160 [error] <0.3895.0>@Undefined:Undefined:Undefined Supervisor {<0.3895.0>,libp2p_simple_sup} had child 1 started with {libp2p_yamux_stream,receive_stream,undefined} at <0.3899.0> exit with reason noproc in context shutdown_error
2021-08-28 15:41:00.388 160 [warning] <0.1562.0>@miner_ebus:handle_message:96 Unhandled dbus message "org.freedesktop.DBus.Introspectable.Introspect"
2021-08-28 15:41:00.693 160 [info] <0.1491.0>@blockchain_worker:start_sync:758 new block sync starting with Pid: <0.3953.0>, Ref: #Ref<0.3948368610.301989890.257699>, Peer: "/p2p/112qFasyie2CphGzjL4Ri1YFEsFq2xenW2sj5JZKeCbxcuBU5q3a"
2021-08-28 15:41:01.828 160 [info] <0.3954.0>@blockchain_txn:validate:290 valid: [], invalid: []
2021-08-28 15:41:10.337 160 [info] <0.1565.0>@miner_poc_statem:find_request:884 got block <<149,166,227,191,200,152,77,239,247,160,204,42,155,90,57,149,127,246,76,91,76,36,42,85,154,28,20,186,33,231,178,96>> checking content
2021-08-28 15:41:10.370 160 [info] <0.1539.0>@miner:handle_info:461 non-consensus block 986008
2021-08-28 15:41:10.388 160 [info] <0.3954.0>@blockchain_gossip_handler:add_block:88 got gossipped block 986008
2021-08-28 15:41:10.431 160 [info] <0.1517.0>@blockchain_txn:validate:290 valid: [{blockchain_txn_poc_request_v1,1}], invalid: []
2021-08-28 15:41:10.481 160 [info] <0.1565.0>@miner_poc_statem:find_request:904 request not found in block <<149,166,227,191,200,152,77,239,247,160,204,42,155,90,57,149,127,246,76,91,76,36,42,85,154,28,20,186,33,231,178,96>>
2021-08-28 15:41:15.444 160 [info] <0.3977.0>@blockchain_txn:validate:290 valid: [], invalid: []
2021-08-28 15:41:22.263 160 [info] <0.1565.0>@miner_poc_statem:find_request:884 got block <<40,179,86,168,44,160,168,99,141,24,43,165,215,97,170,65,194,184,99,110,133,34,187,84,136,60,178,101,93,20,189,83>> checking content
2021-08-28 15:41:22.287 160 [info] <0.1539.0>@miner:handle_info:461 non-consensus block 986009
2021-08-28 15:41:22.290 160 [info] <0.3977.0>@blockchain_gossip_handler:add_block:88 got gossipped block 986009
2021-08-28 15:41:22.355 160 [info] <0.1517.0>@blockchain_txn:validate:290 valid: [{blockchain_txn_poc_request_v1,1}], invalid: []
2021-08-28 15:41:22.377 160 [info] <0.1565.0>@miner_poc_statem:find_request:904 request not found in block <<40,179,86,168,44,160,168,99,141,24,43,165,215,97,170,65,194,184,99,110,133,34,187,84,136,60,178,101,93,20,189,83>>
2021-08-28 15:41:32.457 160 [warning] <0.1562.0>@miner_ebus:handle_message:96 Unhandled dbus message "org.freedesktop.DBus.Introspectable.Introspect"
2021-08-28 15:42:02.978 160 [info] <0.4055.0>@blockchain_txn:validate:290 valid: [], invalid: []
2021-08-28 15:42:04.274 160 [info] <0.4055.0>@blockchain_ledger_v1:maybe_gc_scs:2204 gcing old state_channels...
2021-08-28 15:42:04.534 160 [warning] <0.1562.0>@miner_ebus:handle_message:96 Unhandled dbus message "org.freedesktop.DBus.Introspectable.Introspect"
2021-08-28 15:42:06.108 160 [info] <0.4055.0>@blockchain_ledger_v1:maybe_gc_scs:2204 gcing old state_channels...
2021-08-28 15:42:06.206 160 [info] <0.1565.0>@miner_poc_statem:find_request:884 got block <<237,231,215,92,166,24,51,151,137,180,224,243,27,118,236,150,134,38,112,251,60,60,183,141,7,89,61,221,102,146,251,167>> checking content
2021-08-28 15:42:06.222 160 [info] <0.4055.0>@blockchain_gossip_handler:add_block:88 got gossipped block 986010
2021-08-28 15:42:06.233 160 [info] <0.1539.0>@miner:handle_info:461 non-consensus block 986010
2021-08-28 15:42:06.300 160 [info] <0.1517.0>@blockchain_txn:validate:290 valid: [{blockchain_txn_poc_request_v1,1}], invalid: []
2021-08-28 15:42:06.353 160 [info] <0.1565.0>@miner_poc_statem:find_request:904 request not found in block <<237,231,215,92,166,24,51,151,137,180,224,243,27,118,236,150,134,38,112,251,60,60,183,141,7,89,61,221,102,146,251,167>>
2021-08-28 15:42:26.823 160 [info] <0.1491.0>@blockchain_worker:start_sync:758 new block sync starting with Pid: <0.4095.0>, Ref: #Ref<0.3948368610.301989890.259125>, Peer: "/p2p/11296rBjhccdK1n68fW6kpYQXs62iF78He8HDU4UqscNyK2aFAMr"
2021-08-28 15:42:36.604 160 [warning] <0.1562.0>@miner_ebus:handle_message:96 Unhandled dbus message "org.freedesktop.DBus.Introspectable.Introspect"
2021-08-28 15:42:40.749 160 [info] <0.4124.0>@blockchain_txn:validate:290 valid: [], invalid: []
2021-08-28 15:42:41.619 160 [warning] <0.1562.0>@miner_ebus:handle_message:96 Unhandled dbus message "org.freedesktop.DBus.Introspectable.Introspect"
2021-08-28 15:42:44.269 160 [info] <0.1565.0>@miner_poc_statem:find_request:884 got block <<91,230,235,129,139,71,139,220,250,91,119,244,152,52,124,56,147,125,170,81,193,196,211,228,237,198,215,147,70,69,195,102>> checking content
2021-08-28 15:42:44.288 160 [info] <0.4124.0>@blockchain_gossip_handler:add_block:88 got gossipped block 986011
2021-08-28 15:42:44.342 160 [info] <0.1539.0>@miner:handle_info:461 non-consensus block 986011
2021-08-28 15:42:44.373 160 [info] <0.1517.0>@blockchain_txn:validate:290 valid: [{blockchain_txn_poc_request_v1,1}], invalid: []
2021-08-28 15:42:44.444 160 [info] <0.1565.0>@miner_poc_statem:find_request:904 request not found in block <<91,230,235,129,139,71,139,220,250,91,119,244,152,52,124,56,147,125,170,81,193,196,211,228,237,198,215,147,70,69,195,102>>
2021-08-28 15:43:07.676 160 [warning] <0.1562.0>@miner_ebus:handle_message:96 Unhandled dbus message "org.freedesktop.DBus.Introspectable.Introspect"
2021-08-28 15:43:26.542 160 [info] <0.4158.0>@blockchain_txn:validate:290 valid: [], invalid: []
2021-08-28 15:43:33.860 160 [info] <0.1565.0>@miner_poc_statem:find_request:884 got block <<114,206,35,186,13,50,20,140,138,55,21,216,255,242,14,43,7,119,200,13,207,74,219,150,57,148,197,106,72,18,145,210>> checking content
2021-08-28 15:43:33.895 160 [info] <0.1539.0>@miner:handle_info:461 non-consensus block 986012
2021-08-28 15:43:34.002 160 [info] <0.1565.0>@miner_poc_statem:find_request:904 request not found in block <<114,206,35,186,13,50,20,140,138,55,21,216,255,242,14,43,7,119,200,13,207,74,219,150,57,148,197,106,72,18,145,210>>
2021-08-28 15:43:35.059 160 [warning] <0.1517.0>@blockchain_txn:validate:327 invalid txn blockchain_txn_state_channel_close_v1 : {error,not_found} / type=state_channel_close, state_channel={blockchain_state_channel_v1_pb,
                                         <<39,92,74,38,60,166,13,59,153,125,
                                           174,131,100,4,115,198,192,215,134,
                                           252,181,163,227,137,14,195,176,72,
                                           150,50,199,149>>,
                                         <<0,241,20,68,146,24,117,226,239,116,
                                           53,81,58,29,31,27,15,164,158,50,66,
                                           149,106,36,56,57,18,236,93,79,25,64,
                                           119>>,
                                         250000,29203,
                                         [{blockchain_state_channel_summary_v1_pb,
                                           <<0,163,141,188,100,107,180,12,28,
                                             144,56,147,209,199,141,91,90,168,
                                             155,104,157,56,237,40,5,117,142,
                                             113,102,91,120,124,30>>,
                                           9,15},
                                          {blockchain_state_channel_summary_v1_pb,
                                           <<0,52,84,86,67,134,245,42,239,149,
                                             5,159,251,190,57,67,73,97,192,65,
                                             51,78,96,85,61,199,120,110,62,206,
                                             187,91,220>>,
                                           5,10},
                                          {blockchain_state_channel_summary_v1_pb,
                                           <<0,106,38,58,121,87,107,76,36,78,
                                             212,46,20,123,234,155,27,28,132,
                                             199,142,10,200,9,105,123,145,42,
                                             48,155,172,7,201>>,
                                           10,20},
                                          {blockchain_state_channel_summary_v1_pb,
                                           <<0,20,202,251,157,19,53,127,206,
                                             252,235,249,115,131,102,2,88,157,
                                             11,173,164,57,140,107,141,67,56,
                                             159,98,43,193,172,51>>,
                                           6,12},
                                          {blockchain_state_channel_summary_v1_pb,
                                           <<0,99,61,190,73,88,165,24,132,55,
                                             122,93,34,98,122,122,150,77,42,
                                             234,177,218,195,88,56,85,244,20,
                                             251,175,108,170,27>>,
                                           4,8},
                                          {blockchain_state_channel_summary_v1_pb,
                                           <<0,207,156,103,132,195,108,28,252,
                                             175,19,38,235,149,197,94,169,26,
                                             134,215,113,57,57,63,186,86,47,
                                             119,189,108,56,182,60>>,
                                           7,14},
                                          {blockchain_state_channel_summary_v1_pb,
                                           <<0,18,207,27,179,206,87,40,55,201,
                                             161,89,227,245,127,28,198,243,36,
                                             87,15,199,186,39,206,55,136,82,68,
                                             95,206,95,206>>,
                                           17,34},
                                          {blockchain_state_channel_summary_v1_pb,
                                           <<0,109,175,146,168,41,125,231,114,
                                             105,225,151,108,133,22,174,236,
              ...
2021-08-28 15:43:35.060 160 [info] <0.1517.0>@blockchain_txn:validate:290 valid: [{blockchain_txn_poc_request_v1,1}], invalid: [{blockchain_txn_state_channel_close_v1,1}]
2021-08-28 15:43:35.070 160 [info] <0.1517.0>@blockchain_txn_mgr:process_invalid_txn:431 txn declared invalid with reason {error,{invalid,not_found}}, removing from cache and invoking callback: <<247,227,155,119,203,7,20,216,115,102,83,29,199,33,85,101,176,126,191,169,70,188,47,244,214,201,181,141,208,88,134,95>>
2021-08-28 15:43:35.491 160 [error] <0.4163.0>@blockchain_worker:send_txn:1068 failed to submit txn: type=state_channel_close, state_channel={blockchain_state_channel_v1_pb,
                                         <<39,92,74,38,60,166,13,59,153,125,
                                           174,131,100,4,115,198,192,215,134,
                                           252,181,163,227,137,14,195,176,72,
                                           150,50,199,149>>,
                                         <<0,241,20,68,146,24,117,226,239,116,
                                           53,81,58,29,31,27,15,164,158,50,66,
                                           149,106,36,56,57,18,236,93,79,25,64,
                                           119>>,
                                         250000,29203,
                                         [{blockchain_state_channel_summary_v1_pb,
                                           <<0,163,141,188,100,107,180,12,28,
                                             144,56,147,209,199,141,91,90,168,
                                             155,104,157,56,237,40,5,117,142,
                                             113,102,91,120,124,30>>,
                                           9,15},
                                          {blockchain_state_channel_summary_v1_pb,
                                           <<0,52,84,86,67,134,245,42,239,149,
                                             5,159,251,190,57,67,73,97,192,65,
                                             51,78,96,85,61,199,120,110,62,206,
                                             187,91,220>>,
                                           5,10},
                                          {blockchain_state_channel_summary_v1_pb,
                                           <<0,106,38,58,121,87,107,76,36,78,
                                             212,46,20,123,234,155,27,28,132,
                                             199,142,10,200,9,105,123,145,42,
                                             48,155,172,7,201>>,
                                           10,20},
    ... error: {invalid,not_found}
2021-08-28 15:43:37.141 160 [info] <0.4158.0>@blockchain_gossip_handler:add_block:88 got gossipped block 986012
2021-08-28 15:43:39.764 160 [warning] <0.1562.0>@miner_ebus:handle_message:96 Unhandled dbus message "org.freedesktop.DBus.Introspectable.Introspect"
2021-08-28 15:44:11.839 160 [warning] <0.1562.0>@miner_ebus:handle_message:96 Unhandled dbus message "org.freedesktop.DBus.Introspectable.Introspect"
2021-08-28 15:44:28.393 160 [info] <0.4185.0>@blockchain_txn:validate:290 valid: [], invalid: []
2021-08-28 15:44:30.888 160 [info] <0.4185.0>@blockchain_txn_state_channel_close_v1:absorb:348 Closing with conflict false
2021-08-28 15:44:34.653 160 [info] <0.1565.0>@miner_poc_statem:find_request:884 got block <<26,40,229,148,145,38,172,183,87,226,96,239,252,224,222,43,7,35,248,64,108,255,218,47,250,249,59,205,176,254,14,248>> checking content
2021-08-28 15:44:34.666 160 [info] <0.4185.0>@blockchain_gossip_handler:add_block:88 got gossipped block 986013
2021-08-28 15:44:34.709 160 [info] <0.1539.0>@miner:handle_info:461 non-consensus block 986013
2021-08-28 15:44:34.723 160 [info] <0.1517.0>@blockchain_txn:validate:290 valid: [], invalid: []
2021-08-28 15:44:43.904 160 [warning] <0.1562.0>@miner_ebus:handle_message:96 Unhandled dbus message "org.freedesktop.DBus.Introspectable.Introspect"
2021-08-28 15:45:07.320 160 [info] <0.1491.0>@blockchain_worker:start_sync:758 new block sync starting with Pid: <0.4209.0>, Ref: #Ref<0.3948368610.302252033.53793>, Peer: "/p2p/11u9D6Abckexf5pnQ...

============= miner crashes and restarts ===========================

2021-08-28 15:46:37.260 169 [info] <0.1443.0>@blockchain_sup:init:71 blockchain_sup init with [{key,{{ecc_compact,{{'ECPoint',<<4,45,171,167,233,117,136,78,18,127,57,202,65,250,25,152,233,231,...
2021-08-28 15:46:37.261 169 [info] <0.1446.0>@blockchain_swarm:init:93 blockchain_swarm init with [{key,{{ecc_compact,{{'ECPoint',<<4,45,171,167,233,117,136,78,18,127,57,202,65,250,25,152,233,231,...
2021-08-28 15:46:37.705 169 [info] <0.1483.0>@libp2p_nat_server:init:69 libp2p_nat_server init with [blockchain_swarm]
2021-08-28 15:46:37.706 169 [info] <0.1484.0>@libp2p_relay_server:init:105 libp2p_relay_server init with blockchain_swarm
2021-08-28 15:46:37.706 169 [info] <0.1485.0>@libp2p_proxy_server:init:93 libp2p_proxy_server init with [blockchain_swarm,50]
2021-08-28 15:46:37.708 169 [info] <0.1490.0>@blockchain_worker:init:323 blockchain_worker init with [{port,44158},{base_dir,"/var/data"},{update_dir,"/opt/miner/update"}]
2021-08-28 15:46:37.708 169 [info] <0.1490.0>@blockchain_worker:get_quick_sync_height_and_hash:1101 quick syncing with blessed_snapshot: {<<166,157,155,29,22,90,236,73,29,63,28,53,167,191,251,124,177,132,206,136,77,3,189,20,245,65,250,135,209,220,80,136>>,979921}
evanmcc commented 3 years ago

I don't see anything in these logs to suggest that the miner is crashing, can you clarify what you mean?

jerryyip commented 3 years ago

Sorry @evanmcc , I have just made the log more clear. You can find that the miner restarts at 2021-08-28 15:46:37.260

Domain4dmin commented 2 years ago

Any update @evanmcc? Many users report symptoms that may be caused by this issue on Discord

evanmcc commented 2 years ago

I assume that this is an OOM issue? is there a time-correlated OOM message in /var/log/messages?

Otherwise I don't have any clues from these logs. We know that there are some issues around memory use in the state channel stuff. If this is indeed the case, it's a substantial rework, so we'll need to spend some time on it, meaning that it won't happen for a few weeks. A shorter-term solution might be a throttle on packets.

Domain4dmin commented 2 years ago

@jerryyip: can you check that please? Given that Sensecap does not give the owners the login credential to the miner I cannot check.

jerryyip commented 2 years ago

The crash.log from the device: 

2021-09-10 07:54:47 =SUPERVISOR REPORT====
     Supervisor: {<0.1805.0>,libp2p_simple_sup}
     Context:    shutdown_error
     Reason:     noproc
     Offender:   [{pid,<0.1809.0>},{id,1},{mfargs,{libp2p_yamux_stream,open_stream,undefined}},{restart_type,temporary},{shutdown,1000},{child_type,worker}]

2021-09-10 07:54:53 =SUPERVISOR REPORT====
     Supervisor: {<0.1825.0>,libp2p_simple_sup}
     Context:    shutdown_error
     Reason:     noproc
     Offender:   [{pid,<0.1828.0>},{id,1},{mfargs,{libp2p_yamux_stream,open_stream,undefined}},{restart_type,temporary},{shutdown,1000},{child_type,worker}]

2021-09-10 07:56:13 =SUPERVISOR REPORT====
     Supervisor: {<0.1929.0>,libp2p_simple_sup}
     Context:    shutdown_error
     Reason:     noproc
     Offender:   [{pid,<0.1932.0>},{id,1},{mfargs,{libp2p_yamux_stream,open_stream,undefined}},{restart_type,temporary},{shutdown,1000},{child_type,worker}]

Note that the device has 4GB ram, so there is no OOM message appears in the host. Is there a way to check the miner container or the miner's VM? @evanmcc

evanmcc commented 2 years ago

those are unrelated.

2021-08-28 15:45:07.320 160 [info] <0.1491.0>@blockchain_worker:start_sync:758 new block sync starting with Pid: <0.4209.0>, Ref: #Ref<0.3948368610.302252033.53793>, Peer: "/p2p/11u9D6Abckexf5pnQ...

============= miner crashes and restarts ===========================

2021-08-28 15:46:37.260 169 [info] <0.1443.0>@blockchain_sup:init:71 blockchain_sup init with [{key,{{ecc_compact,{{'ECPoint',<<4,45,171,167,233,117,136,78,18,127,57,202,65,250,25,152,233,231,...

the 160 and 169 here do indicate a crash, though (those are the OS PID). That said, I still have nothing to go on. If the erlang code itself was crashing, the logs at the point of restart would be noisier. So this is either an OOM kill or a segfault in the VM.

Can you enable core dumps, if there's no evidence of an OOM kill?

evanmcc commented 2 years ago

Also interesting here is the full 1.5 minutes that pass between the last message and the restart. What was going on there, I wonder?

Domain4dmin commented 2 years ago

@jerryyip: can you please provide the logs or give me the login credentials to my hotspot? Other non-Sensecap hotspots in the vicinity work fine, which may suggest that this is indeed a Sensecap issue.

I don't know who is responsible for the software of the hotspots but if Sensecap has configured the VM then this might be something to look at, as mentioned @evanmcc .

abhay commented 2 years ago

Going to close this as stale for now, @jerryyip. Feel free to comment to reopen if this is still an issue and you have more details to provide.