search

helium / miner

Miner for the helium blockchain
Apache License 2.0
609 stars 265 forks source link

failed to dial challenger : not_found #1407

Closed wolasss closed 2 years ago

wolasss commented 2 years ago

Recently, I have noticed again an increased number of errors "not_found" while connecting to the challengers:

https://pastebin.com/daDTBu9P

This is a log from one miner, but I can provide many more examples if needed - recently it happens much more often than before

adrianformunda commented 2 years ago

Changing the value max_inbound_connections and outbound_gossip_connections on miner-arm64_2022.02.28.0_GAdoesn't seem to work anymore, it crashes my miner. The only reliable settings I can make it work is 96 and 16 respectively. That doesn't help much as my peer book count is around 80K. Anybody else confirm this? any ideas how to properly fix this?

You have a Panther X2 miner?

inigoflores commented 2 years ago

@microtransactions It's working for me, on the Pisces P100 and the Panther X2. No crashes.

Miner version is miner-arm64_2022.02.28.0_GA.

$ sudo docker exec helium-miner miner peer book -c
337150

I'm using these values:

  {peerbook_update_interval, 180000},
  {max_inbound_connections, 200},
  {outbound_gossip_connections, 50}
Ay0hCrypto commented 2 years ago

We took a different route of analysis, also started with much lower figures int/in/out unchanged/12/6 - unchanged/15/5 saw slight improvement before recent firmware through a wrench in things. {peerbook_update_interval, 600000}, {max_inbound_connections, 55}, {outbound_gossip_connections, 25} is what we've most recently moved to. Will update with results in a few hours. any tried reversing the inbound outbound values. 50in/100out? results? will use inigoflores' values after we conclude the testing on 600000/55/25 Machines: SB v2 SD card. SesenCAP m1 4gbRAM x2

adrianformunda commented 2 years ago

We took a different route of analysis, also started with much lower figures int/in/out unchanged/12/6 - unchanged/15/5 saw slight improvement before recent firmware through a wrench in things. {peerbook_update_interval, 600000}, {max_inbound_connections, 55}, {outbound_gossip_connections, 25} is what we've most recently moved to. Will update with results in a few hours. any tried reversing the inbound outbound values. 50in/100out? results? will use inigoflores' values after we conclude the testing on 600000/55/25 Machines: SB v2 SD card. SesenCAP m1 4gbRAM x2

Just wait to see your results.

I tried to keep the same proportion between the 2 values. (60/20). I was definetly better. However if I go higher than 120/40 my sensecap m1 4gbRAM can barelly keep up with the blockchain (basically it could not catch the height after almost 2h - 20 blocks behind and strugling), and everything (including ssh connection) had lag.

robgit77 commented 2 years ago

what's the meaning of peer book count? Just changed values from default to 

{peerbook_update_interval, 600000},
 {max_inbound_connections, 55},
 {outbound_gossip_connections, 25},

on my sensecap 4G, and after restart book is 45.635

On my pantherX2 tried : 

{peerbook_update_interval, 900000},
   {max_inbound_connections, 100},
   {outbound_gossip_connections, 75},

but can't get peer book respose: image

Ay0hCrypto commented 2 years ago

We took a different route of analysis, also started with much lower figures int/in/out unchanged/12/6 - unchanged/15/5 saw slight improvement before recent firmware through a wrench in things. {peerbook_update_interval, 600000}, {max_inbound_connections, 55}, {outbound_gossip_connections, 25} is what we've most recently moved to. Will update with results in a few hours. any tried reversing the inbound outbound values. 50in/100out? results? will use inigoflores' values after we conclude the testing on 600000/55/25 Machines: SB v2 SD card. SesenCAP m1 4gbRAM x2

Just wait to see your results.

I tried to keep the same proportion between the 2 values. (60/20). I was definetly better. However if I go higher than 120/40 my sensecap m1 4gbRAM can barelly keep up with the blockchain (basically it could not catch the height after almost 2h - 20 blocks behind and strugling), and everything (including ssh connection) had lag.

did you lower the update_interval ? we're about 15 minutes out on finish the tests. but with 55/25 it looks to be about a 2% increase in successful witness reports sent. and 1% shifted from not_found to other* issues.

Ay0hCrypto commented 2 years ago

starting Total Witnessed: = 51 Successful: = 2 ( 4%)

adrianformunda commented 2 years ago

The update interval was set to 900000 and 180000

Did not notice any difference

On Mon, Mar 7, 2022 at 11:48 PM Ay0hCrypto @.***> wrote:

We took a different route of analysis, also started with much lower figures int/in/out unchanged/12/6 - unchanged/15/5 saw slight improvement before recent firmware through a wrench in things. {peerbook_update_interval, 600000}, {max_inbound_connections, 55}, {outbound_gossip_connections, 25} is what we've most recently moved to. Will update with results in a few hours. any tried reversing the inbound outbound values. 50in/100out? results? will use inigoflores' values after we conclude the testing on 600000/55/25 Machines: SB v2 SD card. SesenCAP m1 4gbRAM x2

Just wait to see your results.

I tried to keep the same proportion between the 2 values. (60/20). I was definetly better. However if I go higher than 120/40 my sensecap m1 4gbRAM can barelly keep up with the blockchain (basically it could not catch the height after almost 2h - 20 blocks behind and strugling), and everything (including ssh connection) had lag.

did you lower the update_interval ? we're about 15 minutes out on finish the tests. but with 55/25 it looks to be about a 2% increase in successful witness reports sent. and 1% shifted from not_found to other* issues.

— Reply to this email directly, view it on GitHub https://github.com/helium/miner/issues/1407#issuecomment-1061170684, or unsubscribe https://github.com/notifications/unsubscribe-auth/AXJSHBBIY2ZJWC3PHJGRGW3U6Z2TFANCNFSM5NEOALNA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

Ay0hCrypto commented 2 years ago

erforming actions on Node: VM: miner_4633499_2090301 Animal Name: rare-felt-horse Log Location: /mnt/data/docker/volumes/1851574_miner-log/_data/console.log

Total Witnessed: = 436 (55 new) Successful: = 32 ( 7%)+11 successful (8 confirmed on explorer) Unreachable: = 314 ( 72%) +29 Send or Re-send Failed: = 29 ( 6%) + 2 Other (Witness Failures): = 61 ( 13%)+13 ** Final Results Previous results from running 15/5 no change to update_interval

Performing actions on Node: VM: miner_4633499_2090301 Animal Name: rare-felt-horse Log Location: /mnt/data/docker/volumes/1851574_miner-log/_data/console.log **

***Total Witnessed: = 381 (55) Successful: = 21 ( 5%)+11 Unreachable: = 285 ( 74%)+29 Send or Re-send Failed: = 27 ( 7%)+2 Other (Witness Failures): = 48 ( 12%) +13 Was running 15/5 when this data was pulled.

Ay0hCrypto commented 2 years ago

Was some remaining not_found in both send_witness and libp2p_transport/swarm. We will raise the inbound outbound and leave update_interval at 600000. and let it run for a day or so, or until a fw update, means we have to go back and flash the cards.***Also maintained 100% sync status throughout the 5 hours. Going to test 4 hours at 600000/85/35. start time 23:00 utc.

Ay0hCrypto commented 2 years ago

{num_consensus_members, 16}, Has anyone attempted altering this variable? 43 consensus members now... almost always validators in the peer book? Any thoughts on effect of raising the number to 43?

Bfindlay commented 2 years ago

Yesterday I had a really unbeliveable day with peerbook near 200,000 and 02.28.

General Witnesses Overview
----------------------------------
Total witnesses                   =    77
Succesfully delivered             =    72 (93.51%)
Failed                            =     5  (6.49%)
  ├── Max retry    =    5  (6.49%)
  └── Crash/reboot =    0     (0%)

Max Retry Failure Reasons
----------------------------------
Timeout                           =     4    (80%)
Not Found                         =     0     (0%)
Other challenger issues           =     1    (20%)

Challengers
----------------------------------
Not Relayed                       =    73 (94.81%)
Relayed                           =     4  (5.19%)
Unknown Relay Status              =     0     (0%)

What are your max_inbound_connections and outbound_gossip_connections set at for this result

Ay0hCrypto commented 2 years ago

in/out/interval 85/35/600000

Total Witnessed: = 25 Successful: = 9 ( 36%) Unreachable: = 9 ( 36%) Send or Re-send Failed: = 0 ( 0%) Other (Witness Failures): = 7 ( 28%)

adrianformunda commented 2 years ago

I have updated this morning with in/out/interval 100/75/900000

Will see in a few days

On Tue, Mar 8, 2022 at 4:59 AM Ay0hCrypto @.***> wrote:

in/out/interval 85/35/600000

Total Witnessed: = 25 Successful: = 9 ( 36%) Unreachable: = 9 ( 36%) Send or Re-send Failed: = 0 ( 0%) Other (Witness Failures): = 7 ( 28%)

— Reply to this email directly, view it on GitHub https://github.com/helium/miner/issues/1407#issuecomment-1061355924, or unsubscribe https://github.com/notifications/unsubscribe-auth/AXJSHBEJZ25PPYVP342OBFDU627CRANCNFSM5NEOALNA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

JohnnyMirza commented 2 years ago

I have noticeably reduced the number of failed witnesses by increasing the max_inbound_connections and outbound_gossip_connections in the docker's sys.config. For example:

docker exec -it miner sh
vi config/sys.config

I changed

   {max_inbound_connections, 6},                                                                                                                                          
   {outbound_gossip_connections, 2},

to something like

   {max_inbound_connections, 20},                                                                                                                                          
   {outbound_gossip_connections, 10},

Then restarted the docker:

docker stop miner && docker start miner

I normally log the ratio of successfully sent witness entries to failed witness entries (i.e. 'max_retry') from console.log per hour. There is a noticeable jump toward 1.0 when I made this change:

Screen Shot 2022-02-10 at 12 15 32 PM

I presume that this is going to drastically increase my network traffic, but I've got an excellent internet connection, so I'm okay with that if I get more witnesses through

@cuddlyconcretepangolin how did you build that chart of max_retry? Anything you can share on how you pulled those metrics? Thanks

Ay0hCrypto commented 2 years ago

I have updated this morning with in/out/interval 100/75/900000 Will see in a few days … On Tue, Mar 8, 2022 at 4:59 AM Ay0hCrypto @.***> wrote: in/out/interval 85/35/600000

Total Witnessed: = 25 Successful: = 9 ( 36%) Unreachable: = 9 ( 36%) Send or Re-send Failed: = 0 ( 0%) Other (Witness Failures): = 7 ( 28%) ------------------------------ —

we upped to 150/50 last night on test units interval 600000, so far maintaining sync, and improved success rate, lowered failure rates. Miners still requiring a reboot every few hours to maintain full functionality. Have not identified what is causing the miners to stop witnessing after a few hours. Saw the same failure to dial proxy errors creep up after a few hours with the 150/50 in/out values. 85/35 seemed the closest to 100% success rate, but our tracking was a little off, so it's hard to judge total vs. successful vs failed 1+ times vs failed totally. 85/35 >90% success <10% various fail rate 75/25 =88% success 12% various fail 85/15 =73% success 6% max retry failure 21% various fail

microtransactions commented 2 years ago

@microtransactions It's working for me, on the Pisces P100 and the Panther X2. No crashes.

Miner version is miner-arm64_2022.02.28.0_GA.

$ sudo docker exec helium-miner miner peer book -c
337150

I'm using these values:

  {peerbook_update_interval, 180000},
  {max_inbound_connections, 200},
  {outbound_gossip_connections, 50}

Same problem here, if I go above 100 for inbound connections it crashes. It seems that it can't connect to proxy servers?

2022-03-08 16:30:36.732 7 [info] <0.1705.0>@libp2p_group_gossip_server:add_worker_pids:{641,27} 1 new pids of type peerbook 2022-03-08 16:30:36.732 7 [info] <0.1705.0>@libp2p_group_gossip_server:add_worker_pids:{641,27} 0 new pids of type seed 2022-03-08 16:30:36.732 7 [info] <0.1705.0>@libp2p_group_gossip_server:add_worker_pids:{641,27} 0 new pids of type inbound 2022-03-08 16:30:36.755 7 [info] <0.1705.0>@libp2p_group_gossip_server:add_worker_pids:{641,27} 1 new pids of type peerbook 2022-03-08 16:30:36.755 7 [info] <0.1705.0>@libp2p_group_gossip_server:add_worker_pids:{641,27} 0 new pids of type seed 2022-03-08 16:30:36.755 7 [info] <0.1705.0>@libp2p_group_gossip_server:add_worker_pids:{641,27} 0 new pids of type inbound 2022-03-08 16:30:36.788 7 [info] <0.1705.0>@libp2p_group_gossip_server:add_worker_pids:{641,27} 1 new pids of type peerbook 2022-03-08 16:30:36.788 7 [info] <0.1705.0>@libp2p_group_gossip_server:add_worker_pids:{641,27} 0 new pids of type seed 2022-03-08 16:30:36.788 7 [info] <0.1705.0>@libp2p_group_gossip_server:add_worker_pids:{641,27} 0 new pids of type inbound 2022-03-08 16:30:36.854 7 [notice] <0.2194.0>@libp2p_multistream_server:handle_info:{108,13} Failed to handshake client "/ip4/178.203.225.176/tcp/61666": closed 2022-03-08 16:30:36.907 7 [info] <0.1705.0>@libp2p_group_gossip_server:add_worker_pids:{641,27} 1 new pids of type peerbook 2022-03-08 16:30:36.907 7 [info] <0.1705.0>@libp2p_group_gossip_server:add_worker_pids:{641,27} 0 new pids of type seed 2022-03-08 16:30:36.908 7 [info] <0.1705.0>@libp2p_group_gossip_server:add_worker_pids:{641,27} 0 new pids of type inbound 2022-03-08 16:30:36.908 7 [info] <0.1738.0>@libp2p_group_worker:connected:{330,5} Changing target from {"/p2p/11LhnFux6zNS9HqHHPx1Bs8MFejvFiLFdpo6y6GaEeiqL2K7Yrm",{["gossip/1.0.2","gossip/1.0.0"],{libp2p_gossip_stream,[libp2p_group_gossip_server,<0.1705.0>,blockchain_swarm,#Ref<0.2113374580.2093088771.52541>,peerbook,"/p2p/11LhnFux6zNS9HqHHPx1Bs8MFejvFiLFdpo6y6GaEeiqL2K7Yrm"]}}} to {"/p2p/112HpuZo7zhxUeqncPUyNpzGsvGEeFVLLaYKgSyNcGpLw4yxJPMk",{["gossip/1.0.2","gossip/1.0.0"],{libp2p_gossip_stream,[libp2p_group_gossip_server,<0.1705.0>,blockchain_swarm,#Ref<0.2113374580.2093088771.52541>,peerbook,"/p2p/112HpuZo7zhxUeqncPUyNpzGsvGEeFVLLaYKgSyNcGpLw4yxJPMk"]}}} 2022-03-08 16:30:36.933 7 [info] <0.1705.0>@libp2p_group_gossip_server:add_worker_pids:{641,27} 1 new pids of type peerbook 2022-03-08 16:30:36.933 7 [info] <0.1705.0>@libp2p_group_gossip_server:add_worker_pids:{641,27} 0 new pids of type seed 2022-03-08 16:30:36.933 7 [info] <0.1705.0>@libp2p_group_gossip_server:add_worker_pids:{641,27} 0 new pids of type inbound 2022-03-08 16:30:37.052 7 [info] <0.1705.0>@libp2p_group_gossip_server:add_worker_pids:{641,27} 1 new pids of type peerbook 2022-03-08 16:30:37.052 7 [info] <0.1705.0>@libp2p_group_gossip_server:add_worker_pids:{641,27} 0 new pids of type seed 2022-03-08 16:30:37.052 7 [info] <0.1705.0>@libp2p_group_gossip_server:add_worker_pids:{641,27} 0 new pids of type inbound 2022-03-08 16:30:37.240 7 [info] <0.1705.0>@libp2p_group_gossip_server:add_worker_pids:{641,27} 1 new pids of type peerbook 2022-03-08 16:30:37.240 7 [info] <0.1705.0>@libp2p_group_gossip_server:add_worker_pids:{641,27} 0 new pids of type seed 2022-03-08 16:30:37.241 7 [info] <0.1705.0>@libp2p_group_gossip_server:add_worker_pids:{641,27} 0 new pids of type inbound 2022-03-08 16:30:37.591 7 [notice] <0.1993.0>@libp2p_transport:start_server_session:{221,13} Duplicate session for "/ip4/111.22.75.95/tcp/44158" at <0.1941.0> 2022-03-08 16:30:41.709 7 [info] <0.2027.0>@libp2p_transport_proxy:connect:{38,5} init proxy with ["/p2p/112BAwpWQn17EZFHj8pofSr77E9wt5fwWWFtRvzwMuxFx3fcqV6a","/p2p/112K6oa3YMTYoqAoy6gxHnnok4LYkSXDYoV1m672x5rNZVGkd8x3"] 2022-03-08 16:30:41.734 7 [error] <0.2027.0>@libp2p_transport_proxy:connect_to:{69,13} failed to dial proxy server "/p2p/112BAwpWQn17EZFHj8pofSr77E9wt5fwWWFtRvzwMuxFx3fcqV6a" not_found 2022-03-08 16:30:41.809 7 [info] <0.1758.0>@blockchain_worker:start_sync:{885,13} new block sync starting with Pid: <0.2330.0>, Ref: #Ref<0.2113374580.2092957698.54515>, Peer: "/p2p/112HbW11mtzeNDF6GPoqCAjPnGo4gJtKs7v9RUH3cjYTYGEjnE5U" 2022-03-08 16:30:41.857 7 [info] <0.1701.0>@libp2p_transport_tcp:handle_info:{529,13} stungun detected NAT type unknown 2022-03-08 16:30:44.734 7 [info] <0.2333.0>@blockchain_sync_handler:handle_data:{176,29} sending blocks [1258222,1258223,1258224,1258225,1258226] to sync peer 2022-03-08 16:30:50.414 7 [info] <0.2354.0>@libp2p_nat_server:init:{69,5} libp2p_nat_server init with [blockchain_swarm] 2022-03-08 16:30:50.414 7 [error] emulator@Undefined:Undefined:Undefined Error in process <0.2007.0> on node 'miner@127.0.0.1' with exit value: {{badmatch,{error,eagain}},[{natupnp_v2,discover1,3,[{file,"natupnp_v2.erl"},{line,65}]},{natupnp_v2,discover,0,[{file,"natupnp_v2.erl"},{line,55}]},{nat,discover_worker,3,[{file,"nat.erl"},{line,145}]}]} 2022-03-08 16:30:50.415 7 [error] <0.1747.0>@natupnp_v2:discover1:65 Supervisor libp2p_swarm_auxiliary_sup_blockchain_swarm had child nat started with libp2p_nat_server:start_link(blockchain_swarm) at <0.1753.0> exit with reason no match of right hand value {error,eagain} in natupnp_v2:discover1/3 line 65 in context child_terminated 2022-03-08 16:30:52.311 7 [warning] <0.1756.0>@blockchain_event:terminate:{108,5} terminating remove_handler 2022-03-08 16:30:52.312 7 [error] <0.1676.0>@miner_keys:keys:156 Supervisor miner_sup had child miner_restart_sup started with miner_restart_sup:start_link() at undefined exit with reason no match of right hand value {error,too_many_retries} in miner_keys:keys/1 line 156 in context start_error 2022-03-08 16:30:52.316 7 [warning] <0.1756.0>@blockchain_event:terminate:{108,5} terminating remove_handler 2022-03-08 16:30:52.316 7 [warning] <0.1756.0>@blockchain_event:terminate:{108,5} terminating remove_handler 2022-03-08 16:30:52.319 7 [error] <0.2062.0>@miner_keys:keys:156 CRASH REPORT Process <0.2062.0> with 0 neighbours crashed with reason: no match of right hand value {error,too_many_retries} in miner_keys:keys/1 line 156 2022-03-08 16:30:52.319 7 [error] <0.1674.0>@Undefined:Undefined:Undefined CRASH REPORT Process <0.1674.0> with 0 neighbours exited with reason: {{shutdown,{failed_to_start_child,miner_restart_sup,{{badmatch,{error,too_many_retries}},[{miner_keys,keys,1,[{file,"miner_keys.erl"},{line,156}]},{miner_restart_sup,init,1,[{file,"miner_restart_sup.erl"},{line,51}]},{supervisor,init,1,[{file,"supervisor.erl"},{line,330}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,423}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,390}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}}},{miner_app,start,[normal,[]]}} in application_master:init/4 line 142 2022-03-08 16:30:52.320 7 [info] <0.1323.0>@Undefined:Undefined:Undefined Application miner exited with reason: {{shutdown,{failed_to_start_child,miner_restart_sup,{{badmatch,{error,too_many_retries}},[{miner_keys,keys,1,[{file,"miner_keys.erl"},{line,156}]},{miner_restart_sup,init,1,[{file,"miner_restart_sup.erl"},{line,51}]},{supervisor,init,1,[{file,"supervisor.erl"},{line,330}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,423}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,390}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}}},{miner_app,start,[normal,[]]}} 2022-03-08 16:30:52.343 7 [warning] <0.1700.0>@libp2p_swarm_server:terminate:{134,5} going down shutdow

adrianformunda commented 2 years ago

I have reflashed the sd card for sensecap and i am using it with 100/75/900000 It is working

I will provide feedback after several days

On Tue, Mar 8, 2022 at 6:37 PM microtransactions @.***> wrote:

@microtransactions https://github.com/microtransactions It's working for me, on the Pisces P100 and the Panther X2. No crashes.

Miner version is miner-arm64_2022.02.28.0_GA.

$ sudo docker exec helium-miner miner peer book -c 337150

I'm using these values:

{peerbook_update_interval, 180000}, {max_inbound_connections, 200}, {outbound_gossip_connections, 50}

Same problem here, if I go above 100 for inbound connections it crashes. It seems that it can't connect to proxy servers?

2022-03-08 16:30:36.732 7 [info] @._group_gossip_server:add_worker_pids:{641,27} 1 new pids of type peerbook 2022-03-08 16:30:36.732 7 [info] @._group_gossip_server:add_worker_pids:{641,27} 0 new pids of type seed 2022-03-08 16:30:36.732 7 [info] @._group_gossip_server:add_worker_pids:{641,27} 0 new pids of type inbound 2022-03-08 16:30:36.755 7 [info] @._group_gossip_server:add_worker_pids:{641,27} 1 new pids of type peerbook 2022-03-08 16:30:36.755 7 [info] @._group_gossip_server:add_worker_pids:{641,27} 0 new pids of type seed 2022-03-08 16:30:36.755 7 [info] @._group_gossip_server:add_worker_pids:{641,27} 0 new pids of type inbound 2022-03-08 16:30:36.788 7 [info] @._group_gossip_server:add_worker_pids:{641,27} 1 new pids of type peerbook 2022-03-08 16:30:36.788 7 [info] @._group_gossip_server:add_worker_pids:{641,27} 0 new pids of type seed 2022-03-08 16:30:36.788 7 [info] @._group_gossip_server:add_worker_pids:{641,27} 0 new pids of type inbound 2022-03-08 16:30:36.854 7 [notice] @._multistream_server:handle_info:{108,13} Failed to handshake client "/ip4/178.203.225.176/tcp/61666": closed 2022-03-08 16:30:36.907 7 [info] @._group_gossip_server:add_worker_pids:{641,27} 1 new pids of type peerbook 2022-03-08 16:30:36.907 7 [info] @._group_gossip_server:add_worker_pids:{641,27} 0 new pids of type seed 2022-03-08 16:30:36.908 7 [info] @._group_gossip_server:add_worker_pids:{641,27} 0 new pids of type inbound 2022-03-08 16:30:36.908 7 [info] @._group_worker:connected:{330,5} Changing target from {"/p2p/11LhnFux6zNS9HqHHPx1Bs8MFejvFiLFdpo6y6GaEeiqL2K7Yrm",{["gossip/1.0.2","gossip/1.0.0"],{libp2p_gossip_stream,[libp2p_group_gossip_server,<0.1705.0>,blockchain_swarm,#Ref<0.2113374580.2093088771.52541>,peerbook,"/p2p/11LhnFux6zNS9HqHHPx1Bs8MFejvFiLFdpo6y6GaEeiqL2K7Yrm"]}}} to {"/p2p/112HpuZo7zhxUeqncPUyNpzGsvGEeFVLLaYKgSyNcGpLw4yxJPMk",{["gossip/1.0.2","gossip/1.0.0"],{libp2p_gossip_stream,[libp2p_group_gossip_server,<0.1705.0>,blockchain_swarm,#Ref<0.2113374580.2093088771.52541>,peerbook,"/p2p/112HpuZo7zhxUeqncPUyNpzGsvGEeFVLLaYKgSyNcGpLw4yxJPMk"]}}} 2022-03-08 16:30:36.933 7 [info] @._group_gossip_server:add_worker_pids:{641,27} 1 new pids of type peerbook 2022-03-08 16:30:36.933 7 [info] @._group_gossip_server:add_worker_pids:{641,27} 0 new pids of type seed 2022-03-08 16:30:36.933 7 [info] @._group_gossip_server:add_worker_pids:{641,27} 0 new pids of type inbound 2022-03-08 16:30:37.052 7 [info] @._group_gossip_server:add_worker_pids:{641,27} 1 new pids of type peerbook 2022-03-08 16:30:37.052 7 [info] @._group_gossip_server:add_worker_pids:{641,27} 0 new pids of type seed 2022-03-08 16:30:37.052 7 [info] @._group_gossip_server:add_worker_pids:{641,27} 0 new pids of type inbound 2022-03-08 16:30:37.240 7 [info] @._group_gossip_server:add_worker_pids:{641,27} 1 new pids of type peerbook 2022-03-08 16:30:37.240 7 [info] @._group_gossip_server:add_worker_pids:{641,27} 0 new pids of type seed 2022-03-08 16:30:37.241 7 [info] @._group_gossip_server:add_worker_pids:{641,27} 0 new pids of type inbound 2022-03-08 16:30:37.591 7 [notice] @._transport:start_server_session:{221,13} Duplicate session for "/ip4/111.22.75.95/tcp/44158" at <0.1941.0> 2022-03-08 16:30:41.709 7 [info] @._transport_proxy:connect:{38,5} init proxy with ["/p2p/112BAwpWQn17EZFHj8pofSr77E9wt5fwWWFtRvzwMuxFx3fcqV6a","/p2p/112K6oa3YMTYoqAoy6gxHnnok4LYkSXDYoV1m672x5rNZVGkd8x3"] 2022-03-08 16:30:41.734 7 [error] @._transport_proxy:connect_to:{69,13} failed to dial proxy server "/p2p/112BAwpWQn17EZFHj8pofSr77E9wt5fwWWFtRvzwMuxFx3fcqV6a" not_found 2022-03-08 16:30:41.809 7 [info] @._worker:start_sync:{885,13} new block sync starting with Pid: <0.2330.0>, Ref: #Ref<0.2113374580.2092957698.54515>, Peer: "/p2p/112HbW11mtzeNDF6GPoqCAjPnGo4gJtKs7v9RUH3cjYTYGEjnE5U" 2022-03-08 16:30:41.857 7 [info] @._transport_tcp:handle_info:{529,13} stungun detected NAT type unknown 2022-03-08 16:30:44.734 7 [info] @._sync_handler:handle_data:{176,29} sending blocks [1258222,1258223,1258224,1258225,1258226] to sync peer 2022-03-08 16:30:50.414 7 [info] @._nat_server:init:{69,5} libp2p_nat_server init with [blockchain_swarm] 2022-03-08 16:30:50.414 7 [error] @.:Undefined:Undefined Error in process <0.2007.0> on node @.' with exit value: {{badmatch,{error,eagain}},[{natupnp_v2,discover1,3,[{file,"natupnp_v2.erl"},{line,65}]},{natupnp_v2,discover,0,[{file,"natupnp_v2.erl"},{line,55}]},{nat,discover_worker,3,[{file,"nat.erl"},{line,145}]}]} 2022-03-08 16:30:50.415 7 [error] @._v2:discover1:65 Supervisor libp2p_swarm_auxiliary_sup_blockchain_swarm had child nat started with libp2p_nat_server:start_link(blockchain_swarm) at <0.1753.0> exit with reason no match of right hand value {error,eagain} in natupnp_v2:discover1/3 line 65 in context child_terminated 2022-03-08 16:30:52.311 7 [warning] @._event:terminate:{108,5} terminating remove_handler 2022-03-08 16:30:52.312 7 [error] @._keys:keys:156 Supervisor miner_sup had child miner_restart_sup started with miner_restart_sup:start_link() at undefined exit with reason no match of right hand value {error,too_many_retries} in miner_keys:keys/1 line 156 in context start_error 2022-03-08 16:30:52.316 7 [warning] @._event:terminate:{108,5} terminating remove_handler 2022-03-08 16:30:52.316 7 [warning] @._event:terminate:{108,5} terminating remove_handler 2022-03-08 16:30:52.319 7 [error] @._keys:keys:156 CRASH REPORT Process <0.2062.0> with 0 neighbours crashed with reason: no match of right hand value {error,too_many_retries} in miner_keys:keys/1 line 156 2022-03-08 16:30:52.319 7 [error] @.:Undefined:Undefined CRASH REPORT Process <0.1674.0> with 0 neighbours exited with reason: {{shutdown,{failed_to_start_child,miner_restart_sup,{{badmatch,{error,too_many_retries}},[{miner_keys,keys,1,[{file,"miner_keys.erl"},{line,156}]},{miner_restart_sup,init,1,[{file,"miner_restart_sup.erl"},{line,51}]},{supervisor,init,1,[{file,"supervisor.erl"},{line,330}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,423}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,390}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}}},{miner_app,start,[normal,[]]}} in application_master:init/4 line 142 2022-03-08 16:30:52.320 7 [info] @.:Undefined:Undefined Application miner exited with reason: {{shutdown,{failed_to_start_child,miner_restart_sup,{{badmatch,{error,too_many_retries}},[{miner_keys,keys,1,[{file,"miner_keys.erl"},{line,156}]},{miner_restart_sup,init,1,[{file,"miner_restart_sup.erl"},{line,51}]},{supervisor,init,1,[{file,"supervisor.erl"},{line,330}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,423}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,390}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}}},{miner_app,start,[normal,[]]}} 2022-03-08 16:30:52.343 7 [warning] @.***_swarm_server:terminate:{134,5} going down shutdow

— Reply to this email directly, view it on GitHub https://github.com/helium/miner/issues/1407#issuecomment-1061973818, or unsubscribe https://github.com/notifications/unsubscribe-auth/AXJSHBANXZVUZRLEHR7NUSTU6565FANCNFSM5NEOALNA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

Ay0hCrypto commented 2 years ago

@microtransactions connection speed upload/download, modem type docsis 2.0, 3.0, 3.1? We were running some other tests, where we found sub 10mb** upload connections, and sub docsis 3.0 modems or networks with restricted channel usage could not perform as well as miners on networks with higher specs.

microtransactions commented 2 years ago

@microtransactions connection speed upload/download, modem type docsis 2.0, 3.0, 3.1? We were running some other tests, where we found sub 10gb upload connections, and sub docsis 3.0 modems or networks with restricted channel usage could not perform as well as miners on networks with higher specs.

I think this makes sense. My router is on ethernet to the main ISP's switch and that runs on fiber, with 1Gbit up/down connection. The miner is on 2.4GHz wifi due to it being on the exterior. However the 200/50 setting worked for a period of time some weeks ago on older miner versions. With 48/16 settings I reach around 140K peers, it's good enough I guess.

Ay0hCrypto commented 2 years ago

We tested all the way upto 150/50 where we began seeing issues with proxy server connections, more activity p2p, but far less PoC activity. have lowered to 75/25 showing good for now, but 150/50 performed well for the first hour or so, before errors began appearing more regularly. also wondering what stat everyone else is tracking for peers being so high. Heliumtracker.io and our gossip-peer analyzer is showing 20-200 range. nothing in the thousands.

DimitrisSar commented 2 years ago

We tested all the way upto 150/50 where we began seeing issues with proxy server connections, more activity p2p, but far less PoC activity. have lowered to 75/25 showing good for now, but 150/50 performed well for the first hour or so, before errors began appearing more regularly. also wondering what stat everyone else is tracking for peers being so high. Heliumtracker.io and our gossip-peer analyzer is showing 20-200 range. nothing in the thousands.

Using logs in folder /var/log/miner/

General Witnesses Overview

Total witnesses = 247 (17.57/hour) Succesfully delivered = 204 (82.59%) Failed = 43 (17.41%) ├── Max retry = 43 (17.41%) └── Crash/reboot = 0 (0%)

Max Retry Failure Reasons

Timeout = 27 (10.93%) Not Found = 14 (5.67%) Other challenger issues = 2 (0.81%)

Challengers

Not Relayed = 140 (56.68%) Relayed = 78 (31.58%) Unknown (Probably Not Relayed) = 29 (11.74%)

{peerbook_update_interval, 180000}, {max_inbound_connections, 200}, {outbound_gossip_connections, 50}

@Ay0hCrypto : can you open source your log analyzer script and share it with: https://github.com/inigoflores/helium-miner-log-analyzer so we can have a nice new, combined version?

I like your stats also :)

image

My system: Controllino v1 (rPi4b / 8GB RAM) 1Gbps ISP, uptime: 30 hours: 15,9 GB TX !

image image

Ay0hCrypto commented 2 years ago

The analyzer and script is all Source-Code's, but i'll grab the raw script https://raw.githubusercontent.com/saad-akhtar/helium_miner_checker/main/hmc.sh updated for Math and peer statistics Total Witnessed: = 12 |-- Sending: = 12 |-- Resending: = 11 Successful: = 12 ( 100%) Unreachable: = 11 ( 91%) Send or Re-send Failed: = 0 ( 0%) Other (Witness Failures): = 12 ( 100%) Challenger Issues: |-- Challenger Not Found: = 9 |-- Challenger Timed Out: = 2 |-- Challenger Refused Connection: = 0 |-- Challenger Unreachable: = 0 |-- Challenger No Listening Address: = 0 Total Peer Activity: = 95 |-- Timeouts: = 23 |-- Proxy Session Timeouts: = 2 |-- Relay Session Timeouts: = 9 |-- Normal Exit: = 31 |-- Not Found: = 17 |-- Server Down: = 1 |-- Failed to Dial Proxy: = 5 |-- Connection Refused: = 7 |-- Connection Closed: = 8

Bfindlay commented 2 years ago

Do we have any info on the effect of {peerbook_update_interval, 900000},?

I updated to 

  {max_inbound_connections, 20},
  {outbound_gossip_connections, 10},

and saw a noticable drop of failures over 24hrs. However after 24hrs my peerbook size dropped from >100k to 50k and the errors started again.

I have now updated to this and will monitor for another 24 hours

{peerbook_update_interval, 900000},
  {max_inbound_connections, 100},
  {outbound_gossip_connections, 75},
DimitrisSar commented 2 years ago

in theory, the peerbook_update_interval is measured in milli-seconds

so: 900.000 ms = 900 s = 15 m 180.000 ms = 180 s = 3 m

I don't believe that peerbook_update_interval is measured in seconds. that would mean: 10,4 days and 2,09 days respectively

I have tried both 900.000 and 180.000 and I haven't concluded yet if these is a benefit in lowering this below the default. In both cases, the increase in volume of traffic is enormous (we are talking about 20+ Gigs in TX per day!)

We have to identify also the interval between retries before failure. There is a max of 10 retries to deliver a witness report but I haven't identified yet the interval between these tries.

https://github.com/helium/erlang-libp2p/blob/d587854c8a1f093576bbd05ab4616bcc2693e600/src/libp2p_transport_p2p.erl#L81

adrianformunda commented 2 years ago

On my sensecap with 100/75 peerbook size is 176000 On my Panther with 100/75 peerbook size is 97000

On Wed, Mar 9, 2022 at 6:56 AM Brett Findlay @.***> wrote:

Do we have any info on the effect of {peerbook_update_interval, 900000},?

I updated to

{max_inbound_connections, 20}, {outbound_gossip_connections, 10},

and saw a noticable drop of failures over 24hrs. However after 24hrs my peerbook size dropped from >100k to 50k and the errors started again.

I have now updated to this and will monitor for another 24 hours

{peerbook_update_interval, 900000}, {max_inbound_connections, 100}, {outbound_gossip_connections, 75},

— Reply to this email directly, view it on GitHub https://github.com/helium/miner/issues/1407#issuecomment-1062554282, or unsubscribe https://github.com/notifications/unsubscribe-auth/AXJSHBA4SRL7FO5TDHAUPQ3U7AVRNANCNFSM5NEOALNA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

Ay0hCrypto commented 2 years ago

85/35 >90% success <10% various fails (tested 4 hours, longer testing may alter results) 75/25 =88% success 12% various fails (10+ hours) 85/15 =73% success 6% max retry failure 21% various fails (12 hours) Things we noticed:

1 successful Not-RELAYED witnesses achieved after more than 1 failure, had 25% the chance of making it on chain as those with 1 or 0 failures before successfully sent to challenger. *They performed approximately evenly if it was successfully sent to a p2p-circuit address after failing the non-relayed address.

2 higher numbers received more success until we got proxy errors, but didn't always correlate to higher earnings/more witnesses appearing on block.

3 even when higher numbers did produce higher gains, it created a wave pattern in activity, where slightly lower numbers showed more consistent results.

4 We tried lower 15/5 30/10 and higher 85/35 150/50, best results so far found between 75-90 in bound / 25-35 outbound.

5 the longer we tested any variable the less successful they became. except 75/25 which bounced between 72-76% success rate

6 two years ago the sys.config file looked like

image

DimitrisSar commented 2 years ago

excellent info.

so: would you recommend something between 75/25 - 90/35 ?

quick question on your assessment of the peerbook_update_interval: does it play any role and if yes: what would you recommend for its value?

Ay0hCrypto commented 2 years ago

excellent info.

so: would you recommend something between 75/25 - 90/35 ?

quick question on your assessment of the peerbook_update_interval: does it play any role and if yes: what would you recommend for its value?

I'm testing 80/20 to see if values = or <100 do better than values over 100. but so far 75-90 in and 15-35 out is what i'd recommend. We ran mainly on 600000 except 15/5 which was run on the 900000 interval and 85/15 which was run on 400000 interval, no noticed difference in intervals yet.. i'm running 80/20 on 600000, to see how it performs compared to 75/25 and 85/15. then we will adjust interval and see if it has any impact up or down. We're now trying to test for at least 12 hours, as most changes seemed less dramatic after 8 hours, and all seemed super successful during the first 3-4 hours.

adrianformunda commented 2 years ago

Hello

PantherX2 100/75/900000 peer book size 167000 (24h after making the changes)

success rate 82%

On Wed, Mar 9, 2022 at 7:31 PM Ay0hCrypto @.***> wrote:

85/35 >90% success <10% various fails (tested 4 hours, longer testing may alter results) 75/25 =88% success 12% various fails (10+ hours) 85/15 =73% success 6% max retry failure 21% various fails (12 hours) Things we noticed:

1 https://github.com/helium/miner/pull/1 successful witnesses achieved

after more than 1 failure, had 25% the chance of making it on chain as those with 1 or 0 failures before successfully sent to challenger.

2 https://github.com/helium/miner/pull/2 higher numbers received more

success until we got proxy errors, but didn't always correlate to higher earnings/more witnesses appearing on block.

3 https://github.com/helium/miner/pull/3 even when higher numbers did

produce higher gains, it created a wave pattern in activity, where slightly lower numbers showed more consistent results.

4 https://github.com/helium/miner/pull/4 We tried lower 15/5 30/10 and

higher 95/35 150/50, best results so far found between 75-90 in bound / 25-35 outbound.

5 https://github.com/helium/miner/pull/5 the longer we tested any

variable the less successful they became. except 75/25 which bounced between 72-76% success rate

6 https://github.com/helium/miner/pull/6 two years ago the sys.config

file looked like [image: image] https://user-images.githubusercontent.com/98350820/157497789-a4fc52d0-37f5-4c3c-b776-5a63a2337796.png

— Reply to this email directly, view it on GitHub https://github.com/helium/miner/issues/1407#issuecomment-1063178741, or unsubscribe https://github.com/notifications/unsubscribe-auth/AXJSHBF6BTN4CWBC4C6NMF3U7DN6XANCNFSM5NEOALNA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

JohnnyMirza commented 2 years ago

I've been testing max_inbound_connections,75 / outbound_gossip_connections, 25 / peerbook_update_interval,900000 for a couple of days. See below results.

in summary went from 80% failed, to 10% failed. (also xx out all the actually numbers)

/opt/miner # /root/processlogs.php **-s 2022-03-09** -p /var/data/log/
Using logs in folder /var/data/log/
General Witnesses Overview
----------------------------------
Total witnesses                   =   xx
Succesfully delivered             =   xx (89.38%)
Failed                            =    xx (10.62%)
  ├── Max retry    =   x (10.62%)
  └── Crash/reboot =    0     (0%)
Max Retry Failure Reasons
----------------------------------
Timeout                           =    xx  (6.51%)
Not Found                         =     xx  (2.74%)
Other challenger issues           =     x (1.37%)
Challengers
----------------------------------
Not Relayed                       =   xx (92.12%)
Relayed                           =    xx  (5.14%)
Unknown Relay Status              =     xx (2.74%)
/opt/miner # /root/processlogs.php -s **2022-03-08 -e 2022-03-09** -p /var/data/log/
Using logs in folder /var/data/log/
General Witnesses Overview
----------------------------------
Total witnesses                   =   xx
Succesfully delivered             =   xx  (54.2%)
Failed                            =   xx  (45.8%)
  ├── Max retry    =  xx (44.54%)
  └── Crash/reboot =    xx  (1.26%)
Max Retry Failure Reasons
----------------------------------
Timeout                           =    xx  (9.24%)
Not Found                         =    xx (34.87%)
Other challenger issues           =    xx (0.42%)

Challengers
----------------------------------
Not Relayed                       =   xx (55.46%)
Relayed                           =    xx (10.92%)
Unknown Relay Status              =    xx (33.61%)
/opt/miner # /root/processlogs.php -s **2022-03-07 -e 2022-03-08** -p /var/data/log/
Using logs in folder /var/data/log/
General Witnesses Overview
----------------------------------
Total witnesses                   =   xx
Succesfully delivered             =    xx (18.65%)
Failed                            =   xx (81.35%)
  ├── Max retry    =  xx (81.35%)
  └── Crash/reboot =    0     (0%)
Max Retry Failure Reasons
----------------------------------
Timeout                           =    xx  (5.18%)
Not Found                         =   xx (75.65%)
Other challenger issues           =     1  (0.52%)
Challengers
----------------------------------
Not Relayed                       =    xx (19.69%)
Relayed                           =    xx  (6.22%)
Unknown Relay Status              =   xx (74.09%)
adrianformunda commented 2 years ago

what type of miner?

On Wed, Mar 9, 2022 at 11:41 PM Johnny Mirza @.***> wrote:

I've been testing max_inbound_connections,75 / outbound_gossip_connections, 25 / peerbook_update_interval,900000 for a couple of days. See below results.

in summary went from 80% failed, to 10% failed. (also xx out all the actually numbers) /opt/miner # /root/processlogs.php -s 2022-03-09 -p /var/data/log/ Using logs in folder /var/data/log/ General Witnesses Overview Total witnesses = xx Succesfully delivered = xx (89.38%) Failed = xx (10.62%) ├── Max retry = x (10.62%) └── Crash/reboot = 0 (0%) Max Retry Failure Reasons Timeout = xx (6.51%) Not Found = xx (2.74%) Other challenger issues = x (1.37%) Challengers

Not Relayed = xx (92.12%) Relayed = xx (5.14%) Unknown Relay Status = xx (2.74%)

/opt/miner # /root/processlogs.php -s 2022-03-08 -e 2022-03-09 -p /var/data/log/ Using logs in folder /var/data/log/ General Witnesses Overview Total witnesses = xx Succesfully delivered = xx (54.2%) Failed = xx (45.8%) ├── Max retry = xx (44.54%) └── Crash/reboot = xx (1.26%) Max Retry Failure Reasons

Timeout = xx (9.24%) Not Found = xx (34.87%) Other challenger issues = xx (0.42%) Challengers

Not Relayed = xx (55.46%) Relayed = xx (10.92%) Unknown Relay Status = xx (33.61%)

/opt/miner # /root/processlogs.php -s 2022-03-07 -e 2022-03-08 -p /var/data/log/ Using logs in folder /var/data/log/ General Witnesses Overview Total witnesses = xx Succesfully delivered = xx (18.65%) Failed = xx (81.35%) ├── Max retry = xx (81.35%) └── Crash/reboot = 0 (0%) Max Retry Failure Reasons Timeout = xx (5.18%) Not Found = xx (75.65%) Other challenger issues = 1 (0.52%) Challengers

Not Relayed = xx (19.69%) Relayed = xx (6.22%) Unknown Relay Status = xx (74.09%)

— Reply to this email directly, view it on GitHub https://github.com/helium/miner/issues/1407#issuecomment-1063399532, or unsubscribe https://github.com/notifications/unsubscribe-auth/AXJSHBHRZBXR7L2GAKBVR5TU7ELIDANCNFSM5NEOALNA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

JohnnyMirza commented 2 years ago

Milesight UG65

Ay0hCrypto commented 2 years ago

Milesight UG65

That looks pretty solid. Amazing data, how are ya'll pulling multiple days data together? SenseCAP seems to have the data logs stored by day, as they reset at 00:00 UTC, and thus the script we're using can only possibly track 00:00:01 to current time.

Currently, it appears the higher interval or lower inbound connections see a raise in not_found failures Also, appears to give a noticeable improvement to successful challenges and being challenged(beacons sent)

JohnnyMirza commented 2 years ago

Yes thats a great point. I ran the script approx. 4 hours ago which was still the 2022-03-09 and I am assuming the script ran the whole 24 hours, ( or approx. 22 hours when I ran the script.) I have to dig through it in more detail but I am guessing it's using the timestamp of the log events e.g. $datetime = $fields[0] . " " . $fields[1];

I will also comment back once I have one more full day to confirm.

Ay0hCrypto commented 2 years ago

Total Witnessed: = 29 |-- Sending: = 29 |-- Times Retried: = 48 Successful: = 26 ( 90%) Resends: = 51 Send or Re-send Failed: = 4 ( 8%) 12 hours on 80/20/600000.

DimitrisSar commented 2 years ago

General Witnesses Overview

Total witnesses = 861 (31.02/hour) Succesfully delivered = 682 (79.21%) Failed = 179 (20.79%) ├── Max retry = 178 (20.67%) └── Crash/reboot = 1 (0.12%)

Max Retry Failure Reasons

Timeout = 121 (14.05%) Not Found = 49 (5.69%) Other challenger issues = 7 (0.81%)

Challengers

Not Relayed = 506 (58.77%) Relayed = 222 (25.78%) Unknown (Probably Not Relayed) = 133 (15.45%) 

 {peerbook_update_interval, 900000},
 {max_inbound_connections, 200},
 {outbound_gossip_connections, 50}

Controllino v1 (https://github.com/DimitrisSar/Controllino#p2p)

What worries me is the SD Card wear and the huge volume of traffic:

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.16.13.131 netmask 255.255.255.0 broadcast 172.16.13.255 inet6 fc00::d523:1e:d3ae:2c45 prefixlen 64 scopeid 0x0 inet6 fe80::ffd3:a541:a680:89ae prefixlen 64 scopeid 0x20 ether xx:xx:xx:xx:xx:xx txqueuelen 1000 (Ethernet) RX packets 16565248 bytes 4531801453 (4.2 GiB) RX errors 0 dropped 0 overruns 0 frame 2343 TX packets 38984819 bytes 51459591599 (47.9 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

uptime: 2 days, 12:13 (miner was not running all the time. I was doing some modifications. I believe that the stats above are for a ~30 hour period)

Ay0hCrypto commented 2 years ago

almost 24 hours 80/20/600000: 24hours was statistically the same 89.9% success 5.3% max_retry 4.8% other or not_found Total Witnessed: = 71 |-- Sending: = 71 |-- Times Retried: = 140 Successful: = 64 ( 90%) Resends: = 139 Max Retry: = 7 ( 5%) Other (Witness Failures): = 53
Challenger Issues: |-- Challenger Not Found: = 95 (7%) |-- Challenger Timed Out: = 28 |-- Challenger Refused Connection: = 0 |-- Challenger Unreachable: = 0 |-- Challenger No Listening Address: = 0 Total Peer Activity: = 342 |-- Timeouts: = 85 |-- Proxy Session Timeouts: = 2 |-- Relay Session Timeouts: = 47 |-- Normal Exit: = 90 |-- Not Found: = 128 |-- Server Down: = 6 |-- Failed to Dial Proxy: = 20 |-- Connection Refused: = 9 |-- Connection Closed: = 24 Once it got above 25 witnesses to calculate the % stayed very close to 90%. I've also been monitoring the logs to look for what other effects this has on the system. It didn't really matter the settings I chose, once it was above 30/10, the PIDS appears to delete 11 inbound at a time, and refill those over time 1 at a time. then deletes 11 more. gossiping out returns this:image At no point does that number exceed 5001. 135 was the lowest. image The peerbook was second most active, but varied in how many were deleted at a time, still one added at a time. least active was Seed pids: image it only ever holds 2, image

Ay0hCrypto commented 2 years ago

Comparing everyone's results and in/out/interval : Most machines to perform better on numbers = or >100. While some where able to reach 200/100/180000, others who went above in+out=100 saw 10% less success in successfully sending witness reports. lowering the interval below 900000 appears to shift failures from not_found to timed, out but has not yet shown any clear evidence of affecting the outcome compared to alterations to inbound/outbound. @microtransactions I checked logs on the proxy errors, seems to happen after vm or deamon is restarted or reboots, and resolves itself over time, the higher the variables the longer it takes to resolve the errors and more likely they are seen popping up from time to time. Also worth note, that most the changes are irrelevant come light hotspot software. any contrary data? if not going to call in+out=100 the most effective range, check interval, and move on to other variables that might be more beneficial in the future.

DimitrisSar commented 2 years ago 
 {peerbook_update_interval, 300000},
 {max_inbound_connections, 100},
 {outbound_gossip_connections, 50}

General Witnesses Overview

Total witnesses = 374 (34.5/hour) Succesfully delivered = 291 (77.81%) Failed = 83 (22.19%) ├── Max retry = 82 (21.93%) └── Crash/reboot = 1 (0.27%)

Max Retry Failure Reasons

Timeout = 65 (17.38%) Not Found = 10 (2.67%) Other challenger issues = 6 (1.6%)

Challengers

Not Relayed = 197 (52.67%) Relayed = 99 (26.47%) Unknown (Probably Not Relayed) = 78 (20.86%)

10 hours 58 minutes RX packets 4525716 bytes 1116214952 (1.0 GiB) TX packets 11182937 bytes 15133476586 (14.0 GiB)

masterconqueror commented 2 years ago

tip: when your peerbook size reached 100k or 200k, ( mine was near 320k with monthly 1tb upload ) close port 44158. try your devices performance on relayed mode. peerbook size will start decreasing but witness count will be amazing.

i tried in this way, get %50 more witnesses. but in time, it will returned to normal. now, i am trying again. i opened port 44158. and waiting for peerbook size get bigger.

adrianformunda commented 2 years ago

What settings are you using?

joi, 10 mar. 2022, 22:34 masterconqueror @.***> a scris:

tip: when your peerbook size reached 100k or 200k, ( mine was near 320k with monthly 1tb upload ) close port 44158. try your devices performance on relayed mode. peerbook size will start decreasing but witness count will be amazing.

i tried in this way, get %50 more witnesses. but in time, it will returned to normal. now, i am trying again. i opened port 44158. and waiting for peerbook size get bigger.

— Reply to this email directly, view it on GitHub https://github.com/helium/miner/issues/1407#issuecomment-1064479459, or unsubscribe https://github.com/notifications/unsubscribe-auth/AXJSHBCVXRNJLID2F4XGINTU7JMG7ANCNFSM5NEOALNA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

masterconqueror commented 2 years ago

What settings are you using? joi, 10 mar. 2022, 22:34 masterconqueror @.> a scris: … tip: when your peerbook size reached 100k or 200k, ( mine was near 320k with monthly 1tb upload ) close port 44158. try your devices performance on relayed mode. peerbook size will start decreasing but witness count will be amazing. i tried in this way, get %50 more witnesses. but in time, it will returned to normal. now, i am trying again. i opened port 44158. and waiting for peerbook size get bigger. — Reply to this email directly, view it on GitHub <#1407 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AXJSHBCVXRNJLID2F4XGINTU7JMG7ANCNFSM5NEOALNA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you commented.Message ID: @.>

200 / 100 / 900000 sensecap 4gb with 100mbit dl / 8mbit ul internet connection

adrianformunda commented 2 years ago

Thanks

Did you notice how fast The peer book decreases?

joi, 10 mar. 2022, 23:14 masterconqueror @.***> a scris:

What settings are you using? joi, 10 mar. 2022, 22:34 masterconqueror @.

> a scris: … <#m1653081381207251500> tip: when your peerbook size reached 100k or 200k, ( mine was near 320k with monthly 1tb upload ) close port 44158. try your devices performance on relayed mode. peerbook size will start decreasing but witness count will be amazing. i tried in this way, get %50 more witnesses. but in time, it will returned to normal. now, i am trying again. i opened port 44158. and waiting for peerbook size get bigger. — Reply to this email directly, view it on GitHub <#1407 (comment) https://github.com/helium/miner/issues/1407#issuecomment-1064479459>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AXJSHBCVXRNJLID2F4XGINTU7JMG7ANCNFSM5NEOALNA https://github.com/notifications/unsubscribe-auth/AXJSHBCVXRNJLID2F4XGINTU7JMG7ANCNFSM5NEOALNA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you commented.Message ID: @.>

200 / 100 / 900000 sensecap 4gb with 100mbit dl / 8mbit ul internet connection

— Reply to this email directly, view it on GitHub https://github.com/helium/miner/issues/1407#issuecomment-1064510483, or unsubscribe https://github.com/notifications/unsubscribe-auth/AXJSHBAAJ5QAKHFPL34QOKTU7JQ3LANCNFSM5NEOALNA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

Ay0hCrypto commented 2 years ago

@DimitrisSar your results + some others, do seem to point to a lower interval leading to less not_found. 14/14 witness reports sent successfully in the last 4 hours, 0 making it on chain. 00:00UTC adjusted to 80/20/300000 - to see what results occur so far as not_found vs max retry vs other failures

florian-asche commented 2 years ago

@DimitrisSar your results + some others, do seem to point to a lower interval leading to less not_found. will be adjusting to 80/20/300000 - to see what results occur so far as not_found vs max retry vs other failures

less not_found at the first try whould be great. It seems that many retry work well for me, but still missing at the end.

florian-asche commented 2 years ago

Can anyone give me some more information about how you test? Just change the settings and restart the miner container? How long does it take that i see something and how long do you guys test?

Ay0hCrypto commented 2 years ago

i was testing for about 8 hours at first, but that wasn't long enough, so been going 24-30 hours now. I use source-code's HMC analyzer. there is also a another analyzer script linked near the top. both work great. I'm still working on some of the formula's for hmc analyzer, %'s are still a little off after the first 3-4 rows of output but the values are solid enough to gauge the changes.

Can anyone give me some more information about how you test? Just change the settings and restart the miner container? How long does it take that i see something and how long do you guys test?

adrianformunda commented 2 years ago

Thank you

Did you notice how fast peerbook size decreased ?

joi, 10 mar. 2022, 23:14 masterconqueror @.***> a scris:

What settings are you using? joi, 10 mar. 2022, 22:34 masterconqueror @.

> a scris: … <#m_-2950767168315651692_m1653081381207251500> tip: when your peerbook size reached 100k or 200k, ( mine was near 320k with monthly 1tb upload ) close port 44158. try your devices performance on relayed mode. peerbook size will start decreasing but witness count will be amazing. i tried in this way, get %50 more witnesses. but in time, it will returned to normal. now, i am trying again. i opened port 44158. and waiting for peerbook size get bigger. — Reply to this email directly, view it on GitHub <#1407 (comment) https://github.com/helium/miner/issues/1407#issuecomment-1064479459>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AXJSHBCVXRNJLID2F4XGINTU7JMG7ANCNFSM5NEOALNA https://github.com/notifications/unsubscribe-auth/AXJSHBCVXRNJLID2F4XGINTU7JMG7ANCNFSM5NEOALNA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you commented.Message ID: @.>

200 / 100 / 900000 sensecap 4gb with 100mbit dl / 8mbit ul internet connection

— Reply to this email directly, view it on GitHub https://github.com/helium/miner/issues/1407#issuecomment-1064510483, or unsubscribe https://github.com/notifications/unsubscribe-auth/AXJSHBAAJ5QAKHFPL34QOKTU7JQ3LANCNFSM5NEOALNA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

Bfindlay commented 2 years ago

I have been running my config at 100/75/900000 for over 48hrs. Sitting at around 86% success per day and my peerbook is at 200k. I might leave it at this for a while since i noticed i did not perform well with other values

General Witnesses Overview

Total witnesses = 122 Succesfully delivered = 106 (86.89%) Failed = 16 (13.11%) ├── Max retry = 11 (9.02%) └── Crash/reboot = 5 (4.1%)

Max Retry Failure Reasons

Timeout = 7 (5.74%) Not Found = 2 (1.64%) Other challenger issues = 1 (0.82%)

Challengers

Not Relayed = 108 (88.52%) Relayed = 10 (8.2%) Unknown Relay Status = 4 (3.28%)