Lokinet router failing to communicate after service node re-registration

[jagerman]

A couple of times now I've had a service node get decommissioned due to Lokinet unreachability after an unlock-and-reregister. Connectivitity seems fine after a restart of lokinet.

Timeline:

Lokinet-router is running fine as a registered node
Registration expires
SN gets re-registered (without any restarts)

Up to the re-registration we have, every 30 seconds, the expected whining:

Sep 26 19:46:07 skoll lokinet-router-27[1049811]: [ERR] [](53) 2022-09-26 22:46:07.424 GMT [+621h18m16.693s]
    ../llarp/router/router.cpp:1027        We are running as a service node but we seem to be deregistered

which is fine, we aren't a registered service node. These stop after the re-registration, there are no log statements at all for the next 5.5 hours, then we got deregistered because of failing lokinet connectivity. Other nodes at this time were reporting that Lokinet is unreachable; there are no log statements (logging is at warning level) for 5.5 hours, then a decomm, a recomm, and then a dereg.

Lokinet appears to track the decomm/recomm states fine, but isn't reachable by anyone else on the network until it gets restarted.

[jagerman]

On further investigation it appears that this node has an outdated bootstrap file. So the connectivity issue makes sense, but there really needs to be a serious warning when the nodedb is close to empty and we are active.

[ianmacd]

Lokinet appears to track the decomm/recomm states fine, but isn't reachable by anyone else on the network until it gets restarted.

Yes, I narrowly escaped a deregistration once, due to this bug. I also know a couple of other ops who weren't so lucky.

Now, when I bring up a shared node and I don't know when exactly it will finally go live on the network, I insulate myself from this bug with the following job in /etc/crontab.

*/15 * * * * root systemctl restart lokinet-router

[jagerman]

TODO:

• identify invalid outdated bootstrap
• add system default bootstrap config option
• ignore the known-invalid old bootstrap file (the one in question here was a little over two years old, and hasn't been valid for a long time).
• update debs to install and use a system default bootstrap if no (valid) bootstrap is otherwise configure that can get upgraded with the package (by not being in the config directory).
• add "error" level warning if we are in a state where we are an active service node but have too few known routers (e.g. less than 5).

[majestrate]

did this get fixed?