kaloyan-raev
commented
4 years ago We started working on an improvement in the Windows installer to make the storagenode and storagenode-updater services dependent on the dnscache service.
https://review.dev.storj.io/c/storj/storj/+/819
Meanwhile, you should be able to do this manually using the following commands:
sc config storagenode depend= dnscache
sc config storagenode-updater depend= dnscacheThis should be a better workaround than the restarts.
MaxMaximus
profclems
AlexeyALeonov
Hello.
After introducing preflight clock checks node startup fails after a single DNS error. If DNS is unavailable for any reasons at node startup and storagenode fails at resolving host names of satellites into IPs it does not retry and just terminate loading. Resulting in node going offline for a long period of time until owner manual intervention.
I see this situation almost at each of my computers boot/reboot. Looks like storagenode service loads faster and perform first DNS requests while system DNS resolver is not ready yet. And node fail startup. It it is not a big problem by itself as window will try to start services after failure 2 times by default (3 tries total, with 1 minute delay each) and DNS will be available at time of the second try. But current installation package disable this option. So after initial failure service is NOT restarted and node remains offline.
I saw this situation today on my node: there was short power outage (~10 min), UPS trigger correct system shutdown and system restart after power back online, all programs resume running successfully except for Storj due to single DNS error ==> failed preflight check ==> node remained offline for about 12 hours (until i have noticed it and restarted service manually).
I altered storagenode service setting back to windows default (2 additional tries to start after failure) and it fixed problem. But just for me - it still will be a problem for all default setups.
Here example log (i reproduced error few times - just simple correct reboot is enough to trigger it):
If not second try to start service (which a i have added manually) node will be offline after each / almost each system boot.
P.S. I am not sure but DNS delay at OS boot can be caused by a lot of autorun programs on my computers. So it probable will affect only minor % of all nodes (slow or heavy loaded by other programs during boot). But still it is not a normal program behavior when just few dozen seconds of DNS inaccessibility cause the fatal error. Also it probably not isolated to Window only. Can affect other OS too, but I did not test on other OS.