Nodetracker stuck in connect to TS servers.

minzak commented 4 years ago

I got many nodes, and I found some weirds, some time nodetracker stuck to connect with TS server. Need additional checks to resolve it and exit or else, to possible restart node. In my case, if nodetracker will exit - docker restart it and all will be fine. And this behaviur i saw near 20 times in 2000 nodes for last month

2020-10-07 08:46:51 UTC [ndt] No connection to server ts2.eu.
2020-10-07 08:47:06 UTC [ndt] No connection to server ts2.eu.
2020-10-07 08:47:21 UTC [ndt] No connection to server ts2.eu.
2020-10-07 08:47:29 UTC [zend] keypool reserve 3
2020-10-07 08:47:29 UTC [zend] keypool return 3
2020-10-07 08:47:36 UTC [ndt] No connection to server ts2.eu.
2020-10-07 08:47:37 UTC [ndt] Reconnect websocket error
2020-10-07 08:47:40 UTC [ndt] Trying server: ts1.na
2020-10-07 08:47:51 UTC [ndt] No connection to server ts1.na.
2020-10-07 08:48:06 UTC [ndt] No connection to server ts1.na.
2020-10-07 08:48:21 UTC [ndt] No connection to server ts1.na.
2020-10-07 08:48:29 UTC [zend] keypool reserve 3
2020-10-07 08:48:29 UTC [zend] keypool return 3
2020-10-07 08:48:32 UTC [zend] UpdateTip: new best=000000000da0709da929786190fb169dac6ec63e5b3676493cd8f964e0a9ba28  height=808050  log2_work=54.716694  tx=17378149  date=2020-10-07 08:48:31 progress=1.000000  cache=1.5MiB(148tx)
2020-10-07 08:48:36 UTC [ndt] No connection to server ts1.na.
2020-10-07 08:48:51 UTC [ndt] Trying server: ts3.eu
2020-10-07 08:48:51 UTC [ndt] No connection to server ts3.eu.
2020-10-07 08:49:06 UTC [ndt] No connection to server ts3.eu.
2020-10-07 08:49:21 UTC [ndt] No connection to server ts3.eu.
2020-10-07 08:49:29 UTC [zend] keypool reserve 3
2020-10-07 08:49:29 UTC [zend] keypool return 3
2020-10-07 08:49:36 UTC [ndt] No connection to server ts3.eu.
2020-10-07 08:49:41 UTC [zend] ERROR: SSL_read error:00000005:lib(0):func(0):DH lib
2020-10-07 08:49:51 UTC [ndt] No connection to server ts3.eu.
2020-10-07 08:49:54 UTC [ndt] Reconnect websocket error
2020-10-07 08:50:02 UTC [ndt] Trying server: ts4.eu
2020-10-07 08:50:06 UTC [ndt] No connection to server ts4.eu.
2020-10-07 08:50:21 UTC [ndt] No connection to server ts4.eu.
2020-10-07 08:50:30 UTC [zend] keypool reserve 3
2020-10-07 08:50:30 UTC [zend] keypool return 3
2020-10-07 08:50:36 UTC [ndt] No connection to server ts4.eu.
2020-10-07 08:50:51 UTC [ndt] No connection to server ts4.eu.
2020-10-07 08:50:52 UTC [ndt] Reconnect websocket error
2020-10-07 08:51:06 UTC [ndt] No connection to server ts4.eu.
2020-10-07 08:51:13 UTC [ndt] Trying server: ts6.eu
2020-10-07 08:51:21 UTC [ndt] No connection to server ts6.eu.
2020-10-07 08:51:30 UTC [zend] keypool reserve 3
2020-10-07 08:51:30 UTC [zend] keypool return 3
2020-10-07 08:51:36 UTC [ndt] No connection to server ts6.eu.
2020-10-07 08:51:51 UTC [ndt] No connection to server ts6.eu.
2020-10-07 08:52:03 UTC [ndt] Reconnect websocket error
2020-10-07 08:52:06 UTC [ndt] No connection to server ts6.eu.
2020-10-07 08:52:21 UTC [ndt] No connection to server ts6.eu.
2020-10-07 08:52:24 UTC [ndt] Trying server: ts2.na
2020-10-07 08:52:30 UTC [zend] keypool reserve 3
2020-10-07 08:52:30 UTC [zend] keypool return 3
2020-10-07 08:52:36 UTC [ndt] No connection to server ts2.na.
2020-10-07 08:52:51 UTC [ndt] No connection to server ts2.na.
2020-10-07 08:53:06 UTC [ndt] No connection to server ts2.na.
2020-10-07 08:53:22 UTC [ndt] No connection to server ts2.na.
2020-10-07 08:53:30 UTC [zend] keypool reserve 3
2020-10-07 08:53:30 UTC [zend] keypool return 3
2020-10-07 08:53:33 UTC [ndt] Reconnect websocket error
2020-10-07 08:53:35 UTC [ndt] Trying server: ts5.eu
2020-10-07 08:53:37 UTC [ndt] No connection to server ts5.eu.
2020-10-07 08:53:52 UTC [ndt] No connection to server ts5.eu.
2020-10-07 08:54:07 UTC [ndt] No connection to server ts5.eu.
2020-10-07 08:54:22 UTC [ndt] No connection to server ts5.eu.
2020-10-07 08:54:25 UTC [ndt] Reconnect websocket error
2020-10-07 08:54:31 UTC [zend] keypool reserve 3
2020-10-07 08:54:31 UTC [zend] keypool return 3
2020-10-07 08:54:37 UTC [ndt] No connection to server ts5.eu.
2020-10-07 08:54:46 UTC [ndt] Trying server: ts3.na
2020-10-07 08:54:52 UTC [ndt] No connection to server ts3.na.
2020-10-07 08:55:07 UTC [ndt] No connection to server ts3.na.
2020-10-07 08:55:22 UTC [ndt] No connection to server ts3.na.
2020-10-07 08:55:31 UTC [zend] keypool reserve 3
2020-10-07 08:55:31 UTC [zend] keypool return 3
2020-10-07 08:55:37 UTC [ndt] No connection to server ts3.na.
2020-10-07 08:55:49 UTC [ndt] Reconnect websocket error
2020-10-07 08:55:52 UTC [ndt] No connection to server ts3.na.
2020-10-07 08:55:57 UTC [ndt] Trying server: ts4.na
2020-10-07 08:56:07 UTC [ndt] No connection to server ts4.na.
2020-10-07 08:56:22 UTC [ndt] No connection to server ts4.na.
2020-10-07 08:56:31 UTC [zend] keypool reserve 3
2020-10-07 08:56:31 UTC [zend] keypool return 3
2020-10-07 08:56:37 UTC [ndt] No connection to server ts4.na.
2020-10-07 08:56:52 UTC [ndt] No connection to server ts4.na.
2020-10-07 08:57:07 UTC [ndt] No connection to server ts4.na.
2020-10-07 08:57:08 UTC [ndt] Trying server: ts1.eu
2020-10-07 08:57:22 UTC [ndt] No connection to server ts1.eu.
2020-10-07 08:57:31 UTC [zend] keypool reserve 3
2020-10-07 08:57:31 UTC [zend] keypool return 3
2020-10-07 08:57:37 UTC [ndt] No connection to server ts1.eu.
2020-10-07 08:57:52 UTC [ndt] No connection to server ts1.eu.
2020-10-07 08:58:07 UTC [ndt] No connection to server ts1.eu.
2020-10-07 08:58:19 UTC [ndt] Trying server: ts2.eu
2020-10-07 08:58:22 UTC [ndt] No connection to server ts2.eu.

JMF-Networks commented 2 years ago

I have same problem, any idea how to fix?

minzak commented 2 years ago

No fix, only reload node. ( The same behavior is present in v3.0.3, but a little less. Also, I have one idea how to fix it, but need Js developer, and make a patch of ndt.

ADumaine commented 2 years ago

Two ways to reconnect:

Stop the nodetracker. Wait 20 seconds (in case the server has to drop a partially initialized socket connection). Restart the node tracker.

Or

Stop the nodetracker. Change the home server in the nodetracker's config.json file to another server (others are listed in the file). Restart the nodetracker.

The zen node itself does not need to be restarted.

ADumaine commented 1 year ago

This issue has been left open to make it easier to find the steps to change home server above.

HorizenOfficial / nodetracker

Nodetracker stuck in connect to TS servers. #101