search

rocket-pool / smartnode

The CLI package for Rocket Pool smart nodes.
GNU General Public License v3.0
148 stars 112 forks source link

rocketpool-pow-proxy panics every few minutes #137

Closed dodrian closed 3 years ago

dodrian commented 3 years ago

I upgraded to 1.0.0-beta.3 per instructions, bounced service, and everything appeared to be fine (1 successful attestation on each validator, though I may not have watched long enough for this issue to manifest), then I updated config.yml to use image: statusim/nimbus-eth2:amd64-v1.2.0 and bounced service. After seeing (intermittent) failed attestations, apparently because of eth1 client continually restarting, I reverted config.yml and bounced again. Still seeing issues I downgraded back to 1.0.0-beta.2 and bounced, but continued seeing eth1 errors. Running rocketpool service config and choosing infura appears to have resolved the errors (but is no longer using local network node).

No errors were seen in the logs on the separate machine running geth 1.10.1. Its IP address is statically assigned and remains correct.

Sample setting files: config.yml settings.yml (changed CUSTOM_GRAFFITI line to not dox node, DM on discord if needed)

Sample eth1 log errors:

eth1_1        | 2021/04/19 20:42:21 New POST request received from 172.19.0.6:58054
eth1_1        | 2021/04/19 20:42:21 Response sent to 172.19.0.6:58054 successfully
eth1_1        | 2021/04/19 20:42:21 New POST request received from 172.19.0.6:58054
eth1_1        | 2021/04/19 20:42:21 Response sent to 172.19.0.6:58054 successfully
eth1_1        | 2021/04/19 20:42:24 Error connecting to Infura: malformed ws or wss URL
eth1_1        | 2021/04/19 20:42:24 http: response.Write on hijacked connection from fmt.Fprintln (print.go:265)
eth1_1        | panic: runtime error: invalid memory address or nil pointer dereference
eth1_1        | [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x786f97]
eth1_1        | 
eth1_1        | goroutine 51 [running]:
eth1_1        | github.com/gorilla/websocket.(*Conn).WriteMessage(0x0, 0x1, 0xc000354000, 0x49, 0x200, 0x0, 0x0)
eth1_1        |     /go/pkg/mod/github.com/gorilla/websocket@v1.4.2/conn.go:753 +0x37
eth1_1        | github.com/rocket-pool/smartnode/rocketpool-pow-proxy/proxy.(*WsProxyServer).ServeHTTP.func2(0xc0003c0b00, 0x8c8e60, 0xc0002f0540, 0x0, 0xc000026550)
eth1_1        |     /src/rocketpool-pow-proxy/proxy/ws-proxy.go:94 +0x59
eth1_1        | created by github.com/rocket-pool/smartnode/rocketpool-pow-proxy/proxy.(*WsProxyServer).ServeHTTP
eth1_1        |     /src/rocketpool-pow-proxy/proxy/ws-proxy.go:80 +0x55a
[EOF]
dodrian commented 3 years ago

I have gone through the process again of upgrading to beta.3


Rocket Pool client version: 1.0.0-beta.3
Rocket Pool service version: 1.0.0-beta.3
Selected Eth 1.0 client: Infura (rocketpool/smartnode-pow-proxy:v1.0.0-beta.0)
Selected Eth 2.0 client: Nimbus (statusim/nimbus-eth2:amd64-v1.1.0)```

Running with Infura and do not see any issues.  But running `rocketpool service config` and switching to my local network client (http://192.168.0.247:8545) I start to see the errors again.
dodrian commented 3 years ago

At the begining of the eth1 startup logs I see

eth1_1        | 2021/04/20 18:06:03 Proxy server listening on port 8546
eth1_1        | 2021/04/20 18:06:03 Proxy server listening on port 8545

But I don't understand where the 8546 proxy server is being started from.

dodrian commented 3 years ago

Proof of concept fix: https://github.com/dodrian/smartnode/commit/11a7ff5edecf0c7ad0b33d41a9105bb2a66d2b0d#diff-f614cae8758ae827e657079e8c95415e68686770491d3ccee396e40669aeb28b

dodrian commented 3 years ago

Fix for install (note: image line in config.yml still needs to be updated) https://github.com/dodrian/smartnode-install/commit/cfb6453e36cecf3348ff545b1e7cd6aaf8b553f1

jclapis commented 3 years ago

@dodrian helped me test #139 and rocket-pool/smartnode-install#47, and confirmed that they fixed this issue for him.