calamenium commented 4 years ago

🐞 Bug Report

Description

Upon upgrading to alpha-29 the validator process will not start and produces an error due to being unable to open browser via xdg-open.

Has this worked before in a previous version?

First release with --web

🔬 Minimal Reproduction

Has been reproduced by myself and a few others noted in discord. May be related to xdg-utils not operating properly within a systemd process. (I followed SomerEsats guide).

🔥 Error


On reinstall with the --web command in the systemd startup for the validator I get the following error: "Oct 12 12:34:38 parti6le-NUC8i5BEH prysm.sh[3163]: /usr/bin/xdg-open: 869: www-browser: not found
Oct 12 12:34:38 parti6le-NUC8i5BEH prysm.sh[3163]: /usr/bin/xdg-open: 869: links2: not found
Oct 12 12:34:38 parti6le-NUC8i5BEH prysm.sh[3163]: /usr/bin/xdg-open: 869: elinks: not found
Oct 12 12:34:38 parti6le-NUC8i5BEH prysm.sh[3163]: /usr/bin/xdg-open: 869: links: not found
Oct 12 12:34:38 parti6le-NUC8i5BEH prysm.sh[3163]: /usr/bin/xdg-open: 869: lynx: not found
Oct 12 12:34:38 parti6le-NUC8i5BEH prysm.sh[3163]: /usr/bin/xdg-open: 869: w3m: not found
Oct 12 12:34:38 parti6le-NUC8i5BEH prysm.sh[3163]: xdg-open: no method available for opening 'http://127.0.0.1:4242/'
Oct 12 12:34:38 parti6le-NUC8i5BEH prysm.sh[3116]: time="2020-10-12 12:34:38" level=error msg="Could not open Prysm web UI in browser" error="exit status

🌍 Your Environment

Operating System: Ubuntu 20.04.1

What version of Prysm are you running? alpha-29

Anything else relevant (validator index / public key)? Everything works if --web removed.

@nisdas Recommended posting this issue via discord.

OisinKyne commented 4 years ago

@terencechain I want to highlight that this issue seems to be more serious than just not being able to use the UI!! I had a similar error and it looks like it made my validator hang entirely due to xdg-open failing. Will detail here but let me know if you want me to reopen this as a fresh issue.

What am I running

Prysm Alpha.29 with docker and yaml config on GCE

What happened?

I was validating on Zinken, and decided to restart my validator with the web: true yaml config to check the UI out.

The shutdown and restart docker logs looked like this:


time="2020-10-13 13:49:01" level=info msg="Attestation schedule" attesters=1 prefix=validator pubKeys=[0xa8e4fce403c1] slot=7774
time="2020-10-13 13:49:29" level=info msg="Submitted new attestations" AggregatorIndices=[] AttesterIndices=[2381] BeaconBlockRoot=0x1fc80b69a76d CommitteeIndex=0 Slot=7746 SourceEpoch=241 SourceRoot=0x0e32ef3ba23c TargetEpoch=242 TargetRoot=0x3598fceb13d7 prefix=validator
time="2020-10-13 13:49:45" level=info msg="Got interrupt, shutting down..." prefix=node
time="2020-10-13 13:49:45" level=info msg="Stopping service" prefix=validator
time="2020-10-13 13:49:45" level=info msg="Stopping Prysm validator" prefix=node
time="2020-10-13 13:49:45" level=warning msg="-- flag is not specified (default: Medalla), this will become required from next release! " prefix=flags
time="2020-10-13 13:49:45" level=info msg="Enabling web portal to manage the validator client" prefix=node
time="2020-10-13 13:49:45" level=info msg="Checking DB" databasePath="/data" prefix=node
time="2020-10-13 13:49:45" level=info msg="Starting validator node" prefix=node version="Prysm/v1.0.0-alpha.28/551b03d6e6506b64ac8957510c9c250197236206. Built at: 2020-10-10 02:12:50+00:00"
time="2020-10-13 13:49:45" level=info msg="Starting Prysm web UI" address="127.0.0.1:4242" prefix=prysm-web
time="2020-10-13 13:49:45" level=info msg="Starting gRPC gateway" address="127.0.0.1:7500" prefix=gateway
time="2020-10-13 13:49:45" level=warning msg="You are using an insecure gRPC connection. If you are running your beacon node and validator on the same machines, you can ignore this message. If you want to know how to enable secure connections, see: https://docs.prylabs.network/docs/prysm-usage/secure-grpc" prefix=validator
time="2020-10-13 13:49:45" level=info msg="gRPC server listening on address" address="127.0.0.1:7000" prefix=rpc
time="2020-10-13 13:49:46" level=error msg="Could not open Prysm web UI in browser" error="exec: "xdg-open": executable file not found in $PATH" prefix=prysm-web

I was able to access the web UI so I thought everything was working normally
My validator immediately started missing attestations https://zinken.beaconcha.in/block/7715 (validator 2383 https://zinken.beaconcha.in/validator/0xa4a5bea067ee96970249c0d4be448fc9f5d44ddbf7ba5c895b10919e2da492f0c6b0b1fc8bc3f6a7c9dc4e46e50f766f#attestations)
Today I noticed my validator was offline, and the timeline exactly matched when I restarted it with the web flag
I restarted the validator with web: false
I got the rest of the logs when the shutdown signal was received by the docker container (Notice almost 24 hours between this log: time="2020-10-13 13:49:46" level=error msg="Could not open Prysm web UI in browser" error="exec: "xdg-open": executable file not found in $PATH" prefix=prysm-web and the shutdown log.)


time="2020-10-13 13:49:01" level=info msg="Attestation schedule" attesters=1 prefix=validator pubKeys=[0xa8e4fce403c1] slot=7774
time="2020-10-13 13:49:29" level=info msg="Submitted new attestations" AggregatorIndices=[] AttesterIndices=[2381] BeaconBlockRoot=0x1fc80b69a76d CommitteeIndex=0 Slot=7746 SourceEpoch=241 SourceRoot=0x0e32ef3ba23c TargetEpoch=242 TargetRoot=0x3598fceb13d7 prefix=validator
time="2020-10-13 13:49:45" level=info msg="Got interrupt, shutting down..." prefix=node
time="2020-10-13 13:49:45" level=info msg="Stopping service" prefix=validator
time="2020-10-13 13:49:45" level=info msg="Stopping Prysm validator" prefix=node
time="2020-10-13 13:49:45" level=warning msg="-- flag is not specified (default: Medalla), this will become required from next release! " prefix=flags
time="2020-10-13 13:49:45" level=info msg="Enabling web portal to manage the validator client" prefix=node
time="2020-10-13 13:49:45" level=info msg="Checking DB" databasePath="/data" prefix=node
time="2020-10-13 13:49:45" level=info msg="Starting validator node" prefix=node version="Prysm/v1.0.0-alpha.28/551b03d6e6506b64ac8957510c9c250197236206. Built at: 2020-10-10 02:12:50+00:00"
time="2020-10-13 13:49:45" level=info msg="Starting Prysm web UI" address="127.0.0.1:4242" prefix=prysm-web
time="2020-10-13 13:49:45" level=info msg="Starting gRPC gateway" address="127.0.0.1:7500" prefix=gateway
time="2020-10-13 13:49:45" level=warning msg="You are using an insecure gRPC connection. If you are running your beacon node and validator on the same machines, you can ignore this message. If you want to know how to enable secure connections, see: https://docs.prylabs.network/docs/prysm-usage/secure-grpc" prefix=validator
time="2020-10-13 13:49:45" level=info msg="gRPC server listening on address" address="127.0.0.1:7000" prefix=rpc
time="2020-10-13 13:49:46" level=error msg="Could not open Prysm web UI in browser" error="exec: "xdg-open": executable file not found in $PATH" prefix=prysm-web
time="2020-10-14 13:13:03" level=info msg="Got interrupt, shutting down..." prefix=node
time="2020-10-14 13:13:03" level=error msg="Failed to start validator web server" error="http: Server closed" prefix=prysm-web
time="2020-10-14 13:13:03" level=info msg="Stopping service" prefix=validator
time="2020-10-14 13:13:03" level=info msg="Stopping Prysm validator" prefix=node
time="2020-10-14 13:13:03" level=fatal msg="Wallet is not ready: context canceled" prefix=validator
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x20114e2]

goroutine 158 [running]:
github.com/prysmaticlabs/prysm/shared/slotutil.(*SlotTicker).Done.func1(0x0)
    shared/slotutil/slotticker.go:37 +0x22
created by github.com/prysmaticlabs/prysm/shared/slotutil.(*SlotTicker).Done
    shared/slotutil/slotticker.go:36 +0x3f
time="2020-10-14 13:13:04" level=warning msg="Running on Zinken Testnet" prefix=flags
time="2020-10-14 13:13:04" level=info msg="Opened validator wallet" keymanager-kind=direct prefix=node wallet="/config/direct"
time="2020-10-14 13:13:04" level=info msg="Checking DB" databasePath="/config/direct" prefix=node
time="2020-10-14 13:13:04" level=info msg="Starting validator node" prefix=node version="Prysm/v1.0.0-alpha.28/551b03d6e6506b64ac8957510c9c250197236206. Built at: 2020-10-10 02:12:50+00:00"
time="2020-10-14 13:13:04" level=warning msg="You are using an insecure gRPC connection. If you are running your beacon node and validator on the same machines, you can ignore this message. If you want to know how to enable secure connections, see: https://docs.prylabs.network/docs/prysm-usage/secure-grpc" prefix=validator
time="2020-10-14 13:13:04" level=info msg="Waiting for beacon chain start log from the ETH 1.0 deposit contract" prefix=validator
time="2020-10-14 13:13:04" level=info msg="Validating for public key" prefix=validator publicKey=0x98350cc6d98b
time="2020-10-14 13:13:04" level=info msg="Validating for public key" prefix=validator publicKey=0xa8e4fce403c1
time="2020-10-14 13:13:04" level=info msg="Validating for public key" prefix=validator publicKey=0xb6c9c1b00f35

After restart without web the validator went back to attesting as expected. https://zinken.beaconcha.in/block/14784

CC @prestonvanloon

rauljordan commented 4 years ago

Hi @calamenium, thanks for bringing this up. This one is a duplicate of https://github.com/prysmaticlabs/prysm-web-ui/issues/72, please subscribe to that issue to be notified when a fix is available. We will resolve this by next release

prysmaticlabs / prysm

Could not use validator client with web UI on alpha-29 #7502