multinet automated testing - validator crash

[sinkingsugar]

Follow up on #7014 discussion with @nisdas

I manage to progress but ran into this issue with the validator:

prysm_1       | time="2020-08-26 03:11:32" level=error msg="gRPC panicked!" error="runtime error: invalid memory address or nil pointer dereference" stack="goroutine 703 [running]:

I would appreciate if you could take a peek!, full log here: multinet.txt

P.S. The current multinet repo I'm working on is here (bit out of date as WIP): https://github.com/sinkingsugar/multinet

Command lines used:

bazel run //beacon-chain -- \
  $BOOTNODES_ARG \
  --force-clear-db \
  --datadir=/tmp/beacon-prysm \
  --pprof \
  --verbosity=debug \
  --interop-eth1data-votes \
  --chain-config-file=$TESTNET_DIR/config.yaml \
  --contract-deployment-block=0 \
  --deposit-contract=0x8A04d14125D0FDCDc742F4A05C051De07232EDa4 \
  --interop-genesis-state=$TESTNET_DIR/genesis.ssz &

bazel run //validator -- \
  --chain-config-file=$TESTNET_DIR/config.yaml \
  --disable-accounts-v2=true \
  --verbosity=trace \
  --password="" \
  --keymanager=wallet \
  --keymanageropts=$PRY_DATADIR/prysm/keymanager_opts.json

[nisdas]

Hey thanks for posting this issue, unfortunately this looks like an issue with the validator not existing in the state currently. In the wallet, are all keys of those that have already been included in the genesis state ?

[sinkingsugar]

Thanks for answering! Interesting, I am actually generating deposits using @protolambda tools ( https://github.com/protolambda/eth2-val-tools.git ) (Proto, maybe you know something about this issue ?) So far to the tool I pass fork-version but that's about it.

Btw nimbus and lighthouse do not have this issue/requirement.

I'm completely skipping POW in those tests btw, no connection to goerli or so.

P.S. I'm mostly a libp2p engineer so my eth2 spec knowledge is complete yet.

[nisdas]

How do you generate the state in this case ? It might be the case where by the tool generates validators that do not exist in the state yet. I can try maybe fixing that so that it doesn't error for you in that case.

[sinkingsugar]

Using an internal tool in nim beacon chain here: https://github.com/status-im/nim-beacon-chain/blob/a84a8ba192e2f4c7bae22470b44b2db12f95ab0f/beacon_chain/beacon_node.nim#L1102 (cc @zah , could you please have look here)

[sinkingsugar]

Yes the tool that generates validators is detached indeed in this case (I'm not using the nim beacon chain generated ones, but rather Proto's tool ones).

[nisdas]

Ok thanks for clarifying this. I will try to get this fixed by today, so that we do not have this strict requirement and the validator can proceed as expected even if it is not in the state.

[sinkingsugar]

Sounds good! cheers!

[nisdas]

@sinkingsugar #7114 will fix your issue, let me know if you have any other issues with your multinet scripts.

[sinkingsugar]

thanks a lot @nisdas , do I have to turn on any particular flag or it's already fine with those I have? never-the-less I will try it now as they are.

[nisdas]

it should be fine as it is. You will not get anymore panics from it

[sinkingsugar]

No panics anymore indeed, now I get a constant spam every x seconds with

prysm_1       | time="2020-08-26 11:39:52" level=info msg="Waiting for deposit to be observed by beacon node" prefix=validator pubKey=0x8cda0b626db5 status="UNKNOWN_STATUS"
prysm_1       | time="2020-08-26 11:39:52" level=info msg="Waiting for deposit to be observed by beacon node" prefix=validator pubKey=0xa610f5fd4a25 status="UNKNOWN_STATUS"
prysm_1       | time="2020-08-26 11:39:52" level=info msg="Waiting for deposit to be observed by beacon node" prefix=validator pubKey=0x90de431010ef status="UNKNOWN_STATUS"

I guess it's awaiting an activation that will never happen, is it safe or that means it won't progress?

[nisdas]

This is fine as long as you expect the provided keys to do nothing. If you expect the provided keys to prysm to be valid and current then that probably means that those keys are not correct in the context of the current state.

[protolambda]

Hey, I might be able to help get a correct beacon-state set up. Can you describe your approach with assigning keys, and building the genesis state? Also, maybe discuss on discord instead?

[sinkingsugar]

I was thinking about you @protolambda cos I am generating the keys with your tool 😄 , discord is good yes! telegram fine too

[nisdas]

I can help test out any interop issues with prysm, @sinkingsugar you can just ping me on discord too, would be faster to communicate through it.