Validator isNewAttSlashable panic

[tsudmi]

🐞 Bug Report

Description

Validator client panics on isNewAttSlashable.

🔥 Error

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1eef9a1]

goroutine 683200 [running]:
github.com/prysmaticlabs/prysm/validator/client/polling.isNewAttSlashable(0x0, 0xfca, 0xfcb, 0x217afa0)
        validator/client/polling/attest.go:268 +0x41
github.com/prysmaticlabs/prysm/validator/client/polling.(*validator).preSigningValidations(0xc000706000, 0x879da0, 0xc000792120, 0xc000a33d40, 0x17965fa2490c0799, 0x9cf7d5e901f27000, 0xe1b0a6c2366ad6a8, 0x9b04c82d76edaca5, 0x4b0c0ddb1d73323d, 0x730b352b17995937, ...)
        validator/client/polling/attest.go:152 +0x46a
github.com/prysmaticlabs/prysm/validator/client/polling.(*validator).SubmitAttestation(0xc000706000, 0x879da0, 0xc000792120, 0x1f967, 0x17965fa2490c0799, 0x9cf7d5e901f27000, 0xe1b0a6c2366ad6a8, 0x9b04c82d76edaca5, 0x4b0c0ddb1d73323d, 0x730b352b17995937)
        validator/client/polling/attest.go:69 +0xc4f
github.com/prysmaticlabs/prysm/validator/client/polling.run.func1(0xc000243430, 0x88f520, 0xc000706000, 0x879d60, 0xc000a33740, 0x1f967, 0xc0000bb960, 0x965fa2490c079901, 0xf7d5e901f2700017, 0xb0a6c2366ad6a89c, ...)
        validator/client/polling/runner.go:126 +0x1ea
created by github.com/prysmaticlabs/prysm/validator/client/polling.run
        validator/client/polling/runner.go:122 +0xd56

🌍 Your Environment

What version of Prysm are you running? (Which release)

  
  Prysm/v1.0.0-alpha.13/6c7131cb544413e1ca73aeb18e6ff5b9acc813b7
  

[prestonvanloon]

Looks like attesterHistoryByPubKey is returning nil and we aren't handling it properly. https://github.com/prysmaticlabs/prysm/blob/6c7131cb544413e1ca73aeb18e6ff5b9acc813b7/validator/client/polling/attest.go#L150

@0xKiwi Is this your feature domain? Can you take a look please?

[0xKiwi]

Sure thing @prestonvanloon, will look into this today.

[tsudmi]

Hi @0xKiwi , any thoughts about why it happens? My validators are getting stopped quite frequently. 😞

[0xKiwi]

Hi @0xKiwi , any thoughts about why it happens? My validators are getting stopped quite frequently. 😞

Hello, are you running with the validator protection flags? If so, try disabling them until the next release.

[tsudmi]

Hi @0xKiwi , any thoughts about why it happens? My validators are getting stopped quite frequently. 😞

Hello, are you running with the validator protection flags? If so, try disabling them until the next release.

Yes, I use protection flags. Will disable them for the time being. Thanks!

[stefa2k]

Maybe related to #6519

[stefa2k]

Got the same thing now, running HEAD-c26151 Logs: validator.txt

[prestonvanloon]

I recommend we check there is something in the map.

diff --git a/validator/client/attest_protect.go b/validator/client/attest_protect.go
index d196c5dc3..8a49324a8 100644
--- a/validator/client/attest_protect.go
+++ b/validator/client/attest_protect.go
@@ -19,8 +19,11 @@ func (v *validator) preAttSignValidations(ctx context.Context, indexedAtt *ethpb
        fmtKey := fmt.Sprintf("%#x", pubKey[:])
        if featureconfig.Get().LocalProtection {
                v.attesterHistoryByPubKeyLock.RLock()
-               attesterHistory := v.attesterHistoryByPubKey[pubKey]
+               attesterHistory, ok := v.attesterHistoryByPubKey[pubKey]
                v.attesterHistoryByPubKeyLock.RUnlock()
+               if !ok {
+                       return nil
+               }
                if isNewAttSlashable(attesterHistory, indexedAtt.Data.Source.Epoch, indexedAtt.Data.Target.Epoch) {
                        if v.emitAccountMetrics {
                                ValidatorAttestFailVec.WithLabelValues(fmtKey).Inc()

[terencechain]

Reported by users again:

[2020-08-04 20:37:24]  WARN roughtime: Roughtime reports your clock is off by more than 2 seconds offset=4h0m0.345467424s
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1186808]

goroutine 15805 [running]:
github.com/prysmaticlabs/prysm/validator/client.isNewAttSlashable(0x0, 0x46, 0x47, 0x2537140)
        validator/client/attest_protect.go:71 +0x38
github.com/prysmaticlabs/prysm/validator/client.(*validator).preAttSignValidations(0xc0001b69a0, 0x195a5e0, 0xc000e42f90, 0xc000c434a0, 0x8f8c0ccd5767e396, 0x5e0512bc5770e63e, 0xfd19de8745935aa5, 0x5fd2a4fd4057a3b8, 0x93d276a3aaf6471e, 0x68704d8ed0b13732, ...)
        validator/client/attest_protect.go:24 +0x1d9
github.com/prysmaticlabs/prysm/validator/client.(*validator).SubmitAttestation(0xc0001b69a0, 0x195a5e0, 0xc000e42f90, 0x8ef, 0x8f8c0ccd5767e396, 0x5e0512bc5770e63e, 0xfd19de8745935aa5, 0x5fd2a4fd4057a3b8, 0x93d276a3aaf6471e, 0x68704d8ed0b13732)
        validator/client/attest.go:69 +0xc59
github.com/prysmaticlabs/prysm/validator/client.run.func1(0xc00001df30, 0x1971820, 0xc0001b69a0, 0x195a5a0, 0xc000c42b40, 0x8ef, 0xc0008070a0, 0x8c0ccd5767e39601, 0x512bc5770e63e8f, 0x19de8745935aa55e, ...)
        validator/client/runner.go:133 +0x1ea
created by github.com/prysmaticlabs/prysm/validator/client.run
        validator/client/runner.go:129 +0xd76 

d33pz on Discord. I'm bumping this to critical

[prestonvanloon]

As a temporary workaround, please set --enable-local-protection=false.