Nil block elected

[acroca]

I'm running a validator and when I run validator.start(), after a few blocks I get this error:

INFO [09-28|10:15:33.929] Pre vote sub-election 
DEBUG[09-28|10:15:33.929] Proposal's block is nil, voting nil 
ERROR[09-28|10:15:33.930] a duplicate vote in voting table 0; blockHash [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]; voteHash [194 255 190 143 34 21 135 141 77 239 207 67 32 253 166 2 69 27 114 81 229 77 12 232 128 191 186 106 130 93 149 224]. Error: 
    Vote(c2ffbe8f2215878d4defcf4320fda602451b7251e54d0ce880bfba6a825d95e0)
    Block Number:       332
    Block Hash:         0000000000000000000000000000000000000000000000000000000000000000
    Round:              0
    Type:               0
    V:                  0x28
    R:                  0xe60d0347e6928bbb8cb58f6756e54d849528246ec678dcbcc5157f2c1421ccf5
    S:                  0x14a310e35219011893e4c9099fdc2f2b1c005049ee2830e1d2f8baac50c2687f
    Hex:                f869a0000000000000000000000000000000000000000000000000000000000000000082014c808028a0e60d0347e6928bbb8cb58f6756e54d849528246ec678dcbcc5157f2c1421ccf5a014a310e35219011893e4c9099fdc2f2b1c005049ee2830e1d2f8baac50c2687f

ERROR[09-28|10:15:33.930] Failed to add own vote to voting table   err="duplicate vote"                       blockHash=000000…000000 hash=c2ffbe…5d95e0
INFO [09-28|10:15:33.930] Waiting for a majority in the pre-vote sub-election 
INFO [09-28|10:15:33.930] There's a majority in the pre-vote sub-election! 
INFO [09-28|10:15:33.930] Pre commit sub-election 
DEBUG[09-28|10:15:33.930] Majority of validators pre-voted nil 
ERROR[09-28|10:15:33.930] a duplicate vote in voting table 1; blockHash [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]; voteHash [214 203 86 71 209 235 178 9 38 250 58 100 236 221 103 235 189 173 138 154 132 236 81 40 136 232 126 82 81 140 99 225]. Error: 
    Vote(d6cb5647d1ebb20926fa3a64ecdd67ebbdad8a9a84ec512888e87e52518c63e1)
    Block Number:       332
    Block Hash:         0000000000000000000000000000000000000000000000000000000000000000
    Round:              0
    Type:               1
    V:                  0x27
    R:                  0xae079c4a52a2c10478f31accf42ed472d8e9524975bcf9c7cc9397cf6c2341e8
    S:                  0xd714c00fd910d02bc0271a8977a51f4f229852b7f08672a36ef7c48745ed811
    Hex:                f869a0000000000000000000000000000000000000000000000000000000000000000082014c800127a0ae079c4a52a2c10478f31accf42ed472d8e9524975bcf9c7cc9397cf6c2341e8a00d714c00fd910d02bc0271a8977a51f4f229852b7f08672a36ef7c48745ed811

ERROR[09-28|10:15:33.930] Failed to add own vote to voting table   err="duplicate vote"                       blockHash=000000…000000 hash=d6cb56…8c63e1
INFO [09-28|10:15:33.930] Waiting for a majority in the pre-commit sub-election 
INFO [09-28|10:15:33.930] There's a majority in the pre-commit sub-election! event="(*event.TypeMuxEvent)(<nil>)\n"
DEBUG[09-28|10:15:33.930] No one block wins! 
INFO [09-28|10:15:33.930] Starting a new voting round              start time=2018-09-28T10:15:32+0000 block number=332 round=0
INFO [09-28|10:15:33.930] Waiting for the proposal                 addr=0x2429F4Aa5cF9D23feA0961780Ffb4ff8916A26a0
INFO [09-28|10:15:34.001] Received Proposal 
ERROR[09-28|10:15:34.001] Failed to validate the state             err="invalid merkle root (remote: b7fa03836700aba8f0b21fe270f60252fc9b29877d0907a0602bdf0ff7a5a8a2 local: a4b57ecd81ad807ffd6bbc8504f2cea46aa8c986234661a8cfdfa87dbefe41fc)" round=0 block=334           fragment="&{Index:0 Data:[249 2 71 249 1 236 160 122 83 45 192 156 35 245 20 30 163 153 141 232 58 1 208 93 228 156 69 49 166 200 223 21 141 228 61 110 125 243 126 148 36 41 244 170 92 249 210 63 234 9 97 120 15 251 79 248 145 106 38 160 160 183 250 3 131 103 0 171 168 240 178 31 226 112 246 2 82 252 155 41 135 125 9 7 160 96 43 223 15 247 165 168 162 160 86 232 31 23 27 204 85 166 255 131 69 230 146 192 248 110 91 72 224 27 153 108 173 192 1 98 47 181 227 99 180 33 160 86 232 31 23 27 204 85 166 255 131 69 230 146 192 248 110 91 72 224 27 153 108 173 192 1 98 47 181 227 99 180 33 160 231 70 60 88 61 147 157 16 240 145 181 244 247 56 4 67 182 217 14 163 124 107 199 126 187 21 100 255 255 41 230 3 160 236 149 93 179 39 91 6 54 205 116 206 173 206 47 212 221 68 145 222 47 177 29 64 254 163 212 187 165 125 39 173 142 185 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 130 1 78 131 71 231 196 128 132 91 173 255 70 128 248 85 234 233 160 122 83 45 192 156 35 245 20 30 163 153 141 232 58 1 208 93 228 156 69 49 166 200 223 21 141 228 61 110 125 243 126 130 1 78 128 1 128 128 128 233 160 122 83 45 192 156 35 245 20 30 163 153 141 232 58 1 208 93 228 156 69 49 166 200 223 21 141 228 61 110 125 243 126 130 1 78 128 1 128 128 128 192] Proof:[252 195 228 191 131 80 35 229 187 251 180 174 200 51 58 201 213 239 219 66 179 242 158 64 192 132 4 208 212 240 174 255]}" block="&{header:0xc43a7a5680 lastCommit:0xc43a8c9300 transactions:[] hash:{noCopy:{} v:[246 8 156 192 245 65 240 130 212 42 164 201 156 55 90 124 74 227 47 78 41 207 207 93 15 144 115 220 199 73 215 146]} size:{noCopy:{} v:586} ReceivedAt:0001-01-01 00:00:00 +0000 UTC ReceivedFrom:<nil>}"
CRIT [09-28|10:15:34.001] Failed to validate the state             err="invalid merkle root (remote: b7fa03836700aba8f0b21fe270f60252fc9b29877d0907a0602bdf0ff7a5a8a2 local: a4b57ecd81ad807ffd6bbc8504f2cea46aa8c986234661a8cfdfa87dbefe41fc)"

/cc @rgeraldes @JekaMas

[acroca]

logs.txt Happened again. Adding more logs in case there's something useful

[JekaMas]

@acroca Was it fast sync mode?

[acroca]

No, full sync

[acroca]

The problem still happens in 3.0.0. I think the error is really the invalid merkle root part

[JekaMas]

I've debugged to the AddBlockFragment place. So it's a new error with invalid merkle trie.