Credits earned on 1.18 are significantly lower than 1.17

[michaelh-laine]

We first trialled 1.18.12 a few weeks on when this was requested on MB, where we noticed this issue and called it out on Discord.

We started running 1.18.14 at the end of epoch 617 and have again noted the issue.

Validator Vote account: GE6atKoWiQ2pt3zL7N13pjNHjdLVys8LinG8qeJLcAiL

Typical (consistent across last couple dozen epochs) behaviour is that the validator is in the top 100 in terms of credits earned (e.g. solana validators -r -n --sort=credits.

On 1.18.x I immediately notice the validator begins dropping in the rankings. Now in epoch 618, having run 1.18.14 without interruption and with almost no skipped leader slots, the validator is ranked ~1654 by credits and has approx 3.2% fewer credits than the 50th ranked validator (leaving out the top 49 as these contain modded nodes).

Of the 5 validators running 1.18.14 at the time of writing this issue 4 are ranked 1600+ by credits. If filtering by 1.18 with all patch versions there are 4 validators at the top and all others at the bottom.

We are running jito + crental scheduler. Will try without central scheduler to see if this helps and might explain the few validators at the top.

cc @apfitzge

[apfitzge]

Will be interesting if this goes away when not using central-scheduler. Central scheduler should not affect the way that voting is done by node to my knowledge.

[michaelh-laine]

Will be interesting if this goes away when not using central-scheduler. Central scheduler should not affect the way that voting is done by node to my knowledge.

Just a further data point, as of currently there are no 1.18 nodes in the top 1000 ranked by credits, except for three obvious vote modded nodes. I will switch to the old scheduler before the epoch ends to see the effect in the new epoch. This issue so far is persisting

[diman-io]

34120 i suppose

it can be tested empirically by commenting

// (VOTE_THRESHOLD_DEPTH_SHALLOW, SWITCH_FORK_THRESHOLD),

[michaelh-laine]

I applied Diman's suggestion and can confirm this so far appears to have stopped the loss in credits

The issue here is that validators will mod their clients to remove this change then as it results in higher credits.

[apfitzge]

cc/ @AshwinSekar @t-nelson

[AshwinSekar]

solana-labs#34120 i suppose it can be tested empirically by commenting

// (VOTE_THRESHOLD_DEPTH_SHALLOW, SWITCH_FORK_THRESHOLD),

We've moved this shallow check to log-only, and backported for the next 1.18 release. This should resolve the decrease in vote credits for 1.18. We will monitor metrics and reintroduce this check in the future in such a way that maximizes vote credits while still having the intended effect of reducing minor fork depth.