[BUG] - VRFKeyBadNonceOVERLAY on epoch cutover

[Scitz0]

Exernal

Summary It has been observed running a node on private Guild Operators network that blocks minted right after epoch cutover(under 10s after cutover, exact time not confirmed but at least up to 6s) get rejected, trace: TraceForgedInvalidBlock, kind: VRFKeyBadNonceOVERLAY.

Steps to reproduce Have a block producing node with enough stake to regularly make blocks right after epoch cutover.

Expected behavior Blocks successfully made and adopted.

System info (please complete the following information):

• OS: Ubuntu
• Version: 20.04
• Node version: cardano-node 1.13.0 - linux-x86_64 - ghc-8.6 git rev 9f49515afc8ca3473cc3f1e1104765e0e357d888

Screenshots and attachments Full JSON trace of VRFKeyBadNonceOVERLAY

{
  "at": "2020-06-19T09:00:02.13Z",
  "env": "1.13.0:c8ec2",
  "ns": [
    "cardano.node.Forge"
  ],
  "data": {
    "kind": "TraceForgedInvalidBlock",
    "reason": {
      "kind": "ValidationError",
      "error": {
        "kind": "HeaderProtocolError",
        "error": {
          "kind": "PredicateFailure",
          "failures": [
            {
              "kind": "VRFKeyBadNonceOVERLAY",
              "blockNonce": "CertifiedVRF {certifiedNatural = 257496154028758090498779421544410045289, certifiedProof = CertSimpleVRF {certU = Point 2262937906362959516303148039147879 4848089730085357766205839523249491, certC = 54060608959167033178960862473799130198, certS = 4126235464891015280355460803085182}}",
              "currentSlot": "229501",
              "previousHashAsNonce": "Nonce 24ec739bd17a04e65b8931cc8d6a2c6e5f0eaf64f561e344ebee79c92f630a50",
              "seedNonce": "Nonce 6e340b9cffb37a989ca544e6bb780a2c78901d3fb33738768511a30617afa01d"
            }
          ]
        }
      }
    },
    "slot": 229501
  },
  "app": [],
  "msg": "",
  "pid": "3477",
  "loc": null,
  "host": "cnode-ah",
  "sev": "Error",
  "thread": "46"
}

Image from my block collector implemented in guild operators cntools script tool to get a feeling for timestamps. Epoch cutover for guild network is exactly on the hour with 30min epochs. Shows 5 consecutive epochs for the first blocks made by my block producing node. The Hash field contains a base64 encoded representation of JSON trace like the one above.

Additional context Genesis for Guild network: https://github.com/cardano-community/guild-operators/blob/master/files/ptn0/files/genesis.json

We have checked if it's attempting to make a block when there is an overlay schedule with higher priority (due to d= 0.5) but this was not the case, was not on BFT schedule.

I have only ever seen the issue right after epoch cutover.

[kevinhammond]

The correlation with the epoch cutover suggests there may be an issue with e.g. the amount of time that is taken for rewards calculation.

[vix-io]

hi @nc6 is this still an issue?

[mrbrinker]

Has this issue been solved?

[Jimbo4350]

Closing this. If this is still relevant please re-open.