Block fits onto some fork

[Robinyo]

Internal/External External

Area cardano-node

Summary As per the leadership schedule we (ORCA) were supposed to mint 12 blocks this epoch. However, for some reason we didn't mint one of the blocks:

     SlotNo                          UTC Time
-------------------------------------------------------------
     79949089                        2022-12-20 05:49:40 UTC

Log entries for slot: 79949089

$ sudo docker logs --since=2022-12-20T05:49:35Z core-node-1 2>&1 | grep "79949089"
[41025f04:cardano.node.LeadershipCheck:Info:589] [2022-12-20 05:49:40.00 UTC] {"chainDensity":4.863708e-2,"credentials":"Cardano","delegMapSize":1245559,"kind":"TraceStartLeadershipCheck","slot":79949089,"utxoSize":9519831}
[41025f04:cardano.node.ChainDB:Info:579] [2022-12-20 05:49:40.54 UTC] Block fits onto some fork: 8c3b1d29913a9e4beb32afd140c0b83aa9f05dc67df9a64cecb39b5c661e36c1 at slot 79949089
[41025f04:cardano.node.ChainDB:Notice:579] [2022-12-20 05:49:40.55 UTC] Switched to a fork, new tip: 8c3b1d29913a9e4beb32afd140c0b83aa9f05dc67df9a64cecb39b5c661e36c1 at slot 79949089

We successfully minted blocks either side of this slot, for example, slot 79931329

$ sudo docker logs --since=2022-12-20T00:53:35Z core-node-1 2>&1 | grep "79931329"
[41025f04:cardano.node.LeadershipCheck:Info:589] [2022-12-20 00:53:40.00 UTC] {"chainDensity":4.860314e-2,"credentials":"Cardano","delegMapSize":1245482,"kind":"TraceStartLeadershipCheck","slot":79931329,"utxoSize":9524514}
[41025f04:cardano.node.ChainDB:Info:579] [2022-12-20 00:53:40.07 UTC] Valid candidate 7af020c49b9c9fb65bfe953693eb23ba0fc63e485a2a9661db1fd167417ef1ab at slot 79931329
[41025f04:cardano.node.ChainDB:Notice:579] [2022-12-20 00:53:40.07 UTC] Chain extended, new tip: 7af020c49b9c9fb65bfe953693eb23ba0fc63e485a2a9661db1fd167417ef1ab at slot 79931329

Expected behavior We mint the block as per the leadership schedule.

System info (please complete the following information):

• OS Name: Debian
• OS Version: 11.5
• Node version: cardano-node 1.35.3 - linux-aarch64 - ghc-8.10
• CLI version: cardano-cli 1.35.3 - linux-aarch64 - ghc-8.10

See: https://github.com/input-output-hk/cardano-node/issues/4726#issuecomment-1371208494

[Robinyo]

Internal/External External

Area cardano-node

Summary As per the leadership schedule we were supposed to mint a block:

     SlotNo                          UTC Time
-------------------------------------------------------------
     81542542                   2023-01-07 16:27:13 UTC

Log entries for slot: 81542542

sudo docker logs core-node-1 2>&1 | grep "81542542"

As you can see there were no log entries for this slot?

We successfully minted other blocks in this epoch, for example, slot 81436892

$ sudo docker logs core-node-1 2>&1 | grep "81436892"
[c45c89b7:cardano.node.LeadershipCheck:Info:721] [2023-01-06 11:06:23.00 UTC] {"chainDensity":5.000579e-2,"credentials":"Cardano","delegMapSize":1253224,"kind":"TraceStartLeadershipCheck","slot":81436892,"utxoSize":9591754}
[c45c89b7:cardano.node.ChainDB:Info:711] [2023-01-06 11:06:23.14 UTC] Valid candidate 40e0fb2dc14e00d10a0ee98b9322af3a5a4d234b179b4036d46f9d44a6f95844 at slot 81436892
[c45c89b7:cardano.node.ChainDB:Notice:711] [2023-01-06 11:06:23.14 UTC] Chain extended, new tip: 40e0fb2dc14e00d10a0ee98b9322af3a5a4d234b179b4036d46f9d44a6f95844 at slot 81436892

Expected behavior We mint the block as per the leadership schedule.

System info (please complete the following information):

• OS Name: Debian
• OS Version: 11.5
• Node version: cardano-node 1.35.4 - linux-aarch64
• CLI version: cardano-cli 1.35.4 - linux-aarch64

[amesgen]

Thanks for the report! Judging from the logs you have provided, it is not possible to say whether your node actually determined that you were leading the specified slots (i.e. neither NodeIsLeader nor NodeNotLeader occurs in the logs).

One plausible guess is that you are experiencing a bug in the leadership schedule command, which is known to erroneously tells you that you are leading a specific slot. See https://github.com/input-output-hk/cardano-node/issues/4583#issuecomment-1339519511 and https://github.com/input-output-hk/cardano-node/pull/4106 for this, which should be included in 1.36.0.

Closing this issue for now, feel free to reopen if you have additional information suggesting that something different is going on here :+1: