[BUG] - Node Process Killed When Calculating Next Epoch's Leadership Schedule

[Qifeng-Wu99]

Internal/External External otherwise.

Area Other Any other topic (Delegation, Ranking, ...).

Summary Node process killed when calculating next epoch's leadership schedule

Steps to reproduce Steps to reproduce the behavior:

1. Start a node with

   /usr/local/bin/cardano-node run +RTS -N -A16m -qg -qb -RTS --topology ${TOPOLOGY} --database-path ${DB_PATH} --socket-path ${SOCKET_PATH} --host-addr ${HOSTADDR} --port ${PORT} --config ${CONFIG} --shelley-kes-key ${KES} --shelley-vrf-key ${VRF} --shelley-operational-certificate ${CERT}

2. Calculate the next epoch's leadership schedule with

   cardano-cli query leadership-schedule \
   --mainnet \
   --genesis $NODE_HOME/shelley-genesis.json \
   --stake-pool-id $(cat $NODE_HOME/stakepoolid.txt) \
   --vrf-signing-key-file $NODE_HOME/vrf.skey \
   --next
   --socket-path db/socket

3. The node process is killed a few minutes after execute step 2, and the node is restarted automatically
4. The output of step 2 pop out during the restarting of the node process and is empty:

   SlotNo                          UTC Time
   -------------------------------------------------------------

5. The Restarting of the node takes time longer than usual

Expected behavior The output should looks like below

SlotNo                          UTC Time
-------------------------------------------------------------
     4073                   2021-12-29 17:26:54.998001755 UTC
     4126                   2021-12-29 17:27:00.298001755 UTC
     4206                   2021-12-29 17:27:08.298001755 UTC

System info (please complete the following information):

• OS Name: Ubuntu

• OS Version: 20.04

• Node version: cardano-node 8.9.2 - linux-x86_64 - ghc-8.10 git rev 424983fa186786397f5a99539f51710abf62c37b

• CLI version: cardano-cli 8.20.3.0 - linux-x86_64 - ghc-8.10 git rev 424983fa186786397f5a99539f51710abf62c37b

• CPU: 8 core / 2.0 GHz

• RAM: 32 GiB

• Disk: 400 GiB

Screenshots and attachments Part of the log related this issue is shown below, before and after the node process is killed:

May 28 13:31:36 ada-node cardano-node[1118012]: [ada-node:cardano.node.ChainDB:Notice:14260] [2024-05-28 13:31:36.23 UTC] Chain extended, new tip: 5a5299bed0f203b79e62eb5a23ac1a9d4b3a249e69bfd90cc40b6dea276ab027 at slot 1>
May 28 13:31:37 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:37.00 UTC] {"chainDensity":4.6852283e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:37 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:37.05 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:38 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:38.00 UTC] {"chainDensity":4.6852283e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:38 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:38.05 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:39 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:39.00 UTC] {"chainDensity":4.6852283e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:39 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:39.04 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:40 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:40.00 UTC] {"chainDensity":4.6852283e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:40 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:40.04 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:41 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:41.00 UTC] {"chainDensity":4.6852283e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:41 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:41.05 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:42 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:42.00 UTC] {"chainDensity":4.6852283e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:42 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:42.05 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:43 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:43.00 UTC] {"chainDensity":4.6852283e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:43 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:43.05 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:44 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:44.00 UTC] {"chainDensity":4.6852283e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:44 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:44.05 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:45 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:45.00 UTC] {"chainDensity":4.6852283e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:45 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:45.05 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:46 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:46.00 UTC] {"chainDensity":4.6852283e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:46 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:46.05 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:47 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:47.00 UTC] {"chainDensity":4.6852283e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:47 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:47.05 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:48 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:48.00 UTC] {"chainDensity":4.6852283e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:48 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:48.05 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:49 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:49.00 UTC] {"chainDensity":4.6852283e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:49 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:49.06 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:50 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:50.00 UTC] {"chainDensity":4.6852283e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:50 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:50.08 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:51 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:51.00 UTC] {"chainDensity":4.6852283e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:51 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:51.05 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:51 ada-node cardano-node[1118012]: [ada-node:cardano.node.ChainDB:Notice:14260] [2024-05-28 13:31:51.39 UTC] Chain extended, new tip: 22c923a0aab9b9656f877a87ad47d3635c31aeb30a3d2f724ad3afd7b47a3dd3 at slot 1>
May 28 13:31:52 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:52.00 UTC] {"chainDensity":4.6877716e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:52 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:52.05 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:53 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:53.00 UTC] {"chainDensity":4.6877716e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:53 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:53.05 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:54 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:54.00 UTC] {"chainDensity":4.6877716e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:54 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:54.06 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:55 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:55.00 UTC] {"chainDensity":4.6877716e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:55 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:55.06 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:56 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:56.00 UTC] {"chainDensity":4.6877716e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:56 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:56.05 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:57 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:57.00 UTC] {"chainDensity":4.6877716e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:57 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:57.04 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:58 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:58.00 UTC] {"chainDensity":4.6877716e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:58 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:58.04 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:31:59 ada-node cardano-node[1118012]: [ada-node:cardano.node.LeadershipCheck:Info:14276] [2024-05-28 13:31:59.00 UTC] {"chainDensity":4.6877716e-2,"credentials":"Cardano","delegMapSize":1344902,"kind":"TraceStar>
May 28 13:31:59 ada-node cardano-node[1118012]: [ada-node:cardano.node.Forge:Info:14276] [2024-05-28 13:31:59.05 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader">
May 28 13:32:01 ada-node cardano-node[1117999]: /home/ada/cardano-my-node/startCardanoNode.sh: line 11: 1118012 Killed                  /usr/local/bin/cardano-node run +RTS -N -A16m -qg -qb -RTS --topology ${TOPOLOGY} --d>
May 28 13:32:07 ada-node cardano-node[2985725]: Node configuration: NodeConfiguration {ncSocketConfig = SocketConfig {ncNodeIPv4Addr = Last {getLast = Just 0.0.0.0}, ncNodeIPv6Addr = Last {getLast = Nothing}, ncNodePortNu>
May 28 13:32:07 ada-node cardano-node[2985725]: Listening on http://127.0.0.1:12798
May 28 13:32:07 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.protocol:Notice:11] [2024-05-28 13:32:07.54 UTC] Byron; Shelley
May 28 13:32:07 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.version:Notice:11] [2024-05-28 13:32:07.54 UTC] 8.9.1
May 28 13:32:07 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.commit:Notice:11] [2024-05-28 13:32:07.54 UTC] da945ea983d4722a9ffe54250edba9a193a57cf0
May 28 13:32:07 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.nodeStartTime:Notice:11] [2024-05-28 13:32:07.54 UTC] 2024-05-28 13:32:07.541878989 UTC
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.systemStartTime:Notice:11] [2024-05-28 13:32:07.54 UTC] 2017-09-23 21:44:51 UTC
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.slotLengthByron:Notice:11] [2024-05-28 13:32:07.54 UTC] 20s
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.epochLengthByron:Notice:11] [2024-05-28 13:32:07.54 UTC] 21600
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.slotLengthShelley:Notice:11] [2024-05-28 13:32:07.54 UTC] 1s
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.epochLengthShelley:Notice:11] [2024-05-28 13:32:07.54 UTC] 432000
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.slotsPerKESPeriodShelley:Notice:11] [2024-05-28 13:32:07.54 UTC] 129600
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.slotLengthAllegra:Notice:11] [2024-05-28 13:32:07.54 UTC] 1s
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.epochLengthAllegra:Notice:11] [2024-05-28 13:32:07.54 UTC] 432000
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.slotsPerKESPeriodAllegra:Notice:11] [2024-05-28 13:32:07.54 UTC] 129600
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.slotLengthMary:Notice:11] [2024-05-28 13:32:07.54 UTC] 1s
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.epochLengthMary:Notice:11] [2024-05-28 13:32:07.54 UTC] 432000
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.slotsPerKESPeriodMary:Notice:11] [2024-05-28 13:32:07.54 UTC] 129600
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.slotLengthAlonzo:Notice:11] [2024-05-28 13:32:07.54 UTC] 1s
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.epochLengthAlonzo:Notice:11] [2024-05-28 13:32:07.54 UTC] 432000
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.slotsPerKESPeriodAlonzo:Notice:11] [2024-05-28 13:32:07.54 UTC] 129600
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.slotLengthBabbage:Notice:11] [2024-05-28 13:32:07.54 UTC] 1s
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.epochLengthBabbage:Notice:11] [2024-05-28 13:32:07.54 UTC] 432000
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.slotsPerKESPeriodBabbage:Notice:11] [2024-05-28 13:32:07.54 UTC] 129600
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.slotLengthConway:Notice:11] [2024-05-28 13:32:07.54 UTC] 1s
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.epochLengthConway:Notice:11] [2024-05-28 13:32:07.54 UTC] 432000
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.basicInfo.slotsPerKESPeriodConway:Notice:11] [2024-05-28 13:32:07.54 UTC] 129600
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.startup:Info:11] [2024-05-28 13:32:08.22 UTC] Config path /home/ada/cardano-my-node/config.json, Network magic NetworkMagic {unNetworkMagic = 76482407>
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.startup:Info:11] [2024-05-28 13:32:08.22 UTC] Era Byron, Slot length 20s, Epoch length 21600
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.startup:Info:11] [2024-05-28 13:32:08.23 UTC] Era Shelley, Slot length 1s, Epoch length 432000, Slots per KESPeriod 129600
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.startup:Info:11] [2024-05-28 13:32:08.23 UTC] Era Allegra, Slot length 1s, Epoch length 432000, Slots per KESPeriod 129600
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.startup:Info:11] [2024-05-28 13:32:08.23 UTC] Era Mary, Slot length 1s, Epoch length 432000, Slots per KESPeriod 129600
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.startup:Info:11] [2024-05-28 13:32:08.23 UTC] Era Alonzo, Slot length 1s, Epoch length 432000, Slots per KESPeriod 129600
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.startup:Info:11] [2024-05-28 13:32:08.23 UTC] Era Babbage, Slot length 1s, Epoch length 432000, Slots per KESPeriod 129600
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.startup:Info:11] [2024-05-28 13:32:08.23 UTC] Era Conway, Slot length 1s, Epoch length 432000, Slots per KESPeriod 129600
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.startup:Warning:11] [2024-05-28 13:32:08.23 UTC] You are using an early release of peer-to-peer capabilities, please report any issues.
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.startup:Info:11] [2024-05-28 13:32:08.23 UTC] startup time: 1716903129
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.startup:Info:11] [2024-05-28 13:32:08.23 UTC]
May 28 13:32:08 ada-node cardano-node[2985725]: node addresses:          0.0.0.0:6000
May 28 13:32:08 ada-node cardano-node[2985725]: local socket:            /home/ada/cardano-my-node/db/socket
May 28 13:32:08 ada-node cardano-node[2985725]: node-to-node versions:   [7,8,9,10,11,12,13]
May 28 13:32:08 ada-node cardano-node[2985725]: node-to-client versions: [9,10,11,12,13,14,15]
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.startup:Info:11] [2024-05-28 13:32:08.23 UTC] initiator and responder diffusion mode
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.startup:Info:11] [2024-05-28 13:32:08.23 UTC]
May 28 13:32:08 ada-node cardano-node[2985725]: Local Root Groups:
May 28 13:32:08 ada-node cardano-node[2985725]:   (HotValency {getHotValency = 1},WarmValency {getWarmValency = 1},[(RelayAccessAddress "159.203.146.191" 6000,(DoNotAdvertisePeer,IsNotTrustable))])
May 28 13:32:08 ada-node cardano-node[2985725]: Public Roots:
May 28 13:32:08 ada-node cardano-node[2985725]:   
May 28 13:32:08 ada-node cardano-node[2985725]: Don't use ledger to get root peers.
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.ChainDB:Info:11] [2024-05-28 13:32:08.23 UTC] Started opening Chain DB
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.ChainDB:Info:11] [2024-05-28 13:32:08.23 UTC] Started opening Immutable DB
May 28 13:32:08 ada-node cardano-node[2985725]: [ada-node:cardano.node.ChainDB:Info:11] [2024-05-28 13:32:08.55 UTC] Validating chunk no. 0 out of 5800. Progress: 0.00%
May 28 13:32:09 ada-node cardano-node[2985725]: [ada-node:cardano.node.ChainDB:Info:11] [2024-05-28 13:32:09.26 UTC] Validated chunk no. 0 out of 5800. Progress: 0.02%
May 28 13:32:09 ada-node cardano-node[2985725]: [ada-node:cardano.node.ChainDB:Info:11] [2024-05-28 13:32:09.26 UTC] Validating chunk no. 1 out of 5800. Progress: 0.02%
May 28 13:32:10 ada-node cardano-node[2985725]: [ada-node:cardano.node.ChainDB:Info:11] [2024-05-28 13:32:10.22 UTC] Validated chunk no. 1 out of 5800. Progress: 0.03%
May 28 13:32:10 ada-node cardano-node[2985725]: [ada-node:cardano.node.ChainDB:Info:11] [2024-05-28 13:32:10.22 UTC] Validating chunk no. 2 out of 5800. Progress: 0.03%
May 28 13:32:11 ada-node cardano-node[2985725]: [ada-node:cardano.node.ChainDB:Info:11] [2024-05-28 13:32:11.17 UTC] Validated chunk no. 2 out of 5800. Progress: 0.05%
May 28 13:32:11 ada-node cardano-node[2985725]: [ada-node:cardano.node.ChainDB:Info:11] [2024-05-28 13:32:11.17 UTC] Validating chunk no. 3 out of 5800. Progress: 0.05%
May 28 13:32:12 ada-node cardano-node[2985725]: [ada-node:cardano.node.ChainDB:Info:11] [2024-05-28 13:32:12.18 UTC] Validated chunk no. 3 out of 5800. Progress: 0.07%
May 28 13:32:12 ada-node cardano-node[2985725]: [ada-node:cardano.node.ChainDB:Info:11] [2024-05-28 13:32:12.18 UTC] Validating chunk no. 4 out of 5800. Progress: 0.07%

Additional context I noticed this issue recently, the epoch leader calculation was fine in April this year.

[rdlrt]

You likely ran out of memory, on a typical system you can check dmesg -T | grep [c]ardano-node

[Qifeng-Wu99]

You likely ran out of memory, on a typical system you can check dmesg -T | grep [c]ardano-node

Thanks for the comment, the command you suggested gives the following output, which indicates that the node process was killed due to OOM:

[Tue May 28 18:03:19 2024] [2985725]  1000 2985725 268450990  4969040 39989248        0             0 cardano-node
[Tue May 28 18:03:19 2024] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/cardano-node.service,task=cardano-node,pid=2985725,uid=1000
[Tue May 28 18:03:19 2024] Out of memory: Killed process 2985725 (cardano-node) total-vm:1073803960kB, anon-rss:19876160kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:39052kB oom_score_adj:0

However, this is unexpected since the server I am using has RAM larger than suggested by Cardano official docs, and the server is purely used for Cardano node operating.

Would you suggest any solutions other than upgrade RAM space?

[syndeton]

If you're using -xn for any RTS options for your cardano-node run command, that might be the issue. I have to remove that option before running leaderlogs or node will crash with OOM.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 120 days.