Open Qifeng-Wu99 opened 3 months ago
You likely ran out of memory, on a typical system you can check dmesg -T | grep [c]ardano-node
You likely ran out of memory, on a typical system you can check
dmesg -T | grep [c]ardano-node
Thanks for the comment, the command you suggested gives the following output, which indicates that the node process was killed due to OOM:
[Tue May 28 18:03:19 2024] [2985725] 1000 2985725 268450990 4969040 39989248 0 0 cardano-node
[Tue May 28 18:03:19 2024] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/cardano-node.service,task=cardano-node,pid=2985725,uid=1000
[Tue May 28 18:03:19 2024] Out of memory: Killed process 2985725 (cardano-node) total-vm:1073803960kB, anon-rss:19876160kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:39052kB oom_score_adj:0
However, this is unexpected since the server I am using has RAM larger than suggested by Cardano official docs, and the server is purely used for Cardano node operating.
Would you suggest any solutions other than upgrade RAM space?
If you're using -xn for any RTS options for your cardano-node run command, that might be the issue. I have to remove that option before running leaderlogs or node will crash with OOM.
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 120 days.
Internal/External External otherwise.
Area Other Any other topic (Delegation, Ranking, ...).
Summary Node process killed when calculating next epoch's leadership schedule
Steps to reproduce Steps to reproduce the behavior:
Expected behavior The output should looks like below
System info (please complete the following information):
OS Name: Ubuntu
OS Version: 20.04
Node version:
cardano-node 8.9.2 - linux-x86_64 - ghc-8.10
git rev 424983fa186786397f5a99539f51710abf62c37b
CLI version:
cardano-cli 8.20.3.0 - linux-x86_64 - ghc-8.10
git rev 424983fa186786397f5a99539f51710abf62c37b
CPU: 8 core / 2.0 GHz
RAM: 32 GiB
Disk: 400 GiB
Screenshots and attachments Part of the log related this issue is shown below, before and after the node process is killed:
Additional context I noticed this issue recently, the epoch leader calculation was fine in April this year.