[Bug] Full Node minimal requirements are outdated

[Ealrann]

What happened?

As stated by official install documentation, the minimal requirements are:

The minimum supported specs are that of the Raspberry Pi 4:178,553

    Quad core 1.5Ghz CPU (must be 64 bit)
    2 GB Ram
    Python 3.7 and above

These specs are outdated:

1. Low end servers are loosing sync during dust storms (still rare, but are becoming more and more regular). They are missing lots of challenges during these storms.
2. The 2 GB ram is also a bad advice: Full node requires more and more RAM weeks after weeks. Running a node with less than 4GB is a bad idea since the server will have to use swap rather quickly, and 8 GB will probably be needed in 2022.

For example, a full node running since September (runs Chia full node only, not even a wallet, and was rebooted regularly).

We were all enthusiast about using a Pi to run Chia, but it's a misleading advertising for now.

Maybe I missed it, but if not: the Chia team should make an official statement about what hardware we should use to correctly maintain the network.

Version

1.2.11

What platform are you using?

Linux

What ui mode are you using?

CLI

Relevant log output

No response

[mariano54]

That does seem like a lot of RAM. Can you confirm that it's the Chia application? Which process is using the most RAM?

[Ealrann]

Here the firsts line of top:

top - 13:24:23 up 21 days, 13:22,  1 user,  load average: 0.16, 0.26, 0.21
Tasks: 126 total,   1 running, 125 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.1 us,  0.1 sy,  0.0 ni, 98.7 id,  0.0 wa,  0.1 hi,  0.1 si,  0.0 st
MiB Mem :   7758.4 total,    247.6 free,   3602.8 used,   3908.0 buff/cache
MiB Swap:   8192.0 total,   8189.5 free,      2.5 used.   3856.4 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 311778 ealrann   20   0 2531200   1.9g  20736 S   4.3  25.1   4573:21 chia_full_node
 313008 ealrann   20   0 1582472 955688   7324 S   0.0  12.0 240:59.72 chia_full_node
 311990 ealrann   20   0 1416616 935972   7324 S   0.0  11.8 241:23.69 chia_full_node
 311989 ealrann   20   0 1415668 930392   6520 S   0.0  11.7 434:17.98 chia_full_node
 311986 ealrann   20   0 1268204 930348   6520 S   0.0  11.7 433:18.29 chia_full_node
 311443 ealrann   20   0  430728  62064  18084 S   0.0   0.8  40:47.59 chia_daemon
    255 root      20   0  100600  49452  48396 S   0.0   0.6   4:23.43 systemd-journal

With a Monthly graph, we can see a bit more:

The system load spikes match to the dust storms, we can see some extra memory usage was accumulated from there.

The memory usage drops correspond to a restart of the full node application.

[Ealrann]

In fact, I'm not really surprised that a serious farmer needs descent hardware for the full node.

The problem is that for now, we don't really know what are the requirements to correctly pass the dust storms. I can say that even with a 'descent' i3 8100 (not really powerful, but still light years from the Raspberry Pi), I missed many challenges during the storm.

[emlowe]

I updated the wiki page to say 4GB ram. Currently, on the pi4 we suggest also adding in 1GB of swap space.

The pi4 is capable of maintaining sync and responding to SPs during dust storms, but there is considerable variance in the community. Some people with the pi4 have no trouble, others have trouble even with much more substantial machines. Much depends on the quality of the disk where the blockchain DB is, for example, do not use the SD card on a pi as it cannot handle the I/O during dust storms. We continue to evaluate best practices and also continue to optimize code. The pi4 4gb model continues to be our target base system.

[mariano54]

We need to look at the BLS pairings cache as this was increased recently to improve performance, but we might have gone too far there (too high memory usage).

[Ealrann]

I updated the wiki page to say 4GB ram.

Thank you, I think it is more accurate for now indeed.

The pi4 is capable of maintaining sync and responding to SPs during dust storms, but there is considerable variance in the community.

In fact, I didn't really notice any lost on sync during the storm (but maybe some short cuts happened?), but my farmer missed far more challenges than usual.

[tvadim]

I struggle to start the full node even with 4GB: chia_full_node gets repeatedly killed by oom_reaper

[...]  oom_kill_process+0x21f/0x420
[...] [ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[...] oom_reaper: reaped process 17697 (chia_full_node), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

Chia: 1.2.11 OS: Ubuntu 18.04.6 LTS Python: 3.7.12

Would appreciate any advise.

[alexluft]

I'm also experiencing oom issues with 1.2.11 on 8gb rpi

[mariano54]

Question to anyone using high memory: Are you syncing or have you been synced for a long time?

[alexluft]

@mariano54 yes, actually syncing a backlog of about 5 hours.

[mariano54]

How many hours or days have you been syncing?

[alexluft]

It takes about 8 minutes until oom. Maybe it's something different...

[tvadim]

@derverzweifler This could be a corrupt blockchain DB (just an idea). I tried removing it and node began re-syncing without oom

[alexluft]

@derverzweifler This could be a corrupt blockchain DB (just an idea). I tried removing it and node began re-syncing without oom

I guess, it's the case. RAM usage is very stable after db removal.