The Issue:
The biggest problem chia fork farmers face is ram usage. Many farmers simply do not have the ram required to run all the forks and my 64GB machine is at its limits as well
One Cause:
By default, chia and its forks create number of logical cores - 2 start_full_node process. Each worker process takes 50MB RAM. On a typical 8 core, 16 threads CPU this is 700MB of RAM per chia fork. For 38 Forks this is about 26GB of ram for the full_nodes alone
One Solution:
Give farmers the ability to limit the number of worker processes. Make this configurable, so pure, designated full_nodes can still use all cpu cores for their task, but allow farmers to reduce the CPU and RAM load of running a full_node
Pro:
On an 8 core / 16 thread cpu this can easily save up to 600MB RAM per fork
On an 32 core / 64 thread cpu this can easily save 3GB RAM per fork
The ram saving is especially noticable on dual core machines, which likely have little ram available
Users can choose to reduce ram usage, which allows them to farm more forks including yours if they there previously unable to
Cons:
Reducing the number of worker processes can reduce sync speed. This does not affect ongoing synchronization of synced nodes, only a node that is way behind the chain really benefits from many worker processes. The user has full control over that and can reduce the number of worker processes once synced or set them to a good compromise between resource consumption and sync speed, tailored for his machine
Changes:
5 Lines of code (two of those line are imports). In a place where the chia team already limits the number of worker processes to 61 I added a config check and apply the configured limit if one is set
1 new config option in config.yaml
Who is effected by this change:
This change only effects users who add the config option multiprocessing_limit to their config.yaml
The Issue: The biggest problem chia fork farmers face is ram usage. Many farmers simply do not have the ram required to run all the forks and my 64GB machine is at its limits as well
One Cause: By default, chia and its forks create number of logical cores - 2 start_full_node process. Each worker process takes 50MB RAM. On a typical 8 core, 16 threads CPU this is 700MB of RAM per chia fork. For 38 Forks this is about 26GB of ram for the full_nodes alone
One Solution: Give farmers the ability to limit the number of worker processes. Make this configurable, so pure, designated full_nodes can still use all cpu cores for their task, but allow farmers to reduce the CPU and RAM load of running a full_node
Pro:
Cons: Reducing the number of worker processes can reduce sync speed. This does not affect ongoing synchronization of synced nodes, only a node that is way behind the chain really benefits from many worker processes. The user has full control over that and can reduce the number of worker processes once synced or set them to a good compromise between resource consumption and sync speed, tailored for his machine
Changes:
Who is effected by this change:
multiprocessing_limitto their config.yaml