nerc-project / operations

Issues related to the operation of the NERC OpenShift environment
2 stars 0 forks source link

Should we adjust transparent huge pages (THP) kernel configuration? #228

Closed larsks closed 5 months ago

larsks commented 1 year ago

The ArgoCD redis service reports:

WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled (set to 'madvise' or 'never').

The OpenShift documentation describes how to tune this setting via the Node Tuning Operator:

larsks commented 1 year ago

The Redis documentation says:

Unfortunately when a Linux kernel has transparent huge pages enabled, Redis incurs to a big latency penalty after the fork call is used in order to persist on disk. Huge pages are the cause of the following issue:

  1. Fork is called, two processes with shared huge pages are created.
  2. In a busy instance, a few event loops runs will cause commands to target a few thousand of pages, causing the copy on write of almost the whole process memory.
  3. This will result in big latency and big memory usage. Make sure to disable transparent huge pages using the following command:
echo never > /sys/kernel/mm/transparent_hugepage/enabled
joachimweyl commented 5 months ago

Closing until/if this comes up again.