502 errors in VMs

[gjost]

Micah reports intermittent 502 errors from Nginx in her DDR VM. Been happening for a couple days now.

pkikawa: she can usually resolve this by rebooting the VM, it's just odd that it's happening fwiw restarting redis-server didn't resolve the 502

[gjost]

densho@ddreditor:~$ sudo tail /var/log/ddr/ddrlocal.log
2023-06-22 14:27:50,562 ERROR    [redis.on_connection_error]  Connection to Redis lost: Retry (10/20) in 1.00 second.
2023-06-22 14:27:50,566 ERROR    [redis.on_connection_error]  Connection to Redis lost: Retry (10/20) in 1.00 second.
2023-06-22 14:27:51,565 ERROR    [redis.on_connection_error]  Connection to Redis lost: Retry (11/20) in 1.00 second.
2023-06-22 14:27:51,572 ERROR    [redis.on_connection_error]  Connection to Redis lost: Retry (11/20) in 1.00 second.
2023-06-22 14:27:52,565 ERROR    [redis.on_connection_error]  Connection to Redis lost: Retry (12/20) in 1.00 second.
2023-06-22 14:27:52,574 ERROR    [redis.on_connection_error]  Connection to Redis lost: Retry (12/20) in 1.00 second.
2023-06-22 14:27:53,573 ERROR    [redis.on_connection_error]  Connection to Redis lost: Retry (13/20) in 1.00 second.
2023-06-22 14:27:53,635 ERROR    [redis.on_connection_error]  Connection to Redis lost: Retry (13/20) in 1.00 second.
2023-06-22 14:27:54,576 ERROR    [redis.on_connection_error]  Connection to Redis lost: Retry (14/20) in 1.00 second.
2023-06-22 14:27:54,667 ERROR    [redis.on_connection_error]  Connection to Redis lost: Retry (14/20) in 1.00 second.

sudo less /var/log/redis/redis-server.log
6007:M 26 Jun 2023 11:25:14.607 # Server initialized
6007:M 26 Jun 2023 11:25:14.607 # WARNING overcommit_memory is set to 0! Backgro
und save may fail under low memory condition. To fix this issue add 'vm.overcomm
it_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm
.overcommit_memory=1' for this to take effect.
6007:M 26 Jun 2023 11:25:14.607 # WARNING you have Transparent Huge Pages (THP)
support enabled in your kernel. This will create latency and memory usage issues
 with Redis. To fix this issue run the command 'echo madvise > /sys/kernel/mm/tr
ansparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order t
o retain the setting after a reboot. Redis must be restarted after THP is disabl
ed (set to 'madvise' or 'never').
6007:M 26 Jun 2023 11:25:14.608 * Loading RDB produced by version 6.0.16
6007:M 26 Jun 2023 11:25:14.608 * RDB age 0 seconds
6007:M 26 Jun 2023 11:25:14.608 * RDB memory usage when created 0.78 Mb
6007:M 26 Jun 2023 11:25:14.608 * DB loaded from disk: 0.000 seconds
6007:M 26 Jun 2023 11:25:14.608 * Ready to accept connections

[gjost]

Found these from googling "Transparent Huge Pages" redis:

• Redis docs: Diagnosing latency issues https://redis.io/docs/management/optimization/latency/
• Transparent Huge Pages and Alternative Memory Allocators: A Cautionary Tale (Digital Ocean) https://www.digitalocean.com/blog/transparent-huge-pages-and-alternative-memory-allocators
• Redis (Arch wiki) https://wiki.archlinux.org/title/Redis
• Transparent Hugepage Support (Linux kernel user's and administrator's guide) https://www.kernel.org/doc/html/latest/admin-guide/mm/transhuge.html

[GeoffFroh]

This issue needs more research and documentation before addressing.

@pkikawa: please add all of the additional info you have about instances of this kind of failure. Record them here moving forward.