search

hassio-addons / addon-node-red

Node-RED - Home Assistant Community Add-ons
https://addons.community
MIT License
529 stars 122 forks source link

Out Of Memory #1562

Closed spurgelaurels closed 1 year ago

spurgelaurels commented 1 year ago

Add-on version: 14.0.3 You are running the latest version of this add-on. System: Arch Linux (amd64 / qemux86-64) Home Assistant Core: 2023.2.5 Home Assistant Supervisor: 2023.01.1

Problem/Motivation

Node-red addon in HASS no longer starts.

Expected behavior

Nodered to start....

Actual behavior

Log snippet from docker logs

s6-rc: info: service base-addon-banner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service base-addon-log-level: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service base-addon-log-level successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service init-nginx: starting
s6-rc: info: service init-customizations: starting
s6-rc: info: service init-customizations successfully started
s6-rc: info: service init-nodered: starting
patch: **** out of memory
s6-rc: warning: unable to start service init-nodered: command exited 2
s6-rc: info: service init-nginx successfully started
/run/s6/basedir/scripts/rc.init: warning: s6-rc failed to properly bring all the services up! Check your logs (in /run/uncaught-logs/current if you have in-container logging) for more information.
/run/s6/basedir/scripts/rc.init: fatal: stopping the container.
s6-rc: info: service init-nginx: stopping
s6-rc: info: service init-customizations: stopping
s6-rc: info: service init-nginx successfully stopped
s6-rc: info: service init-customizations successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service base-addon-log-level: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service base-addon-log-level successfully stopped
s6-rc: info: service base-addon-banner: stopping
s6-rc: info: service base-addon-banner successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped

Memory stats at time of attempting to run docker: 

                    total        used        free      shared  buff/cache   available
Mem:           3.8Gi       1.2Gi       1.1Gi       1.0Mi       1.5Gi       2.3Gi

Recent changes: Flows are quite simple in my nodered install, no loops or conditions that would cause this sort of issue. I did remove a flow between two nodes last weekend to stop something from running, but re-deployed with the flow back in place. I don't believe this was related to the issue above.

This issue started a few days later, and restarting the addon didn't help. Rebooting the host OS also didn't help.

I upgraded to the latest version today, and the issue persists.

agners commented 1 year ago

The same issue appears to happen on Home Assistant OS 10.0.rc1, see https://github.com/home-assistant/operating-system/discussions/2426.

I can reprooduce this problem on ODROID-N2. It seems that a simple mmap call fails :thinking: 

root@19ca41362b22:/opt/node_modules/node-red-dashboard$ strace patch -p1 < /etc/node-red/patches/node-red-dashboard-show-dashboard.patch 
execve("/usr/bin/patch", ["patch", "-p1"], 0xffffdf9d5748 /* 23 vars */) = 0
set_tid_address(0xffffb18f3230)         = 53
brk(NULL)                               = 0xaaaad43e4000
brk(0xaaaad43e6000)                     = 0xaaaad43e6000
mmap(0xaaaad43e4000, 4096, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xaaaad43e4000
mprotect(0xaaaabdaee000, 8192, PROT_READ) = 0
clock_gettime(CLOCK_REALTIME, {tv_sec=1679728473, tv_nsec=461112126}) = 0
mmap(NULL, 28672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xffffb18e6000
rt_sigaction(SIGCHLD, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0xffffb1890a90}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGHUP, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RT_1 RT_2], NULL, 8) = 0
rt_sigaction(SIGHUP, {sa_handler=0xaaaabdab8e20, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0xffffb1890a90}, NULL, 8) = 0
rt_sigaction(SIGPIPE, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGPIPE, {sa_handler=0xaaaabdab8e20, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0xffffb1890a90}, NULL, 8) = 0
rt_sigaction(SIGTERM, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGTERM, {sa_handler=0xaaaabdab8e20, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0xffffb1890a90}, NULL, 8) = 0
rt_sigaction(SIGXCPU, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGXCPU, {sa_handler=0xaaaabdab8e20, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0xffffb1890a90}, NULL, 8) = 0
rt_sigaction(SIGXFSZ, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGXFSZ, {sa_handler=0xaaaabdab8e20, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0xffffb1890a90}, NULL, 8) = 0
rt_sigaction(SIGINT, NULL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGINT, {sa_handler=0xaaaabdab8e20, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0xffffb1890a90}, NULL, 8) = 0
fstat(0, {st_mode=S_IFREG|0644, st_size=2132, ...}) = 0
lseek(0, 0, SEEK_CUR)                   = 0
lseek(0, 0, SEEK_SET)                   = 0
lseek(0, 0, SEEK_CUR)                   = 0
read(0, "--- a/nodes/ui_base.html\n+++ b/n"..., 1024) = 1024
lseek(0, 0, SEEK_CUR)                   = 1024
lseek(0, 0, SEEK_CUR)                   = 1024
openat(AT_FDCWD, "nodes", O_RDONLY|O_LARGEFILE|O_NOFOLLOW|O_DIRECTORY) = 3
prlimit64(0, RLIMIT_NOFILE, NULL, {rlim_cur=1073741816, rlim_max=1073741816}) = 0
mmap(NULL, 5368709188, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Out of memory)
writev(2, [{iov_base="patch: **** out of memory\n", iov_len=26}, {iov_base=NULL, iov_len=0}], 2patch: **** out of memory
) = 26
lseek(0, -952, SEEK_CUR)                = 72
exit_group(2)                           = ?
+++ exited with 2 +++

Probably the most likely culprit is the upgrade to Docker 23.0.1.

@spurgelaurels are you running Supervised? What version of Docker are you using?

agners commented 1 year ago

In HAOS case, it seems that the culprit is missing swap space. So this is a HAOS 10.0.rc1 issue.

gateriderz commented 1 year ago

Same here, after update of HAOS 10.0.rc1, Node-red out of memory. I have not read any workaround at the moment.

agners commented 1 year ago

You can downgrade using ha os update --version 9.5.

I'll likely build a rc2 on Monday.

gateriderz commented 1 year ago

@agners , thanks for your quick response. Downgrade to 9.5 completed successfully and nodered working now.

gateriderz commented 1 year ago

Hi @agners , 10.0.rc2 still same issue with NodeRed, Error: out of memory.

Ran free -h and swapon

image

I ran the update to 10.0.dev20230317 and still the same issue:

image

image

NodeRed still shows out of memory:

image

Thanks for your help

agners commented 1 year ago

It really seems that patch allocates an excessive amount of memory. I've also reported the problem in Alpine (see https://gitlab.alpinelinux.org/alpine/aports/-/issues/14763).

However, for our add-on we can also side step the issue by applying the patch at build time.