Cannot install new nodes on fresh Docker image

rcarmo commented 3 months ago

What are the steps to reproduce?

I deployed a fresh stack in Portainer (inspect dump at the bottom) and tried to install nodes (the example below is for the sqlite node, which is smaller and self-contained when compared to the other ones). The same happens when I try to install any other node. A container image I maintain with 3.1.6 and Node 18 does not exhibit this behavior (but I cannot use mine in this environment).

What happens?

Welcome to Node-RED
===================
8 Mar 14:36:21 - [info] Node-RED version: v3.1.6
8 Mar 14:36:21 - [info] Node.js  version: v16.20.2
8 Mar 14:36:21 - [info] Linux 6.5.13-1-pve x64 LE
8 Mar 14:36:24 - [info] Loading palette nodes
8 Mar 14:36:28 - [info] Settings file  : /data/settings.js
8 Mar 14:36:28 - [info] Context store  : 'default' [module=memory]
8 Mar 14:36:28 - [info] User directory : /data
8 Mar 14:36:28 - [warn] Projects disabled : editorTheme.projects.enabled=false
8 Mar 14:36:28 - [info] Flows file     : /data/flows.json
8 Mar 14:36:28 - [info] Creating new flow file
8 Mar 14:36:28 - [warn] 
---------------------------------------------------------------------
Your flow credentials file is encrypted using a system-generated key.
If the system-generated key is lost for any reason, your credentials
file will not be recoverable, you will have to delete it and re-enter
your credentials.
You should set your own key using the 'credentialSecret' option in
your settings file. Node-RED will then re-encrypt your credentials
file using your chosen key the next time you deploy a change.
---------------------------------------------------------------------
8 Mar 14:36:28 - [warn] Encrypted credentials not found
8 Mar 14:36:28 - [info] Starting flows
8 Mar 14:36:28 - [info] Started flows
8 Mar 14:36:28 - [info] Server now running at http://127.0.0.1:1880/
8 Mar 14:36:49 - [info] Installing module: node-red-node-sqlite, version: 1.1.0
8 Mar 14:37:07 - [warn] Installation of module node-red-node-sqlite failed:
8 Mar 14:37:07 - [warn] ------------------------------------------
8 Mar 14:37:07 - [warn] npm WARN config production Use `--omit=dev` instead.
8 Mar 14:37:07 - [warn] ------------------------------------------
Error: Install failed
    at /usr/src/node-red/node_modules/@node-red/registry/lib/installer.js:285:25
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
8 Mar 14:37:07 - [error] Error: Install failed

What do you expect to happen?

I should be able to install nodes without errors.

Please tell us about your environment:

[x] Platform/OS: macOS Sonoma, Fedora 39, Windows 11 - basically irrelevant for this case

[x] Browser: Safari, Firefox, Edge - same

[x] Server: Ubuntu Linux 22.04 with Docker CE and latest Portainer.

{
    "AppArmorProfile": "",
    "Args": [],
    "Config": {
        "AttachStderr": true,
        "AttachStdin": false,
        "AttachStdout": true,
        "Cmd": null,
        "Domainname": "",
        "Entrypoint": [
            "./entrypoint.sh"
        ],
        "Env": [
            "PUID=1000",
            "PGID=1000",
            "PATH=/usr/src/node-red/node_modules/.bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
            "NODE_VERSION=16.20.2",
            "YARN_VERSION=1.22.19",
            "NODE_RED_VERSION=v3.1.6",
            "NODE_PATH=/usr/src/node-red/node_modules:/data/node_modules",
            "FLOWS=flows.json"
        ],
        "ExposedPorts": {
            "1880/tcp": {}
        },
        "Healthcheck": {
            "Test": [
                "CMD-SHELL",
                "node /healthcheck.js"
            ]
        },
        "Hostname": "9cdcb532d122",
        "Image": "nodered/node-red",
        "Labels": {
            "authors": "Dave Conway-Jones, Nick O'Leary, James Thomas, Raymond Mouthaan",
            "com.centurylinklabs.watchtower.enable": "true",
            "com.docker.compose.config-hash": "22b698e95b2fa54437a768db6559ed92829863540ad8197c14273761c09d330a",
            "com.docker.compose.container-number": "1",
            "com.docker.compose.depends_on": "",
            "com.docker.compose.image": "sha256:fcb9a25b2c5c8b4e4a7498834f7b9c15bf680cb945ef0845464fb47766a8c74f",
            "com.docker.compose.oneoff": "False",
            "com.docker.compose.project": "automation",
            "com.docker.compose.project.config_files": "/data/compose/14/docker-compose.yml",
            "com.docker.compose.project.working_dir": "/data/compose/14",
            "com.docker.compose.replace": "b46c4cb19fd034d5b57b5c3364459d318d3a72626e32fc463a5dfa232cf381c3",
            "com.docker.compose.service": "node-red",
            "com.docker.compose.version": "2.20.2",
            "org.label-schema.arch": "",
            "org.label-schema.build-date": "2024-03-01T11:55:26Z",
            "org.label-schema.description": "Low-code programming for event-driven applications.",
            "org.label-schema.docker.dockerfile": ".docker/Dockerfile.alpine",
            "org.label-schema.license": "Apache-2.0",
            "org.label-schema.name": "Node-RED",
            "org.label-schema.url": "https://nodered.org",
            "org.label-schema.vcs-ref": "",
            "org.label-schema.vcs-type": "Git",
            "org.label-schema.vcs-url": "https://github.com/node-red/node-red-docker",
            "org.label-schema.version": "3.1.6"
        },
        "OnBuild": null,
        "OpenStdin": false,
        "StdinOnce": false,
        "Tty": false,
        "User": "node-red",
        "Volumes": null,
        "WorkingDir": "/usr/src/node-red"
    },
    "Created": "2024-03-08T13:27:35.071434455Z",
    "Driver": "overlay2",
    "ExecIDs": null,
    "GraphDriver": {
        "Data": {
            "LowerDir": "/var/lib/docker/overlay2/24ff60c7b53836b1a41cf1a7a0db5d6ba422ba77f5ca6acb091a695c038c6de5-init/diff:/var/lib/docker/overlay2/f4a8269509caaafd92190b6ff11112c459caa61695f23298f75c7ad28485969f/diff:/var/lib/docker/overlay2/d124e45103c266fa7b095fef3ce4b9be9db2f0a5831a5c1caab3b5dd64932996/diff:/var/lib/docker/overlay2/70df87568b63160f6258a9aa892a0c66c2ffa1c26feefb62507407bace6b75d9/diff:/var/lib/docker/overlay2/adf07242b8788ce717f1ce1eb26e3474272dcd47d1c41e0644aa5dae85e5e407/diff:/var/lib/docker/overlay2/8d0d89e3007a07ce194e067a0a6628ab7ac079a5e122cbe4b479b9132a3f4045/diff:/var/lib/docker/overlay2/f10338c4414af57017fd80557840256ef0287d91d3524833a0ef2602f32fe90b/diff:/var/lib/docker/overlay2/3edce3940eb3fe437c19872d51c2ece0db8739f007a978753fe03dec467b44da/diff:/var/lib/docker/overlay2/eee8aaa03f0a489a27120a7efb05c11e32941f504ca1a64a1b03e0d27e74b828/diff:/var/lib/docker/overlay2/2e427562b7305acd6cfd23628e580975204ee0b2d56e8e34a754b0f72f9b5d6d/diff:/var/lib/docker/overlay2/61eda0d269fce5fb961644b276e89c1f97ba903c46be907b104734cd3228bfb2/diff:/var/lib/docker/overlay2/8a9178cd78a41d194d2589840a9d349cc3c3d1f1206dbfcc7ba9c85d3e2d49f6/diff:/var/lib/docker/overlay2/2e221b2dbffc88c6bbbde0b2fc26d8c305aa450873425bf6ade8e62e9ea3f39f/diff:/var/lib/docker/overlay2/9fd92d13cd5f6414179cb0f941e1a606d018587541a2d13f64331ca2c5f25d8e/diff:/var/lib/docker/overlay2/7472566ad0a01b606662a399f473226d866c19c1dad94a22efac7222f94a5bdd/diff:/var/lib/docker/overlay2/410242a2472ceb457dd3ad05d14fbdb81ac83863b464009ae15754cbd51eb353/diff:/var/lib/docker/overlay2/f1a622ca31964a197953aff5b0d74263b8ba77bd69690fe5bcf942a2660ef1cd/diff:/var/lib/docker/overlay2/59b92be6fef3686ef9d944b0b5b4d2093ea6b2d3ec12daec396f131b5e6731cf/diff",
            "MergedDir": "/var/lib/docker/overlay2/24ff60c7b53836b1a41cf1a7a0db5d6ba422ba77f5ca6acb091a695c038c6de5/merged",
            "UpperDir": "/var/lib/docker/overlay2/24ff60c7b53836b1a41cf1a7a0db5d6ba422ba77f5ca6acb091a695c038c6de5/diff",
            "WorkDir": "/var/lib/docker/overlay2/24ff60c7b53836b1a41cf1a7a0db5d6ba422ba77f5ca6acb091a695c038c6de5/work"
        },
        "Name": "overlay2"
    },
    "HostConfig": {
        "AutoRemove": false,
        "Binds": [
            "/etc/localtime:/etc/localtime:ro",
            "/dev/rtc:/dev/rtc:ro",
            "/root/portainer/stacks/automation/node-red/data:/data:rw",
            "/mnt/downloads:/downloads:rw"
        ],
        "BlkioDeviceReadBps": null,
        "BlkioDeviceReadIOps": null,
        "BlkioDeviceWriteBps": null,
        "BlkioDeviceWriteIOps": null,
        "BlkioWeight": 0,
        "BlkioWeightDevice": null,
        "CapAdd": null,
        "CapDrop": null,
        "Cgroup": "",
        "CgroupParent": "",
        "CgroupnsMode": "private",
        "ConsoleSize": [
            0,
            0
        ],
        "ContainerIDFile": "",
        "CpuCount": 0,
        "CpuPercent": 0,
        "CpuPeriod": 0,
        "CpuQuota": 0,
        "CpuRealtimePeriod": 0,
        "CpuRealtimeRuntime": 0,
        "CpuShares": 0,
        "CpusetCpus": "",
        "CpusetMems": "",
        "DeviceCgroupRules": null,
        "DeviceRequests": null,
        "Devices": null,
        "Dns": null,
        "DnsOptions": null,
        "DnsSearch": null,
        "ExtraHosts": [],
        "GroupAdd": null,
        "IOMaximumBandwidth": 0,
        "IOMaximumIOps": 0,
        "IpcMode": "private",
        "Isolation": "",
        "Links": null,
        "LogConfig": {
            "Config": {},
            "Type": "json-file"
        },
        "MaskedPaths": [
            "/proc/asound",
            "/proc/acpi",
            "/proc/kcore",
            "/proc/keys",
            "/proc/latency_stats",
            "/proc/timer_list",
            "/proc/timer_stats",
            "/proc/sched_debug",
            "/proc/scsi",
            "/sys/firmware",
            "/sys/devices/virtual/powercap"
        ],
        "Memory": 134217728,
        "MemoryReservation": 0,
        "MemorySwap": 134217728,
        "MemorySwappiness": null,
        "NanoCpus": 1000000000,
        "NetworkMode": "automation_default",
        "OomKillDisable": null,
        "OomScoreAdj": 0,
        "PidMode": "",
        "PidsLimit": null,
        "PortBindings": {
            "1880/tcp": [
                {
                    "HostIp": "",
                    "HostPort": "80"
                }
            ]
        },
        "Privileged": false,
        "PublishAllPorts": false,
        "ReadonlyPaths": [
            "/proc/bus",
            "/proc/fs",
            "/proc/irq",
            "/proc/sys",
            "/proc/sysrq-trigger"
        ],
        "ReadonlyRootfs": false,
        "RestartPolicy": {
            "MaximumRetryCount": 0,
            "Name": "always"
        },
        "Runtime": "runc",
        "SecurityOpt": null,
        "ShmSize": 67108864,
        "UTSMode": "",
        "Ulimits": null,
        "UsernsMode": "",
        "VolumeDriver": "",
        "VolumesFrom": null
    },
    "HostnamePath": "/var/lib/docker/containers/9cdcb532d12296c2de203de784d6a4c734a171b56a8ce4fa71da23a515a24673/hostname",
    "HostsPath": "/var/lib/docker/containers/9cdcb532d12296c2de203de784d6a4c734a171b56a8ce4fa71da23a515a24673/hosts",
    "Id": "9cdcb532d12296c2de203de784d6a4c734a171b56a8ce4fa71da23a515a24673",
    "Image": "sha256:fcb9a25b2c5c8b4e4a7498834f7b9c15bf680cb945ef0845464fb47766a8c74f",
    "LogPath": "/var/lib/docker/containers/9cdcb532d12296c2de203de784d6a4c734a171b56a8ce4fa71da23a515a24673/9cdcb532d12296c2de203de784d6a4c734a171b56a8ce4fa71da23a515a24673-json.log",
    "MountLabel": "",
    "Mounts": [
        {
            "Destination": "/downloads",
            "Mode": "rw",
            "Propagation": "rprivate",
            "RW": true,
            "Source": "/mnt/downloads",
            "Type": "bind"
        },
        {
            "Destination": "/etc/localtime",
            "Mode": "ro",
            "Propagation": "rprivate",
            "RW": false,
            "Source": "/etc/localtime",
            "Type": "bind"
        },
        {
            "Destination": "/dev/rtc",
            "Mode": "ro",
            "Propagation": "rprivate",
            "RW": false,
            "Source": "/dev/rtc",
            "Type": "bind"
        },
        {
            "Destination": "/data",
            "Mode": "rw",
            "Propagation": "rprivate",
            "RW": true,
            "Source": "/root/portainer/stacks/automation/node-red/data",
            "Type": "bind"
        }
    ],
    "Name": "/node-red",
    "NetworkSettings": {
        "Bridge": "",
        "EndpointID": "",
        "Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "HairpinMode": false,
        "IPAddress": "",
        "IPPrefixLen": 0,
        "IPv6Gateway": "",
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "MacAddress": "",
        "Networks": {
            "automation_default": {
                "Aliases": [
                    "node-red",
                    "node-red",
                    "9cdcb532d122"
                ],
                "DriverOpts": null,
                "EndpointID": "d986be4e54235fcd1ef72adf5d848d09a94e732a31b75c5f7bba0f443c4e3357",
                "Gateway": "172.21.0.1",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "IPAMConfig": null,
                "IPAddress": "172.21.0.2",
                "IPPrefixLen": 16,
                "IPv6Gateway": "",
                "Links": null,
                "MacAddress": "02:42:ac:15:00:02",
                "NetworkID": "b0016a2db03dffb4ce55570ae94962cec4e57b67b77f7aab396dd9cff21e9ad0"
            }
        },
        "Ports": {
            "1880/tcp": [
                {
                    "HostIp": "0.0.0.0",
                    "HostPort": "80"
                },
                {
                    "HostIp": "::",
                    "HostPort": "80"
                }
            ]
        },
        "SandboxID": "8a4b65009d73ffd6c7bf3a3ad8278ff021c4470a368f1c71a530473d40f5cf68",
        "SandboxKey": "/var/run/docker/netns/8a4b65009d73",
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null
    },
    "Path": "./entrypoint.sh",
    "Platform": "linux",
    "Portainer": {
        "ResourceControl": {
            "Id": 10,
            "ResourceId": "5_automation",
            "SubResourceIds": [],
            "Type": 6,
            "UserAccesses": [],
            "TeamAccesses": [],
            "Public": false,
            "AdministratorsOnly": true,
            "System": false
        }
    },
    "ProcessLabel": "",
    "ResolvConfPath": "/var/lib/docker/containers/9cdcb532d12296c2de203de784d6a4c734a171b56a8ce4fa71da23a515a24673/resolv.conf",
    "RestartCount": 0,
    "State": {
        "Dead": false,
        "Error": "",
        "ExitCode": 0,
        "FinishedAt": "2024-03-08T14:36:16.1245188Z",
        "Health": {
            "FailingStreak": 0,
            "Log": [
                {
                    "End": "2024-03-08T14:37:49.04385885Z",
                    "ExitCode": 0,
                    "Output": "",
                    "Start": "2024-03-08T14:37:48.620344922Z"
                },
                {
                    "End": "2024-03-08T14:38:19.498953856Z",
                    "ExitCode": 0,
                    "Output": "",
                    "Start": "2024-03-08T14:38:19.067298794Z"
                },
                {
                    "End": "2024-03-08T14:38:49.935607104Z",
                    "ExitCode": 0,
                    "Output": "",
                    "Start": "2024-03-08T14:38:49.51341434Z"
                },
                {
                    "End": "2024-03-08T14:39:20.381967539Z",
                    "ExitCode": 0,
                    "Output": "",
                    "Start": "2024-03-08T14:39:19.952165483Z"
                },
                {
                    "End": "2024-03-08T14:39:50.813624624Z",
                    "ExitCode": 0,
                    "Output": "",
                    "Start": "2024-03-08T14:39:50.395764967Z"
                }
            ],
            "Status": "healthy"
        },
        "OOMKilled": true,
        "Paused": false,
        "Pid": 688696,
        "Restarting": false,
        "Running": true,
        "StartedAt": "2024-03-08T14:36:17.616739652Z",
        "Status": "running"
    }
}

hardillb commented 3 months ago

Is the directory mounted on /data writeable by uid 1000 (the node-red user)?

rcarmo commented 3 months ago

Yes, just checked it via the portainer console. That would have failed with an ERR_WRITE kind of error upon container launch as it bootstraps settings.js (not my first rodeo).

hardillb commented 3 months ago

Actually:

"OOMKilled": true,

Looks significant. Have you applied any memory limits to this container? This is most likely because npm has been killed for using too much memory.

rcarmo commented 3 months ago

Yes, but a similar environment with Node 18 works with the same restrictions, and this is just the sqlite node I'm trying to deploy. I'll check the RAM budget for this node (this is a remote edge deployment). Meanwhile, this is the data dir on a fresh redeploy:

hardillb commented 3 months ago

"Memory": 134217728,

That is only 128mb which is not going to cut it, especially as it's will be shared between Node-RED and npm

rcarmo commented 3 months ago

Well, Node 18 is better at this then. I will check whether I can deploy using npm or just rsync the installed packages (which. again, are working on a lab with just 128M).

hardillb commented 3 months ago

And you can always use the nodered/node-red:latest-18 if you want a NodeJS 18 based container

hardillb commented 3 months ago

But this is not a problem with the shipped Node-RED containers

rcarmo commented 3 months ago

Fair. I got it to work by asking the admin to increase allowed swap (there are edge environments where RAM is at a premium).

node-red / node-red-docker