Closed scottdk closed 5 months ago
Can somebody with repo admin access move this to node-red/node-red-docker please.
While that is happening, start the 1.3.7 container and then run the following:
$ docker run -it -d -p 1880:1880 -v node_red_data:/data --name mynodered nodered/node-red:1.3.7
$ docker exec -it mynodered /bin/sh
~ $ ls -alhd /data
...
~ $ ls -alh /data
...
This will show us what files and permissions the content of the /data
has in the container.
Out of all the docker run
commands all appart from the middle 2 are using named volumes so we don't know what you have mapped to those volumes, it could be the /share/Container/node-red/data
or it could be ephemeral volumes.
I have run the commands as suggested, as well as with the numeric uid and gid.
/ $ ls -alhd /data
drwxrwxr-x 5 node-red root 4.0K May 2 14:07 /data
/ $ ls -alhdn /data
drwxrwxr-x 5 1000 0 4.0K May 2 14:07 /data
/ $ ls -alh /data
total 64K
drwxrwxr-x 5 node-red root 4.0K May 2 14:07 .
drwxr-xr-x 1 root root 4.0K May 2 14:06 ..
-rw-r--r-- 1 node-red node-red 14.2K May 2 14:07 .config.nodes.json
-rw-r--r-- 1 node-red node-red 95 May 2 14:07 .config.runtime.json
drwxr-xr-x 3 node-red node-red 4.0K May 1 05:45 .npm
-rw-rw-r-- 1 root root 1.3K May 1 05:04 flows.json
drwxr-xr-x 3 node-red node-red 4.0K May 2 14:07 lib
drwxr-xr-x 2 node-red node-red 4.0K May 2 14:07 node_modules
-rw-r--r-- 1 node-red node-red 120 May 2 14:07 package.json
-rw-r--r-- 1 node-red node-red 15.3K May 2 14:07 settings.js
/ $ ls -alhn /data
total 64K
drwxrwxr-x 5 1000 0 4.0K May 2 14:07 .
drwxr-xr-x 1 0 0 4.0K May 2 14:06 ..
-rw-r--r-- 1 1000 1000 14.2K May 2 14:07 .config.nodes.json
-rw-r--r-- 1 1000 1000 95 May 2 14:07 .config.runtime.json
drwxr-xr-x 3 1000 1000 4.0K May 1 05:45 .npm
-rw-rw-r-- 1 0 0 1.3K May 1 05:04 flows.json
drwxr-xr-x 3 1000 1000 4.0K May 2 14:07 lib
drwxr-xr-x 2 1000 1000 4.0K May 2 14:07 node_modules
-rw-r--r-- 1 1000 1000 120 May 2 14:07 package.json
-rw-r--r-- 1 1000 1000 15.3K May 2 14:07 settings.js
Nothing looks strange there (apart from the flows.json
being owned by root:root) but the failure is before it would try and open/write to the flow.
There is nothing in change between Docker 1.3.x and 2.x.x that should change the permissions. It's trying to rewrite the .config.nodes.json
will be because a couple of the core nodes got moved to being installed as stand alone packages (rbe and tail iirc)
Had the same problem. Did a chown -R 1000:1000 *
and chown -R 1000:1000 .*
(for the files starting with ".") in the mounted volume and it worked
Careful with chown -R 1000:1000 .*
that will follow the ".." as well. It's better to just do a chown on the top level directory for node-red. If in the mounted container chown -R 1000:1000 /data/
will do it.
I get this error installing 3.0.2-18 on QNAP, same error with and without mapping external volumes
node:internal/fs/utils:348
throw err;
^
Error: EPERM: operation not permitted, copyfile '/usr/src/node-red/node_modules/node-red/settings.js' -> '/data/settings.js'
at Object.copyFileSync (node:fs:2866:3)
at copyFile (/usr/src/node-red/node_modules/fs-extra/lib/copy/copy-sync.js:73:6)
at onFile (/usr/src/node-red/node_modules/fs-extra/lib/copy/copy-sync.js:59:25)
at getStats (/usr/src/node-red/node_modules/fs-extra/lib/copy/copy-sync.js:51:44)
at handleFilterAndCopy (/usr/src/node-red/node_modules/fs-extra/lib/copy/copy-sync.js:36:10)
at Object.copySync (/usr/src/node-red/node_modules/fs-extra/lib/copy/copy-sync.js:29:10)
at Object.<anonymous> (/usr/src/node-red/node_modules/node-red/red.js:129:20)
at Module._compile (node:internal/modules/cjs/loader:1120:14)
at Module._extensions..js (node:internal/modules/cjs/loader:1174:10)
at Module.load (node:internal/modules/cjs/loader:998:32) {
errno: -1,
syscall: 'copyfile',
code: 'EPERM',
path: '/usr/src/node-red/node_modules/node-red/settings.js',
dest: '/data/settings.js'
}
Node.js v18.7.0
@dcolley please supply the full docker run
command line you used to start Node-RED.
Are you sure you haven't changed the effective uid?
With no volumes mounted /data
is owned by the node-red:root
user:group so that should work just fine (works fine when testing here)
On QNAP, the admin user has
# id $USER
uid=0(admin) gid=0(administrators) groups=0(administrators),100(everyone)
# cat /etc/passwd
admin:x:0:0:administrator:/share/homes/admin:/bin/sh
guest:x:65534:65534:guest:/tmp:/bin/sh
httpdusr:x:99:0:Apache httpd user:/tmp:/bin/sh
[sshd]:x:110:65534:SSHD Privilege Separation:/var/empty:/bin/sh
derek:x:1000:100:Linux User,derek@...,,:/share/homes/derek:/bin/sh
# cat /etc/group
administrators:x:0:admin,derek
everyone:x:100:admin,derek
[/share/Containers/node-red/3.0.2] # ls -la
total 12
drwxrwxrwx 2 derek everyone 4096 2022-10-06 14:47 ./
drwxrwxrwx 8 derek everyone 4096 2022-10-06 13:50 ../
# docker volume list
DRIVER VOLUME NAME
local xxx
local yyy
local zzz
local aaa
local bbb
local node-red-data
docker run -it -p 1882:1880 -v node-red-data:/data --name node-red-3.0.2-18-test nodered/node-red:3.0.2-18
... produces the error above.
Also, running it without -v has the same result
Edit:
I also tried --user 1000:100
Without a volume that should just work (and does when run here on my Ubuntu and Fedora machines)
With a volume then you need to ensure that any directly mounted directories are writable by UID 1000, but for a bare local docker volume it should just work. (Though volumes backed by NFS still show up as local as well)
The Node-RED process runs as UID 1000 and as I said has permission to write to /data
inside the container, this hasn't changed in the last 2 years (and even then the change was to use the root group, so basic uid stayed the same)
Sure, on Ubuntu (where users 1000 and root exist) it just works. On QNAP it doesn't work out of the box.
I managed to get it working, but it's along process. The process:
Documented here for the record:
I'm using port 1881 temporarily. I need a working installation so I can port over my old flows currently running on 1880
docker run -it -v /share/Containers/node-red/3.0.2:/data -p 1881:1880 --name node-red-3.0.2-18 amd64/node:16-alpine /bin/sh
# inside container
alias ll='ls -l'
ll
cd
apk add --no-cache \
bash \
tzdata \
iputils \
curl \
nano \
git \
openssl \
openssh-client \
ca-certificates
mkdir -p /usr/src/node-red /data && \
deluser --remove-home node && \
adduser -h /usr/src/node-red -D -H node-red -u 1000
Copy some files into the running container
# docker cp scripts/*.sh node-red-3.0.2-18:/tmp/
docker cp scripts/install_devtools.sh node-red-3.0.2-18:/tmp/
# docker cp scripts/entrypoint.sh node-red-3.0.2-18:/tmp/
docker cp scripts/remove_native_gpio.sh node-red-3.0.2-18:/tmp/
docker cp healthcheck.js node-red-3.0.2-18:/
docker cp known_hosts.sh node-red-3.0.2-18:/usr/src/node-red
docker cp package.json node-red-3.0.2-18:/usr/src/node-red
docker cp flows.js node-red-3.0.2-18:/usr/src/node-red
docker cp scripts/entrypoint.sh node-red-3.0.2-18:/usr/src/node-red
chown -R node-red:root /data && chmod -R g+rwX /data
# ls -la /data
chown -R node-red:root /usr/src/node-red && chmod -R g+rwX /usr/src/node-red
cd /usr/src/node-red
chmod +x known_hosts.sh
chmod +x /tmp/remove_native_gpio.sh
./known_hosts.sh /etc/ssh/ssh_known_hosts
rm /usr/src/node-red/known_hosts.sh
echo "PubkeyAcceptedKeyTypes +ssh-rsa" >> /etc/ssh/ssh_config
apk add --no-cache --virtual buildtools build-base linux-headers udev python3 && \
npm install --unsafe-perm --no-update-notifier --no-audit --no-fund --only=production && \
/tmp/remove_native_gpio.sh
# cp -R node_modules prod_node_modules
chmod +x /tmp/install_devtools.sh
chown -R node-red:root /usr/src/node-red
/tmp/install_devtools.sh
npm config set cache /data/.npm --global
npm config set python `which python3` --global
su - node-red
NODE_RED_VERSION=3.0.2-18 \
NODE_PATH=/usr/src/node-red/node_modules:/data/node_modules \
PATH=/usr/src/node-red/node_modules/.bin:${PATH} \
FLOWS=flows.json
chmod +x ./entrypoint.sh
./entrypoint.sh
After this you will have a working container with correct files/structure in /data and /usr/src/node-red
ctrl-c # stop node-red
ctrl-d # quite/stop the container
docker rm node-red-3.0.2-18
link the container to your /share/Containers/node-red/3.0.2
docker run -it -d -p 1881:1880 -v /share/Containers/node-red/3.0.2:/data --name node-red-3.0.2-18 nodered/node-red:3.0.2-18
When I have copied over my existing flows (not on a -v mount...), I'll delete the instance on 1880 and recreate this instance on 1880
Well I spoke too soon... the method above does not survive a restart...
8 Oct 11:37:29 - [info] Node-RED version: v3.0.2
8 Oct 11:37:29 - [info] Node.js version: v18.7.0
8 Oct 11:37:29 - [info] Linux 4.2.8 x64 LE
8 Oct 11:37:30 - [info] Loading palette nodes
8 Oct 11:37:30 - [error] Failed to start server:
8 Oct 11:37:30 - [error] Error: EPERM: operation not permitted, copyfile '/data/.config.nodes.json' -> '/data/.config.nodes.json.backup'
8 Oct 11:38:46 - [info]
The users shouldn't need to exist on the host OS at all (especially when not mounting a volume). Every thing is just handled by numeric IDs and totally inside the container.
Unless the qnap docker instance is doing something strange
Try running the following on the qnap console (not in the container)
setfacl -Rb /share/Containers/node-red/3.0.2
Unless the qnap docker instance is doing something strange
Most possibly... it's an older version of docker.
# docker --version
Docker version 17.09.1-ce, build 0bbe3ac
I appreciate your help, but this is still not working... setfacl made no difference.
I'm stumped for now, without a QNAP box to play with and poke at the logs I'm not sure what to suggest.
I had the same issues too on my QNAP (QTS 5.0.1.2034) but the following seemed to have worked: cd /share/Container/nodered chown -R 1000:1000 data
docker-compose.yml `nodered: image: nodered/node-red:latest restart: unless-stopped ports:
docker-compose up -d
Note: I did have an existing user with uid 1000 but there wasn't an existing group with gid 1000 Note 2: I am not using traefik at the moment but may do at some point so thought to add the labels anyway.
You do not need named groups/users with the same numerical id on the host platform, the names are just a way to give a label for humans to understand, you can always use chmod/chown with numerical ids.
You do not need named groups/users with the same numerical id on the host platform, the names are just a way to give a label for humans to understand, you can always use chmod/chown with numerical ids.
Edited: I really didn't think you would need to chown the folder on the host system (because it fails even when not specifying a volume).
On QNAP I managed to get the container started with --privileged
.
docker run -it -d \
-p 1880:1880 \
-v /share/Containers/node-red/3.0.2/data:/data \
--privileged \
--name node-red-3.0.2-18 nodered/node-red:3.0.2-18
The container survives a restart so, even though I don't think this is ideal, I'm moving on...
Current Behavior
What has changed between 1.3.7 and 2.x.x regarding permissions to write to the /data volume?
Running any 2.x.x nodred version in my docker fails with the error:
Error: EPERM: operation not permitted, copyfile '/data/.config.nodes.json' -> '/data/.config.nodes.json.backup'
However, running 1.3.7 has no issues.
I have read and followed every article I can find, including: https://nodered.org/docs/getting-started/docker#managing-user-data https://github.com/node-red/node-red-docker/wiki/Permissions-and-Persistence
Where mounting an external folder as the data volume, I have ensured it's owner and group is 1000:1000
I have tried numerous variations on the startup commands in the documentation.
Expected Behavior
The server will run as expected.
Steps To Reproduce
Try to spin up any nodered 2.x.x version and it fails..
Try the same for v1.3.7 and it is successful.
Example flow
No response
Environment