search

node-red / node-red-docker

Repository for all things Node-RED and Docker related
Apache License 2.0
471 stars 382 forks source link

Manage Palette - palette won't update #332

Closed seaverd closed 1 year ago

seaverd commented 1 year ago

Current Behavior

I have several palettes that indicate they have updates available. Also same error occurs when attempting to install a new palette. When I attempt to update/install them after a few seconds, I get the message below in the logs:

PLEASE NOTE: I rolled back my docker tag to 3.0.1 and all functions as it is supposed to and I don't get the error below. The errors below happen with docker tag 3.0.1-1 and 3.0.2 latest.

51 verbose type system
52 verbose stack FetchError: Invalid response body while trying to fetch https://registry.npmjs.org/node-red-contrib-power-monitor: ENOSYS: function not implemented, link '/data/.npm/_cacache/tmp/536b9d89' -> '/data/.npm/_cacache/content-v2/sha512/ca/78/ecf9ea9e429677649e945a71808de04bdb7c3b007549b9a2b8c1e2f24153a034816fdb11649d9265afe902d0c1d845c02ac702ae46967c08ebb58bc2ca53'
52 verbose stack at /usr/local/lib/node_modules/npm/node_modules/minipass-fetch/lib/body.js:168:15
52 verbose stack at async RegistryFetcher.packument (/usr/local/lib/node_modules/npm/node_modules/pacote/lib/registry.js:99:25)
52 verbose stack at async RegistryFetcher.manifest (/usr/local/lib/node_modules/npm/node_modules/pacote/lib/registry.js:124:23)
52 verbose stack at async Arborist.[nodeFromEdge] (/usr/local/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/build-ideal-tree.js:1108:19)
52 verbose stack at async Arborist.[buildDepStep] (/usr/local/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/build-ideal-tree.js:976:11)
52 verbose stack at async Arborist.buildIdealTree (/usr/local/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/build-ideal-tree.js:218:7)
52 verbose stack at async Promise.all (index 1)
52 verbose stack at async Arborist.reify (/usr/local/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/reify.js:153:5)
52 verbose stack at async Install.exec (/usr/local/lib/node_modules/npm/lib/commands/install.js:156:5)
52 verbose stack at async module.exports (/usr/local/lib/node_modules/npm/lib/cli.js:78:5)
53 verbose cwd /data
54 verbose Linux 5.15.46-Unraid
55 verbose node v16.16.0
56 verbose npm v8.11.0
57 error code ENOSYS
58 error syscall link
59 error path /data/.npm/_cacache/tmp/536b9d89
60 error dest /data/.npm/_cacache/content-v2/sha512/ca/78/ecf9ea9e429677649e945a71808de04bdb7c3b007549b9a2b8c1e2f24153a034816fdb11649d9265afe902d0c1d845c02ac702ae46967c08ebb58bc2ca53
61 error errno ENOSYS
62 error Invalid response body while trying to fetch https://registry.npmjs.org/node-red-contrib-power-monitor: ENOSYS: function not implemented, link '/data/.npm/_cacache/tmp/536b9d89' -> '/data/.npm/_cacache/content-v2/sha512/ca/78/ecf9ea9e429677649e945a71808de04bdb7c3b007549b9a2b8c1e2f24153a034816fdb11649d9265afe902d0c1d845c02ac702ae46967c08ebb58bc2ca53'
63 verbose exit 1
64 timing npm Completed in 3896ms
65 verbose unfinished npm timer reify 1661012682451
66 verbose unfinished npm timer reify:loadTrees 1661012682455
67 verbose code 1
68 error A complete log of this run can be found in:
68 error /data/.npm/_logs/2022-08-20T16_24_42_307Z-debug-0.log

Expected Behavior

Update/install without errors. Rolling back to 3.0.1 solves my issue.

Steps To Reproduce

Just have 3.0.1-1 or later nodered installed and attempt to install palette or update palette.

Example flow

paste your flow here

Not required

Environment

hardillb commented 1 year ago

This should have been raised against https://github.com/node-red/node-red-docker (don't worry we will move it).

I assume this is the same problem we were discussing on Reddit?

Please include the full docker command line used to start the instance.

But I still think it is a problem with the volume you have mounted on /data

seaverd commented 1 year ago

@hardillb Yes same problem as reddit. Thought I should create issue here (although looks like I chose wrong repo...thanks for offer to move it).

Confused by the /data volume comment...since this contains my persistent data...if I build the docker with 3.0.1 everthing works as it has for the past year. building it with 3.0.1-1 and later using the same /data volume mapping I get the errors I listed.

I use Unraid and they utilize a template in their GUI. Values get pulled from template to generate the docker command line...anyway...here is what runs after I complete the template:

docker run -d --name='NodeRedNew' --net='host' -e TZ="America/New_York" -e HOST_OS="Unraid" -e HOST_HOSTNAME="Tower" -e HOST_CONTAINERNAME="NodeRedNew" -e 'TCP_PORT_1880'='1880' -e 'NODE_RED_ENABLE_SAFE_MODE'='false' -l net.unraid.docker.managed=dockerman -l net.unraid.docker.webui='http://[IP]:[PORT:1880]' -l net.unraid.docker.icon='https://nodered.org/about/resources/media/node-red-icon.png' -v '/mnt/user/appdata/nodered_new':'/data':'rw' 'nodered/node-red:3.0.1'

hardillb commented 1 year ago

The reason I think it's a problem with the /data volume is that the error ENOSYS for syscall link is because the underlying OS is saying that the filesystem that it's trying to write to does not support the link command.

The difference between 3.0.1and 3.0.1-1 is that the 3.0.1 had the .npm cache directory set to /usr/src/node-red when it should be on /data because it got missed when the fix for npm dropping root privileges (which shouldn't be a problem in this case as it looks like you are running as the standard node-red user).

Can you run the following test.

seaverd commented 1 year ago

@hardillb Attached is the output you requested.
Thanks for helping me with this. Based on what you are saying I may need to report an issue back to the developers of Unraid?

nodered_console_output

hardillb commented 1 year ago

@Steve-Mcl When you get a second can you move this to node-red-docker please.

@seaverd I think we need to double check (not saying I don't believe you, I just want have some proof for when we potentially pass this upstream) what filesystem /mnt/user/appdata/nodered_new actually is. Can you include the output of the mount command on the host os.

seaverd commented 1 year ago

@hardillb Here is the output of the mount from the host os logs nodered_console_output2 .

seaverd commented 1 year ago

@hardillb

Ben when in the host os, if I go to my mount /mnt/cache/appdata/nodered_new and execute the commands you asked me to try...they work. I don't get the function not implemented error and end up with two files in my directory test-soft and test-hard with contents the same as package.json. So it is working on host os....just does not work when within the container.

hardillb commented 1 year ago

That doesn't look right, running mount on the command line should give something like the following which shows the "device", the "mount point", the file system type and finally the options.

(Also it's really helpful to post the actual text not screen shots if possible)

$ mount
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=4096k,nr_inodes=131072,mode=755,inode64)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,size=608984k,nr_inodes=819200,mode=755,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime,seclabel)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
/dev/mapper/fedora-root on / type ext4 (rw,relatime,seclabel)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,nosuid,noexec,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14084)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime,seclabel)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime,seclabel)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime,seclabel)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,seclabel,mode=700)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,seclabel,size=1522460k,nr_inodes=1048576,inode64)
/dev/sda1 on /boot type ext4 (rw,relatime,seclabel)
/dev/mapper/fedora-home on /home type ext4 (rw,relatime,seclabel)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=304488k,nr_inodes=76122,mode=700,uid=1000,gid=1000,inode64)

I'm also confused because I thought shfs was a way to mount files via SSH. But anyway I see the XFS filesystem being mounted on /mnt/cache but I don't see how that is then used for /mnt/user which is the path you are passing as the volume?

hardillb commented 1 year ago

OK, so that implies it's a some interaction between Docker and the filesystem. I think you are going to have to raise this with the unraid folk as running the ln commands in the container implies it's something lower level and node-red/npm.

(At the outside it might be something alpine related as that is the base container for the NodeJS container we build the Node-RED container on top of, but that is a VERY common base container so others would have seen it.)

seaverd commented 1 year ago

@hardillb

Here is the output of mount...my screenshot from previous posting was from the host os logs on boot.


rootfs on / type rootfs (rw,size=16218748k,nr_inodes=4054687,inode64)
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=32768k,mode=755,inode64)
/dev/sda1 on /boot type vfat (rw,noatime,nodiratime,fmask=0177,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,flush,errors=remount-ro)
/boot/bzfirmware on /lib/firmware type squashfs (ro,relatime,errors=continue)
overlay on /lib/firmware type overlay (rw,relatime,lowerdir=/lib/firmware,upperdir=/var/local/overlay/lib/firmware,workdir=/var/local/overlay-work/lib/firmware)
/boot/bzmodules on /lib/modules type squashfs (ro,relatime,errors=continue)
overlay on /lib/modules type overlay (rw,relatime,lowerdir=/lib/modules,upperdir=/var/local/overlay/lib/modules,workdir=/var/local/overlay-work/lib/modules)
hugetlbfs on /hugetlbfs type hugetlbfs (rw,relatime,pagesize=2M)
devtmpfs on /dev type devtmpfs (rw,relatime,size=8192k,nr_inodes=4054689,mode=755,inode64)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /dev/shm type tmpfs (rw,relatime,inode64)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
cgroup_root on /sys/fs/cgroup type tmpfs (rw,relatime,size=8192k,mode=755,inode64)
cpuset on /sys/fs/cgroup/cpuset type cgroup (rw,relatime,cpuset)
cpu on /sys/fs/cgroup/cpu type cgroup (rw,relatime,cpu)
cpuacct on /sys/fs/cgroup/cpuacct type cgroup (rw,relatime,cpuacct)
blkio on /sys/fs/cgroup/blkio type cgroup (rw,relatime,blkio)
memory on /sys/fs/cgroup/memory type cgroup (rw,relatime,memory)
devices on /sys/fs/cgroup/devices type cgroup (rw,relatime,devices)
freezer on /sys/fs/cgroup/freezer type cgroup (rw,relatime,freezer)
net_cls on /sys/fs/cgroup/net_cls type cgroup (rw,relatime,net_cls)
perf_event on /sys/fs/cgroup/perf_event type cgroup (rw,relatime,perf_event)
net_prio on /sys/fs/cgroup/net_prio type cgroup (rw,relatime,net_prio)
hugetlb on /sys/fs/cgroup/hugetlb type cgroup (rw,relatime,hugetlb)
pids on /sys/fs/cgroup/pids type cgroup (rw,relatime,pids)
tmpfs on /var/log type tmpfs (rw,relatime,size=131072k,mode=755,inode64)
cgroup on /sys/fs/cgroup/elogind type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib64/elogind/elogind-cgroups-agent,name=elogind)
rootfs on /mnt type rootfs (rw,size=16218748k,nr_inodes=4054687,inode64)
tmpfs on /mnt/disks type tmpfs (rw,relatime,size=1024k,inode64)
tmpfs on /mnt/remotes type tmpfs (rw,relatime,size=1024k,inode64)
tmpfs on /mnt/rootshare type tmpfs (rw,relatime,size=1024k,inode64)
nfsd on /proc/fs/nfs type nfsd (rw,relatime)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
/dev/md1 on /mnt/disk1 type xfs (rw,noatime,nouuid,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/md2 on /mnt/disk2 type xfs (rw,noatime,nouuid,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/md3 on /mnt/disk3 type xfs (rw,noatime,nouuid,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/md4 on /mnt/disk4 type xfs (rw,noatime,nouuid,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/md5 on /mnt/disk5 type xfs (rw,noatime,nouuid,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/md6 on /mnt/disk6 type xfs (rw,noatime,nouuid,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/sdj1 on /mnt/cache type xfs (rw,noatime,nouuid,attr2,inode64,logbufs=8,logbsize=32k,noquota)
shfs on /mnt/user0 type fuse.shfs (rw,nosuid,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other)
shfs on /mnt/user type fuse.shfs (rw,nosuid,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other)
/dev/sdl1 on /mnt/disks/WDBlack type xfs (rw,noatime,nodiratime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/mnt/cache/docker.img on /var/lib/docker type btrfs (rw,noatime,ssd,space_cache=v2,subvolid=5,subvol=/)
/mnt/cache/docker.img on /var/lib/docker/btrfs type btrfs (rw,noatime,ssd,space_cache=v2,subvolid=5,subvol=/)
/mnt/cache/libvirt.img on /etc/libvirt type btrfs (rw,noatime,ssd,space_cache=v2,subvolid=5,subvol=/)
nsfs on /run/docker/netns/4685a53f3126 type nsfs (rw)
nsfs on /run/docker/netns/3cefbd9ac391 type nsfs (rw)
nsfs on /run/docker/netns/9ee49aea37fd type nsfs (rw)
nsfs on /run/docker/netns/d1c0a32de9ff type nsfs (rw)
nsfs on /run/docker/netns/13ee7676e7db type nsfs (rw)
nsfs on /run/docker/netns/cb368106ad5d type nsfs (rw)
nsfs on /run/docker/netns/ec93e142dcea type nsfs (rw)
nsfs on /run/docker/netns/a781f7aa32bb type nsfs (rw)
nsfs on /run/docker/netns/dc46af377a0e type nsfs (rw)
nsfs on /run/docker/netns/58542d1f57e2 type nsfs (rw)
nsfs on /run/docker/netns/37bce51ee050 type nsfs (rw)
nsfs on /run/docker/netns/5369607aedae type nsfs (rw)
nsfs on /run/docker/netns/1b08deed6986 type nsfs (rw)
nsfs on /run/docker/netns/65f6ad46fa1b type nsfs (rw)
nsfs on /run/docker/netns/951d69dfe608 type nsfs (rw)
nsfs on /run/docker/netns/default type nsfs (rw)
nsfs on /run/docker/netns/f49a0386f25c type nsfs (rw)
nsfs on /run/docker/netns/fbb1d1576ca7 type nsfs (rw)
nsfs on /run/docker/netns/4564a75ca689 type nsfs (rw)
nsfs on /run/docker/netns/5aa429720492 type nsfs (rw)
nsfs on /run/docker/netns/aef4d70e9c0a type nsfs (rw)
nsfs on /run/docker/netns/5ab65eb85772 type nsfs (rw)
nsfs on /run/docker/netns/c39dd8693df7 type nsfs (rw)
nsfs on /run/docker/netns/ef49caa63672 type nsfs (rw)
nsfs on /run/docker/netns/5cc01aa0e7bf type nsfs (rw)
nsfs on /run/docker/netns/feccc2605152 type nsfs (rw)
nsfs on /run/docker/netns/838eddeb1222 type nsfs (rw)
nsfs on /run/docker/netns/e5df28b6bed6 type nsfs (rw)
nsfs on /run/docker/netns/8bb9f16f4d17 type nsfs (rw)
nsfs on /run/docker/netns/851011df61cd type nsfs (rw)
nsfs on /run/docker/netns/3f6a8a7bdd78 type nsfs (rw)
nsfs on /run/docker/netns/15efba4c626e type nsfs (rw)
nsfs on /run/docker/netns/bbb0160662db type nsfs (rw)
nsfs on /run/docker/netns/77bde563d2f0 type nsfs (rw)```
hardillb commented 1 year ago

Please put multi line stuff between triple backticks not just single (you can edit the last comment to add them).

seaverd commented 1 year ago

Done

hardillb commented 1 year ago

This appears to show the same problem. https://www.reddit.com/r/unRAID/comments/njs79e/psa_postgresql_doesnt_like_fuse_on_unraid/

I think you really will need to talk to the unraid folk

seaverd commented 1 year ago

OK, will start a post on Unraid forums once this gets moved as I'll be posting a link to this thread.

seaverd commented 1 year ago

@hardillb

Question, I haven't asked the unraid folks yet...just wondering if it is possible to set the cache directory with a separate mount as part of the docker command line?

hardillb commented 1 year ago

I've never tried to stack volume mounts ( e.g. volume1 on /data and volume2 on /data/.npm) you'd have to try it and see. I would suggest passing the -v '/mnt/user/appdata/nodered_new':'/data':'rw' first and -v '/some/other/path':'/data/.npm':'rw' second if you can influence the order.

hardillb commented 1 year ago

@seaverd moved (the old URL should redirect anyway)

fuzzy01 commented 1 year ago

Solution is to enable hard links in Unraid.

Go to Settings -> Global Share Settings -> Tunable (support hard links) and set it to Yes.