search

gcgarner / IOTstack

docker stack for getting started on IOT on the Raspberry PI
GNU General Public License v3.0
1.52k stars 582 forks source link

Unexplained Node-Red container behaviour on docker-compose pull #116

Closed Paraphraser closed 4 years ago

Paraphraser commented 4 years ago

I was chasing a problem (unrelated to this issue) and had run up a brand new IOTstack on a Raspberry Pi 3B+, mirroring the setup for my "live" system on the RPi4B. Along the way I did this:

$ cd ~/IOTstack
$ docker-compose ps

  Name                 Command                  State                                                 Ports                                          
-----------------------------------------------------------------------------------------------------------------------------------------------------
grafana     /run.sh                          Up             0.0.0.0:3000->3000/tcp                                                                   
influxdb    /entrypoint.sh influxd           Up             0.0.0.0:2003->2003/tcp, 0.0.0.0:8083->8083/tcp, 0.0.0.0:8086->8086/tcp                   
mosquitto   /docker-entrypoint.sh /usr ...   Up             0.0.0.0:1883->1883/tcp, 0.0.0.0:9001->9001/tcp                                           
nodered     npm start -- --userDir /data     Up (healthy)   0.0.0.0:1880->1880/tcp                                                                   
pihole      /s6-init                         Up (healthy)   443/tcp, 0.0.0.0:53->53/tcp, 0.0.0.0:53->53/udp, 0.0.0.0:67->67/udp, 0.0.0.0:8089->80/tcp
portainer   /portainer                       Up             0.0.0.0:9000->9000/tcp

I'm used to only seeing "(healthy)" associated with PiHole so NodeRed gaining such a tag made me sit up and pay attention.

The logical inference is that the nodered pulled onto the newly-built RPi3B+ was a later version.

A bit more digging confirmed that nodered wasn't the only image that needed to be re-pulled.

Fresh system (RPi3B+, built 2019-12-13):

$ docker-compose images
Container       Repository         Tag       Image Id      Size  
-----------------------------------------------------------------
grafana     grafana/grafana       6.3.6    b28afd63765e   149 MB 
influxdb    influxdb              latest   dfd71026b4c9   256 MB 
mosquitto   eclipse-mosquitto     latest   b8c6781b2d13   5.5 MB 
nodered     iotstack_nodered      latest   31bf35788ae0   381 MB 
pihole      pihole/pihole         latest   1ad60393d9c2   278 MB 
portainer   portainer/portainer   latest   8971979f760c   61.6 MB

"Production" system (RPi4B, built circa 2019-11-17):

$ docker-compose images
Container       Repository         Tag       Image Id      Size  
-----------------------------------------------------------------
grafana     grafana/grafana       6.3.6    b28afd63765e   149 MB 
influxdb    influxdb              latest   3f215d0c31e9   215 MB 
mosquitto   eclipse-mosquitto     latest   d7a0e73ad7a1   5.5 MB 
nodered     iotstack_nodered      latest   caf3e3ab51d5   376 MB 
pihole      pihole/pihole         latest   1ad60393d9c2   278 MB 
portainer   portainer/portainer   latest   a7dbced775e5   60.9 MB

I read that as telling me that the only images that have NOT changed are Grafana and PiHole, the inverse of that implying that Influxdb, Mosquitto, Node-Red and Portainer have updates.

So far, so good.

$ docker-compose pull
Pulling portainer ... done
Pulling nodered   ... done
Pulling influxdb  ... done
Pulling grafana   ... done
Pulling mosquitto ... done
Pulling pihole    ... done

$ docker-compose up -d
grafana is up-to-date
Recreating mosquitto ... done
pihole is up-to-date
Recreating portainer ... done
nodered is up-to-date
Recreating influxdb ... done

Influxdb, Mosquitto and Portainer have been updated but Node-Red has been left untouched. Que? Double-check:

$ docker-compose images
Container       Repository         Tag       Image Id      Size  
-----------------------------------------------------------------
grafana     grafana/grafana       6.3.6    b28afd63765e   149 MB 
influxdb    influxdb              latest   dfd71026b4c9   256 MB 
mosquitto   eclipse-mosquitto     latest   b8c6781b2d13   5.5 MB 
nodered     iotstack_nodered      latest   caf3e3ab51d5   376 MB 
pihole      pihole/pihole         latest   1ad60393d9c2   278 MB 
portainer   portainer/portainer   latest   8971979f760c   61.6 MB

The Node-Red image ID of "caf3e3ab51d5" is unchanged and doesn't match the image ID on the RPi3B+ of "31bf35788ae0". Even some brute force (stop, remove, pull, start) won't budge it.

I'm guessing that this gets back to #87 and the Wiki Node-Red topic "unused-node-in-protainer".

If you happen to be editing that page in the Wiki, you could correct that minor typo ("Protainer" -> "Portainer").

How should I go about updating the Node-Red container?

The "how-to" for this probably belongs in the Wiki too.

Doesn't this also imply that ~/IOTstack/scripts/update.sh doesn't do the whole job?

gcgarner commented 4 years ago

Hello Phil

Nodered is made a little differently. I uses a dockerfile to inject the preinstalled nodes. I've looked at alternatives to get past that behaviour.

I may have accidentality cut out the update instructions for nodered when I moved it to the wiki. Because it uses a dockerfile you will need to run docker-compose build after the pull. Once the build is done you can run docker-compose up -d

I think i need to update the update script to include the build and an up -d to do all the required actions.

Paraphraser commented 4 years ago

That still doesn't seem to be enough. The "build" seems to do some work involving Node-Red but the following "up" doesn't do anything. The "images" list still gives an ID of "caf3e3ab51d5" which is the number we first thought of. I didn't necessarily expect any new ID to match the one on the RPi3B+ but I did expect a change, so I'm concluding "no effect". Reasonable?

$ cd ~/IOTstack/

$ docker-compose build
portainer uses an image, skipping
Building nodered
Step 1/3 : FROM nodered/node-red:latest
 ---> b04b04ad7675
Step 2/3 : RUN for addonnodes in node-red-node-pi-gpiod node-red-dashboard node-red-contrib-influxdb ; do npm install ${addonnodes} ;done;
 ---> Using cache
 ---> d02b48939d52
Step 3/3 : RUN npm install --unsafe-perm node-red-node-sqlite
 ---> Using cache
 ---> caf3e3ab51d5
Successfully built caf3e3ab51d5
Successfully tagged iotstack_nodered:latest
influxdb uses an image, skipping
grafana uses an image, skipping
mosquitto uses an image, skipping
pihole uses an image, skipping

$ docker-compose up -d
mosquitto is up-to-date
pihole is up-to-date
portainer is up-to-date
grafana is up-to-date
nodered is up-to-date
influxdb is up-to-date

$ docker-compose images
Container       Repository         Tag       Image Id      Size  
-----------------------------------------------------------------
grafana     grafana/grafana       6.3.6    b28afd63765e   149 MB 
influxdb    influxdb              latest   dfd71026b4c9   256 MB 
mosquitto   eclipse-mosquitto     latest   b8c6781b2d13   5.5 MB 
nodered     iotstack_nodered      latest   caf3e3ab51d5   376 MB 
pihole      pihole/pihole         latest   1ad60393d9c2   278 MB 
portainer   portainer/portainer   latest   8971979f760c   61.6 MB
gcgarner commented 4 years ago

Hello Phil

I've been looking into this. It looks like the Dockerfile for nodered is interfering with the pull. Due to the build process the pull then finds the nodered image locally as "iotstack_nodered" and as far as it is concerned that is up to date.

Next year I'm going to work on removing the need for the Dockerfile so that this is no longer an issue. Now the interim work around would be to stop the container, delete the "iotstack_nodered" image and rebuild or just run docker-compose up -d and it will redo it

something else that'd odd, even though i completely removed all my nodered images and did a full rebuild i dont get the "healthy" state on my RPi4, not even after removing the volume.

Paraphraser commented 4 years ago

Hi Graham,

I've been digging into this a little further and just stopping the "iotstack_nodered" container isn't enough because Docker won't let me remove the images.

The only sequence of commands I've found to be reliable (in the sense of being guaranteed to bring an "older" installation to the point where it mirrors a "new and clean" installation) is:

$ cd ~/IOTstack
$ docker-compose down
$ docker rmi "iotstack_nodered" "nodered/node-red"
$ docker-compose pull
$ docker-compose up --build -d

For those playing along at home and who, like me, have SQLite installed as part of Node-Red, the "build" step for Node-Red takes a small fraction of eternity to complete, emits squillions of compiler warnings as it does so, stutters a fair bit and, at one point, gives every indication of having hung on these lines:

Release/obj/gen/sqlite-autoconf-3300100/sqlite3.c:182556:5: note: here
     default: {
     ^~~~~~~

Coffee and boundless patience pays off!