USB serial input

[Ratsea34]

I have an EON energy monitor with a USB output, form which I collect my electrical usage data. It has worked fine on Buster/Node Red 2.03. I wanted to upgrade the OS to Bullseye and the latest Node Red (3.02) so I built a new SSD, I now cannot access the Serial In node (1.03) - it reports No Connection, rather than Not connected. I can see the data stream in Minicom. It also crashes Node Red quite easily with a segment fault. Any thoughts?

[Paraphraser]

Please include your nodered service definition here (copy it from your docker-compose.yml and paste it between lines of triple back-ticks).

[Ratsea34]

The Node Red section is this:

nodered:
    container_name: nodered
    build:
      context: ./services/nodered/.
      args:
      - DOCKERHUB_TAG=latest
      - EXTRA_PACKAGES=
    restart: unless-stopped
    user: "0"
    environment:
    - TZ=Etc/UTC
    ports:
    - "1880:1880"
    volumes:
    - ./volumes/nodered/data:/data
    - ./volumes/nodered/ssh:/root/.ssh
    - /var/run/docker.sock:/var/run/docker.sock
    - /var/run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket
    devices:
    - "/dev/ttyAMA0:/dev/ttyAMA0"
    - "/dev/vcio:/dev/vcio"
    - "/dev/gpiomem:/dev/gpiomem"

I have tried adding - "/dev/ttyUSB0:/dev/ttyUSB0" but it doesn't run - file not found even if the USB is plugged in

ERROR: for nodered  Cannot start service nodered: error gathering device information while adding custom device "/dev/ttyUBS0": no such file or directory

[Paraphraser]

Try changing the service definition to:

DOCKERHUB_TAG=latest-14

The -14 suffix causes NodeJS version 14 to be selected rather than the default of version 16.

To implement:

$ cd ~/IOTstack
$ docker-compose up --build -d nodered
$ docker system prune -f

A lot of things seem to be a bit wonky in NodeJS 16 which is why I'm sticking with NodeJS 14 and "awaiting developments".

A lot of people report privilege conflicts when they try to upgrade Node-RED. We (IOTstack) tend not to see those because of the user: "0" which overrides Node-RED's attempt to downgrade its privileges at runtime. But the people reporting problems always seem to be running "latest" (implying NodeJS 16) which is why -14 is my go-to solution.

A few weeks/months back I did a simple test with an ESP32 development board (which shows up as a CP2102 on ttyUSB0). The sketch just wrote millis() to the serial port every couple of seconds. A Node-RED flow was able to display what was being received. It's not exactly a conclusive test but it didn't crash under Bullseye and -14.

Give it a whirl and see what happens.

[Ratsea34]

Version 14 makes no difference and neither does 12. I looked at the syslog and I get this, suggesting the device isn't recognised, but Minicom is receiving the data perfectly.

Apr  9 09:53:05 raspberrypi kernel: [    2.307581] usb 1-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Apr  9 09:53:05 raspberrypi kernel: [    2.307597] usb 1-1.4: Product: USB-Serial Controller
Apr  9 09:53:05 raspberrypi kernel: [    2.307611] usb 1-1.4: Manufacturer: Prolific Technology Inc.
Apr  9 09:53:05 raspberrypi kernel: [    6.481611] usbcore: registered new interface driver usbserial_generic
Apr  9 09:53:05 raspberrypi kernel: [    6.482786] usbserial: USB Serial support registered for generic
Apr  9 09:53:05 raspberrypi kernel: [    6.531496] usbcore: registered new interface driver pl2303
Apr  9 09:53:05 raspberrypi kernel: [    6.531646] usbserial: USB Serial support registered for pl2303
Apr  9 09:53:05 raspberrypi kernel: [    6.560817] usb 1-1.4: pl2303 converter now attached to ttyUSB0
Apr  9 09:53:05 raspberrypi kernel: [    7.390962] usbcore: registered new interface driver brcmfmac
Apr  9 09:53:09 raspberrypi ModemManager[517]: <info>  [base-manager] couldn't check support for device '/sys/devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.4': not supported by any plugin

Also the NRed flow gives this:

"[serialconfig:5fc33cf80db10df9] serial port /dev/ttyUSB0 error: Error: Error: No such file or directory, cannot open /dev/ttyUSB0"

[Paraphraser]

Ah. Is the serial node add-on defined in your Dockerfile?

$ cat IOTstack/services/nodered/Dockerfile
# reference argument - omitted defaults to latest
ARG DOCKERHUB_TAG=latest

# Download base image
FROM nodered/node-red:${DOCKERHUB_TAG}

# reference argument - omitted defaults to null
ARG EXTRA_PACKAGES
ENV EXTRA_PACKAGES=${EXTRA_PACKAGES}

# default user is node-red - need to be root to install packages
USER root

# install packages
RUN apk update && apk add --no-cache eudev-dev ${EXTRA_PACKAGES}

# switch back to default user
USER node-red

# variable not needed inside running container
ENV EXTRA_PACKAGES=

# add-on nodes follow

RUN npm install \ 
  node-red-node-pi-gpiod \
  node-red-dashboard \
  node-red-contrib-influxdb \
  node-red-contrib-boolean-logic \
  node-red-node-tail \
  node-red-configurable-ping \
  node-red-node-email \
  node-red-contrib-boolean-logic-ultimate \
  node-red-contrib-chartjs \
  node-red-contrib-md5 \
  node-red-contrib-moment \
  node-red-contrib-tibber-api \
  node-red-node-serialport \
  node-red-contrib-pushsafer

$

The above is mine. The first part is the same as comes out of the IOTstack template but I do the last part as a single command spread over continuation lines - so that I only get a single layer added to the local image, irrespective of the number of add-on nodes. Either method will work but you will need node-red-node-serialport in the list. Rebuilding the local image is the same command as before.

You might also want to run:

$ ~/IOTstack/scripts/nodered_list_installed_nodes.sh

in case any duplicates show up.

[Ratsea34]

Yes I do have node-red-node-serialport in my Dockerfile. I ran the dups check and did remove one for serialport. Originally I had upgraded NR from 2.0 to 3.0 but still on Buster, it failed then, so I had decided to reinstall a new OS and IOTstack. It is still working fine on my old RPi3 with Buster and NR 2..20 with a Green spot - connected - under the Serial In node. If I unplug the USB then it goes Red - not connected - as opposed to Grey - not connected. Perhaps I should just start afresh and reinstall everything again.

[Paraphraser]

OK. Figured it out.

1. Googled: ./entrypoint.sh Segmentation fault nodered serial
2. Useful hit.
3. Useful hit.

I ignored the "rebuild from source" and other nonsense. Life's too short for that kind of thing on a Raspberry Pi.

The magic incantation is to pin the version in the Dockerfile:

node-red-node-serialport@0.15.0

I didn't find this written anywhere so it's pure speculation but, reading between the lines, I'd say the latest version of node-red-node-serialport is incompatible with current versions of Node-RED, and the maintainers of the latter expect the maintainers of the former to sort it out (and possibly vice-versa). I reckon that best explains the relatively long time this has been going on with no movement on the Node-RED front.

My test vehicle was an ESP32 development board. A simple sketch that bungs out millis() every five seconds, along with anything received on the serial port in the meantime. Mounts on the Pi as /dev/ttyUSB0 so I added:

    - "/dev/ttyUSB0:/dev/ttyUSB0"

So, the serial-in node is just dumping whatever it receives, which is the millis() output every 5 seconds. Clicking the Inject node sends the current time out the door and that comes back the next time the ESP32 speaks (an example is the 3rd row of debug output).

For the record:

4GB Raspberry Pi 4 Model B Rev 1.1 running Debian GNU/Linux 11 (bullseye) as full 64-bit OS

$ uname -a
Linux tri-dev 6.1.19-v8+ #1637 SMP PREEMPT Tue Mar 14 11:11:47 GMT 2023 aarch64 GNU/Linux

$ docker exec nodered npm version --json | jq -r '[.["node-red-docker"],.["node"]] | @tsv'
3.0.2   14.20.0

Hope this solves your problem.

[Ratsea34]

Thanks Phill, provided I keep the USB connected that all works now. I am still running Serial version 0.11.1 on the old 'live' version, and that works too with Bullseye/NR3.0. Just for info, by removing the USB input, Docker-compose doesn't run - File not found (/dev/ttyUSB0) so you have to comment it out. It seems that the NR 2.2 can map to the ttyUSB0 port without the mapping in Docker-compose, so it still builds without the USB inserted. Thanks again.

[Paraphraser]

The problem of what happens when a device disconnects or, worse, device enumeration changes the "n" in "/dev/ttyUSBn" is pretty common.

I'll give you two more pointers which you may or may not find helpful:

1. octoprint-docker: when your 3D printer turns on and off
2. IOTstack Tutorial: Logging CPU Temperatures

The first deals with the problem of devices coming and going. Basically:

1. A UDEV rule senses when the 3D printer appears/disappears. The rule:

   • gives the printer a predictable name; and
   • triggers a reaction script running inside the container

2. Via the service definition, the container can:

   • "see" the Pi's /dev (read-only); and
   • "control" a certain range of devices via their major number.

Printer connects. UDEV rule gives it a name (eg /dev/AnyCubicI3Mega). UDEV rule triggers reaction script inside container. Container compares its /dev with the host's /dev, sets up a matching /dev/AnyCubicI3Mega inside the container (ie exactly what would happen if you just mapped /dev/AnyCubicI3Mega:/dev/AnyCubicI3Mega, and restarts the process which becomes aware of the change.

Printer goes away and the inverse happens to tear it all down. The Octoprint container stays up 24x7 to accept queued jobs etc, and doesn't crash every time the printer is switched on or off.

You might be able to adapt some of the techniques to Node-RED.

However, my first thought would be that restarting the Octoprint container as the printer comes and goes might be OK. Waiting a little while for Octoprint to become ready doesn't amount to much. But Node-RED is a different kettle of fish because you probably have a lot of flows doing a lot of things. Interrupting those just because a device went through a power-cycle would never be my first choice.

Over at OctoPrint/octoprint-docker you'll find issues with other variations on this theme. You'll also probably notice feedback from that repo's maintainer where he gets a bit crotchety if someone does something like maps /dev either with or without the read-only flag. His explanation is generally expressed as "bad - for all sorts of reasons" but I have yet to see any elaboration on what those reasons might be or how any risks might be mitigated. I really don't see how a Pi being given read-only access to /dev with holes punched in the read-only via cgroup rules is any big deal. It's not like every container has been given that access so, as far as I can see, the problem is the "what if?" scenario of malware finding its way into the Octoprint container. Seems to me the most such malware could do would be to futz with the firmware on your 3D printer, and it could do that irrespective of whether it's getting access via cgroup rules or has a direct device mapping. In short, I suggest taking the negative vibes with a few grains of salt. On the other hand, if you can spot any serious risks, I'd like to hear about them.

So that brings us to the second link above which is less about the mechanics of recording Pi temperatures and more an example of working with MING (Mosquitto, Influx, Node-RED, Grafana).

The key idea is keeping Node-RED the heck away from the hardware (USB ports, GPIO pin headers, or trying to read the CPU temperature any differently on this machine just because it also happens to be the machine where Node-RED is running). It's about using MQTT (first choice) or HTTP (second choice), and only resorting to Node-RED (third choice) if the first two options are not viable.

In my case, I have a Hiking DDS238-2 ZN/S Class 1 electricity meter. It speaks RS485. I built a project consisting of an ESP32 and a serial-to-RS485 adapter. The ESP32 polls the meter and sends updates via MQTT at 10-second intervals:

2023-04-10T11:44:20+1000 /merle/box-hiking/grid {"T01":513.16,"T02":0.00,"a":0.00,"v":251.20,"w":0.00,"pf":1.00,"f":50.03}
2023-04-10T11:44:30+1000 /merle/box-hiking/grid {"T01":513.16,"T02":0.00,"a":0.00,"v":251.10,"w":0.00,"pf":1.00,"f":50.04}
2023-04-10T11:44:40+1000 /merle/box-hiking/grid {"T01":513.16,"T02":0.00,"a":0.00,"v":251.30,"w":0.00,"pf":1.00,"f":50.00}
2023-04-10T11:44:50+1000 /merle/box-hiking/grid {"T01":513.16,"T02":0.00,"a":0.00,"v":250.90,"w":0.00,"pf":1.00,"f":50.05}

Normally, the topic is just "/merle/hiking/grid". Some electrical changes after a solar inverter replacement meant that the Hiking meter I had inline in the house breaker board became inaccessible until I get some follow-up work done. In the meantime, a separate meter sitting in a box connected to a power point gives me voltage and frequency but not much in the way of current draw - hence the "a":0.00.

Anyway, the Node-RED flow subscribes to that topic and formats the JSON payload into an InfluxDB insert.

Let's suppose I have all this running on Buster and a fairly ancient version of Node-RED. Now suppose I build a Bullseye system and bring my flows across. I haven't had to go near the Hiking meter and ESP32 system. That's gone right on sending. If I'm building Bullseye on a separate Pi, I can have my new instance of Node-RED subscribe to my old MQTT broker on the Buster machine. If I really want to test the new MQTT broker, I can either change the code on the ESP32 to send two MQTT updates, or set up a flow on the Buster machine to relay that topic to the new machine.

By keeping Node-RED away from the hardware, I gain flexibility and testability before I need to commit to any changes. And I never ever have to worry about whether Buster or Bullseye or Bookend, or a Node-RED update, or a NodeJS version breaks something at the hardware level.

Just sayin' …

[Ratsea34]

Hi Phill, Thanks again, and certainly a good idea. I use a number of ESP8266 with Tasmota connected via MQTT to do my other sensors, but this output is a nine element XML which I filter and convert in Node red.

What I do find curious though is my serial input still works without any config changes to Dockerfile/Docker-compose.yml/ or even adding Node-red-Serial to the add-ons list.

This is Buster and NR 2.2 and Serial 0.11.0. And it's not fussy about the USB being present or not. I do know that it broke with NR3.0 and Serial 1.03 but I didn't try them separately.

Loading the new Bullseye/NR 3.02 system with the old backup data so as to retain the history isn't quite straightforward as it need several goes to get right Serial version installed. I suspect the Serial add-on 1.03 is the culprit despite being only 2 month old.