Cannot connect to adapter

I don't doubt you when you say it was working last night but, to be honest, I can't see how it ever worked like that.

I'll walk you through what I would do and you can see if the same approach works for you. I'll be doing this with a Sonoff ZDDongle-P but I believe the same principles work for any adapter.

First off, I assume you've read the information at Supported Adapters, have found your CC2652 variant in the list, and have then followed the instructions to set up the coordinator firmware. If you haven't done that, you should attend to it before doing anything else.

Next, I'm going to jump slightly ahead in time to #706. That Pull Request hasn't been applied to IOTstack yet but, seeing as you're just starting this journey, you may as well use the up-to-date service definition:

  zigbee2mqtt:
    container_name: zigbee2mqtt
    image: koenkk/zigbee2mqtt:latest
    environment:
      - TZ=${TZ:-Etc/UTC}
      - ZIGBEE2MQTT_CONFIG_MQTT_SERVER=mqtt://mosquitto:1883
      - ZIGBEE2MQTT_CONFIG_FRONTEND=true
      - ZIGBEE2MQTT_CONFIG_ADVANCED_LOG_SYMLINK_CURRENT=true
      # - DEBUG=zigbee-herdsman*
    ports:
      - "8080:8080"
    volumes:
      - ./volumes/zigbee2mqtt/data:/app/data
    devices:
      - "${ZIGBEE2MQTT_DEVICE_PATH:?eg echo ZIGBEE2MQTT_DEVICE_PATH=/dev/ttyACM0 >>~/IOTstack/.env}:/dev/ttyACM0"
    restart: unless-stopped
    depends_on:
      - mosquitto

If your Zigbee2MQTT container is running, take it down:

$ cd ~/IOTstack
$ docker-compose rm --force --stop -v zigbee2mqtt

Open your docker-compose.yml in a text editor and replace the existing service definition with the new one from above. Please don't preserve anything from your existing definition.

Also give the container a clean slate:

$ cd ~/IOTstack/volumes
$ sudo mv zigbee2mqtt zigbee2mqtt.off

I did that as a "rename" in case there's anything in the config you want to preserve and copy across later but please don't try to do any of that before we get everything running.

Disconnect your Zigbee adapter from your Pi.

Run the command:

$ lsusb

Connect your adapter, then run the same lsusb command again and compare the two sets of output to identify the line that got added when the device was connected. Here's my "after":

Bus 002 Device 002: ID 04e8:61f5 Samsung Electronics Co., Ltd Portable SSD T5
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 10c4:ea60 Silicon Labs CP210x UART Bridge
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

The line that got added is:

Bus 001 Device 003: ID 10c4:ea60 Silicon Labs CP210x UART Bridge

Using the bus (1) and device (3) from that line to form the "1:3" in the command below, run:

$ lsusb -s 1:3 -v | grep -e "idVendor" -e "idProduct" -e "iProduct" -e "iSerial"
Couldn't open device, some information will be missing
  idVendor           0x10c4 Silicon Labs
  idProduct          0xea60 CP210x UART Bridge
  iProduct                2 Sonoff Zigbee 3.0 USB Dongle Plus
  iSerial                 3 30b0eda67b12ec11805920c7bd930c07

ignore the error

The three critical bits of information in that are:

«vendorID» = "10c4" (drop the 0x)
«productID» = "ea60" (drop the 0x)
«serialNumber» = "30b0eda67b12ec11805920c7bd930c07"

Not all vendors provide unique serial numbers so don't worry if you don't see that.

Now we need a UDEV rule. Here's a template:

SUBSYSTEM=="tty", ATTRS{idVendor}=="«vendorID»", ATTRS{idProduct}=="«productID»", ATTRS{serial}=="«serialNumber»", SYMLINK+="«deviceName»"

Use a text editor. Copy that template into the editor then replace all the «fields» with the relevant values.

If you didn't get a sensible serial number from the last lsusb command then remove the ATTRS{serial}=="«serialNumber»", from the final result.

You need to come up with a sensible name for the «deviceName» field. Here's my completed example:

SUBSYSTEM=="tty", ATTRS{idVendor}=="10c4", ATTRS{idProduct}=="ea60", ATTRS{serial}=="30b0eda67b12ec11805920c7bd930c07", SYMLINK+="Zigbee_Sonoff_P"

Save the file with a name that follows this kind of pattern:

88-tty-«description».rules

My file name is:

88-tty-zigbee-sonoff-p.rules

The idea is to be able to identify a rule easily if you ever need to change anything.

If your text editor was running on something other than your Pi (eg Windows, macOS) then you need to get that file onto your Pi. I'm going to assume it will be in your home directory on the Pi.

If you used a Windows text editor, you need make sure the file has proper LF (0x0A) line endings and doesn't have Windows' CR+LF (0x0D0A) line endings. One way of doing that is with the dos2unix tool. Something like this will do the job:

$ dos2unix 88-tty-zigbee-sonoff-p.rules

The file needs to be owned by root with mode 644 so:

$ chmod 644 88-tty-zigbee-sonoff-p.rules
$ sudo chown root:root 88-tty-zigbee-sonoff-p.rules

The last step is to move the rule to the right place:

$ sudo mv 88-tty-zigbee-sonoff-p.rules /etc/udev/rules.d/

Some guides tell you to restart the service at this point but I've never found that to be necessary on the Pi. Rules files are active as soon as you create or edit them.

Disconnect your adapter then re-connect it and list your devices directory:

$ ls /dev

With any luck you'll spot the name you chose for «deviceName» in the list. Mine is Zigbee_Sonoff_P so I can just look for that in long form:

$ ls -l /dev/Zigbee_Sonoff_P
lrwxrwxrwx 1 root root 7 Jun 22 13:38 /dev/Zigbee_Sonoff_P -> ttyUSB0

So, what did all that do? Answer, it provided a way for the Pi to identify the adapter that is human-readable and persistent, and which doesn't involve any gobbledegook. The alternative for my Sonoff is:

$ ls -l /dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4:1.0-port0 
lrwxrwxrwx 1 root root 13 Jun 22 13:38 /dev/serial/by-path/platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.4:1.0-port0 -> ../../ttyUSB0

Note the colons in that path. As soon as I see something like colons I wonder how that's going to work in a device: clause in docker-compose, where a colon has a special meaning. Those will probably need to be escaped (\:) and might be hard to get right. The whole thing makes me itch!

Also, that platform-… name doesn't exactly scream "here be a Zigbee adapter". I reckon the scream it evokes is "what the heck is that?"

Anyway…

Back to the task at hand. If everything has gone right, you'll have the equivalent of /dev/Zigbee_Sonoff_P turning up whenever your adapter is connected (and going away whenever your adapter is disconnected).

Now, do this:

$ cd ~/IOTstack
$ cat .env

With any luck, you'll get a time-zone definition. Mine is:

TZ=Australia/Sydney

If you don't get at least a timezone (or if cat says "no such file or directory" then it's probably a good idea to set that up:

$ echo "TZ=US/Eastern" >>.env

And then define your Zigbee adapter like this:

$ echo "ZIGBEE2MQTT_DEVICE_PATH=/dev/Zigbee_Sonoff_P" >>.env

So, what did all that do? Well, the new Zigbee service definition, plus an increasing number of other service definitions, have this:

    environment:
      - TZ=${TZ:-Etc/UTC}

It works just like bash substitution. If the TZ from ${TZ is defined in ~/IOTstack/.env then whatever is in the .env file is used, otherwise the timezone defaults to Etc/UTC. Once all service definitions have this construct, there will no longer be any need to edit the TZ in every service. Just do it once in .env!

Similarly, the new Zigbee service definition has:

    devices:
      - "${ZIGBEE2MQTT_DEVICE_PATH:?eg echo ZIGBEE2MQTT_DEVICE_PATH=/dev/ttyACM0 >>~/IOTstack/.env}:/dev/ttyACM0"

If you try to bring up the container without defining ZIGBEE2MQTT_DEVICE_PATH in .env you get:

parsing /home/pi/IOTstack/docker-compose.yml: error while interpolating services.zigbee2mqtt.devices.[]: required variable ZIGBEE2MQTT_DEVICE_PATH is missing a value: eg echo ZIGBEE2MQTT_DEVICE_PATH=/dev/ttyACM0 >>~/IOTstack/.env

It's a long-winded and very messy message but at least it drags your attention to the right place (the zigbee2mqtt service) and gives you an example command for fixing the problem using an echo command.

We've already done the step of adding ZIGBEE2MQTT_DEVICE_PATH to the .env file so, after docker-compose has performed the substitution, the whole clause reduces to:

    devices:
      - "/dev/Zigbee_Sonoff_P:/dev/ttyACM0"

In words, what that means is the external device (the left hand side /dev/Zigbee_Sonoff_P) is mapped to the internal device (the right hand side /dev/ttyACM0).

The Zigbee2MQTT container expects the adapter to appear on /dev/ttyACM0 inside the container. It doesn't matter which type of adapter is actually attached to the Pi. It could be like mine (a Sonoff) or a CC2531 or anything else. Whatever the Pi sees is independent of what the container sees.

This is probably a good place to explain why I said I couldn't see how your setup ever worked like that. You had this:

    devices:
      - /dev/ttyAMA0: /dev/serial/by-id/usb-Silicon_Labs_slae.sh_cc2652rb_stick_-_slaesh_s_iot_stuff_00_12_4B_00_21_CC_00_43-if00-port0

The space after the colon worries me but I don't know if it's significant. I'm going to ignore that.

What that means is the external device /dev/ttyAMA0 is mapped to the internal device /dev/serial/by-id….

You've then tried to override the container's expectation that the adapter (irrespective of its actual model) will always appear at /dev/ttyACM0 by editing the config file.

The thing is, the external device /dev/ttyAMA0 is the Raspberry Pi's Bluetooth adapter. While the rest of what you did may have succeeded in mapping the Bluetooth adapter to the /dev/serial/by-id… device inside the container, the thing at the other end is not a Zigbee adapter.

Why is /dev/ttyAMA0 used if it means "Bluetooth"? It's a placeholder from a convention we are trying to move away from. There's background at #690.

Also, if you really want to override the container's expectation that the adapter always appears at /dev/ttyACM0, the correct way to do that is with an environment variable:

- ZIGBEE2MQTT_CONFIG_SERIAL_PORT=

By correct I mean when Zigbee2MQTT is running in a Docker container. If Zigbee2MQTT was running natively then, sure, editing the config file would be the way to tackle this problem. But when it's running in a container, environment variables are the appropriate mechanism. Not all containers have a full 1:1 mapping between configuration options and environment variables but it's considered "best practice" to do it that way. As far as I'm aware, Zigbee2MQTT does have full 1:1.

However, in practice, it isn't necessary to override the serial port and I can't think of a situation where it would be all that useful.

I hope that this also helps you to understand why I said to rename the ~/IOTstack/volumes/zigbee2mqtt folder. If you don't do that, the container will try to use your edited configuration when we bring it up, and it will get confused.

So, here we go (on my system). First, prove no containers are running:

$ docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

What's defined in my .env?

$ cd ~/IOTstack
$ cat .env
TZ=Australia/Sydney
ZIGBEE2MQTT_DEVICE_PATH=/dev/Zigbee_Sonoff_P

Prove the Pi knows about the adapter:

$ ls -l /dev/Zigbee_Sonoff_P
lrwxrwxrwx 1 root root 7 Jun 22 15:30 /dev/Zigbee_Sonoff_P -> ttyUSB0

Prove the container will have a clean slate when it starts (and will have to initialise its persistent storage from scratch):

$ ls -ld ./volumes/zigbee2mqtt
ls: cannot access './volumes/zigbee2mqtt': No such file or directory

Bring up the container:

$ docker-compose up -d zigbee2mqtt
[+] Running 3/3
 ✔ Network iotstack_default  Created                                                                                                                       0.2s 
 ✔ Container mosquitto       Started                                                                                                                       1.0s 
 ✔ Container zigbee2mqtt     Started                                                                                                                       1.5s

Mosquitto auto-started because of the depends_on clause. The Zigbee2MQTT container takes a while to get going but, eventually:

$ docker logs zigbee2mqtt
Using '/app/data' as data directory
Creating configuration file...
Zigbee2MQTT:info  2023-06-22 15:34:36: Logging to console and directory: '/app/data/log/2023-06-22.15-34-36' filename: log.txt
Zigbee2MQTT:info  2023-06-22 15:34:36: Starting Zigbee2MQTT version 1.31.2 (commit #21f5125)
Zigbee2MQTT:info  2023-06-22 15:34:36: Starting zigbee-herdsman (0.14.117)
Zigbee2MQTT:info  2023-06-22 15:34:49: zigbee-herdsman started (resumed)
Zigbee2MQTT:info  2023-06-22 15:34:49: Coordinator firmware version: '{"meta":{"maintrel":1,"majorrel":2,"minorrel":7,"product":1,"revision":20230507,"transportrev":2},"type":"zStack3x0"}'
Zigbee2MQTT:info  2023-06-22 15:34:49: Currently 0 devices are joined:
Zigbee2MQTT:warn  2023-06-22 15:34:49: `permit_join` set to  `true` in configuration.yaml.
Zigbee2MQTT:warn  2023-06-22 15:34:49: Allowing new devices to join.
Zigbee2MQTT:warn  2023-06-22 15:34:49: Set `permit_join` to `false` once you joined all devices.
Zigbee2MQTT:info  2023-06-22 15:34:49: Zigbee: allowing new devices to join.
Zigbee2MQTT:info  2023-06-22 15:34:49: Connecting to MQTT server at mqtt://mosquitto:1883
Zigbee2MQTT:info  2023-06-22 15:34:49: Connected to MQTT server
Zigbee2MQTT:info  2023-06-22 15:34:49: MQTT publish: topic 'zigbee2mqtt/bridge/state', payload 'online'
Zigbee2MQTT:info  2023-06-22 15:34:49: Started frontend on port 0.0.0.0:8080
Zigbee2MQTT:info  2023-06-22 15:34:49: MQTT publish: topic 'zigbee2mqtt/bridge/config', payload '{"commit":"21f5125","coordinator":{"meta":{"maintrel":1,"majorrel":2,"minorrel":7,"product":1,"revision":20230507,"transportrev":2},"type":"zStack3x0"},"log_level":"info","network":{"channel":11,"extendedPanID":"0xdddddddddddddddd","panID":6754},"permit_join":true,"version":"1.31.2"}'
Zigbee2MQTT:info  2023-06-22 15:34:49: Zigbee2MQTT started!

It's up, has found the adapter, has connected to Mosquitto, and has published a payload.

Unfortunately, it doesn't actually mention /dev/ttyACM0 anywhere in the log. If you want to prove to yourself that that's the device it is using, you need to enable the - DEBUG=zigbee-herdsman* environment variable and "up" the container again. If you do that then you see:

Zigbee2MQTT:debug 2023-06-22 15:37:07: Using zigbee-herdsman with settings: '{"adapter":{"concurrent":null,"delay":null,"disableLED":false},"backupPath":"/app/data/coordinator_backup.json","databaseBackupPath":"/app/data/database.db.backup","databasePath":"/app/data/database.db","network":{"channelList":[11],"extendedPanID":[221,221,221,221,221,221,221,221],"networkKey":"HIDDEN","panID":6754},"serialPort":{"path":"/dev/ttyACM0"}}'

Last point. Here's my freshly-initialised config file:

$ cat ~/IOTstack/volumes/zigbee2mqtt/data/configuration.yaml
# Home Assistant integration (MQTT discovery)
homeassistant: false

# allow new devices to join
permit_join: true

# MQTT settings
mqtt:
  # MQTT base topic for zigbee2mqtt MQTT messages
  base_topic: zigbee2mqtt
  # MQTT server URL
  server: 'mqtt://localhost'
  # MQTT server authentication, uncomment if required:
  # user: my_user
  # password: my_password

# Serial settings
serial:
  # Location of CC2531 USB sniffer
  port: /dev/ttyACM0

Notice the "server:" field pointing to 'mqtt://localhost'. Then go back to the log output above and find:

Connecting to MQTT server at mqtt://mosquitto:1883

The mosquitto:1883 is coming from the environment variable in the service definition:

      - ZIGBEE2MQTT_CONFIG_MQTT_SERVER=mqtt://mosquitto:1883

What actually happens is:

The Zigbee2MQTT process starts
If there is no config file, it is initialised from an internal default.
Then the config file is read.
Then the environment variables are processed.

So, in the case of the "server" field, the presence of ZIGBEE2MQTT_CONFIG_MQTT_SERVER overrode the default of mqtt://localhost.

Conversely, if your existing config file had been left in place then the serial port would have been defined as /dev/serial/by-id… but, because there was no ZIGBEE2MQTT_CONFIG_SERIAL_PORT to override it, the by-id value would have been used, which wouldn't have gone anywhere.

By letting the container re-initialise its persistent store, the default of /dev/ttyACM0 takes effect, which maps through to the adapter.

Hope this helps you to get going.

SensorsIot / IOTstack

Cannot connect to adapter #717

- /dev/ttyAMA0:/dev/ttyACM0