Docker container or home assistant

[RayDog1995]

Love the work yall have done. Couple feature questions since im not sure if the amount of work is justifiable. Could this project be containerized? I also run home assistant (container), wondering if turning this into a container is needed if it could just be run as an integration in HA. I think the first option would be used more by the community and easier to achieve. The second option, the advantage i see with trying to use in HA is being able to use the bluetooth proxy feature (esp relay BT back to HA via wifi) so anywhere on ones premises (or in the world if setup for it) can auto collect data from these usb meters. Also curious to know if multiple usb meters could be used simultaneously. Ill see how far I get making a dockerfile, never made one so lots of doubt haha. (This is my first request so im sorry if I didnt put it in the right spot)

[kolinger]

This app talks to meters via serial ports and/or bluetooth and that's a issue with docker since there is no easy or universal way to access and share such features of the host. I can easily make this app run inside of docker since it's python but that's useless since there is no practical way to connect external devices like meters directly.

I have no idea how bluetooth and/or serial ports work with HA since I never used HA. Thus I don't know if HA bluetooth proxy could be used.

ESP relay could work in way where ESP would sample data from meters and then send them just for displaying but major part of this project is the acquisition of data thus ESP relay would replacement major part of this project thus it would be another project of it's own to develop and maintain.

These options are interesting but I'm afraid this project doesn't land well for such integrations since it's tied directly to meters and that's not easy to change.

Could this project be containerized? Everything is possible but here it likely doesn't make sense to do since it would required to use docker in such way where it would render docker pointless, like running container with --privileged.

Is the amount of work justifiable? Likely not since I don't see easy way and it doesn't make sense to spend major amount of time on something that minority of users would use.

If you want remote rd-usb then I would suggest dedicated raspberry pi - it's likely most cost-effective solution. If you want sample multiple meters you can run multiple rd-usb applications in parallel.

I don't recommend you to spend time on dockerfile unless you have plan how to solve communication with meters - without it it would "run" but it wouldn't connect to anything and that's waste of time.

[wldevries]

You could have a look at https://github.com/thijsputman/tc66c-mqtt

[kolinger]

I see but unfortunately if I read more closely then I see Bluetooth in Docker isn't easy indeed and thus anything what works is with caveats. There is no way to make Docker container with Bluetooth and just start it, you need to know how you want to use it and provide configuration dependent on your OS to make it work. May not be big deal if your targeting only Rasbian but if you want to support any Docker host then it would get more messy for sure.

This example requires to modify AppArmor Bluetooth profile of your OS to make it less strict and if you have capacity to do that, then you don't need Docker in first place IMHO... I see Docker as something that can anyone just turn on anywhere and it works and this just isn't it. If you have root privileges and can change anything including AppArmor profiles then why you want Python in Docker anyway if you can run Python directly? This is what I was talking about previously - you can get it to work only by making Docker redundant.

The original question was if this was easier to run as Docker container and as I see the answer is the opposite - it just makes things more complex for the user and thus I don't see the point of such work only to make it harder to install.

[wldevries]

I have no affiliation with that project at all, but the way I read it is that docker is an alternative. You can just build the app and run it directly.

[kolinger]

I read Docker specific instructions carefully and they say you need to modify your OS AppArmor otherwise Bluetooth won't work in Docker. I didn't try it but they say so.

[kolinger]

I tried it on Debian server and Ubuntu desktop and it's mess. Debian does work with Docker just fine but Ubuntu doesn't - it fails due to the AppArmor policy as the project you posted describes. Custom AppArmor policy does fix it but it's impossible to do it with Docker alone.

Looks like different Linux systems have different configurations so you need to do extra steps outside Docker for your specific system in order to make it work and that's not great for users.

Where you want to use it exactly? I would need to test / verify / find workaround for the most popular systems.

[RayDog1995]

So my interest came when i got my Victron solar controller talking to my HA instance through the ESP bluetooth proxy. I only ever use this tool at home so I could see having multi day tests become easier if the data is dumped into a db. Another idea which would alsso be an unrelated bit of kit is if the app on the phone could send to the db (or this software could do that too if thats a possible workaround). If I really have to i suppose i can just dump the log files your program generates to a db with a script of some sort. Ive also got an order of current sensors coming in so maybe that will work for some of my projects. Where theres a Will theres a Ray

[kolinger]

Docker container now exists so you can try it if it helps you somewhat. It requires special setup though to make it work. Docker doesn't support Bluetooth officially so it's kind of hack but that's the best we can get currently.

I have no idea how or if it can coexists with HA. In this field someone else would need to make it work. I can assist but I have no HA setup or experience so I can't test it or develop it myself.

[RayDog1995]

Ok got it working, was having some weird glitch with one vm I was using and as soon as I switched to a different vm (same os and version) I got readings, was also a different adapter so that might have been it. Now ill make a posst in the HA community seeing if anyone knows integrations well enough to give some pointers. Im very happy with this incredible program you have created and supported.

[Bluscream]

https://community.home-assistant.io/t/bluetooth-usb-energy-monitor/460006

and can you make a armv7, armhf and aarch64 images?

exec /opt/rd-usb/docker/entrypoint.sh: exec format error

I read Docker specific instructions carefully and they say you need to modify your OS AppArmor otherwise Bluetooth won't work in Docker. I didn't try it but they say so.

[blu@minopia ~]$ sudo wget https://raw.githubusercontent.com/kolinger/rd-usb/master/docker/docker-ble -O /etc/apparmor.d/docker-ble
--2024-02-01 14:15:13--  https://raw.githubusercontent.com/kolinger/rd-usb/master/docker/docker-ble
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 2606:50c0:8002::154, 2606:50c0:8003::154, 2606:50c0:8000::154, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|2606:50c0:8002::154|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1774 (1.7K) [text/plain]
Saving to: ‘/etc/apparmor.d/docker-ble’

/etc/apparmor.d/docker- 100%[=============================>]   1.73K  --.-KB/s    in 0s

2024-02-01 14:15:13 (8.94 MB/s) - ‘/etc/apparmor.d/docker-ble’ saved [1774/1774]

[blu@minopia ~]$ sudo apparmor_parser -r -W /etc/apparmor.d/docker-ble
Cache read/write disabled: interface file missing. (Kernel needs AppArmor 2.4 compatibility patch.)
Warning: unable to find a suitable fs in /proc/mounts, is it mounted?
Use --subdomainfs to override.

[kolinger]

I'm not sure what is the difference between armv7 and armhf, I think they are the same thing in this context? I see docker has available linux/arm/v7 and linux/arm64/v8 (this should be aarch64), so I did generated those, it did take it's sweet time, like really long, but it should be available now: https://hub.docker.com/r/kolinger/rd-usb/tags

I'm not sure what is your system but it seems like you don't have AppArmor or something is very different? Try without AppArmor and if you get permissions errors in log then specify what is your system (platform, operating system, ...).

[Bluscream]

Works fine now except bluetooth, thanks anyway ^^ (Had to delete the cached amd64 docker image)

[kolinger]

Do you have some errors around Bluetooth in log? Maybe related to the AppArmor? This is the problem with Docker Bluetooth it can be different on every system...

[Bluscream]

I mean i wasn't able to "install" the apparmor thingy so idk. But surprisingly after starting the container as privileged and root user it was actually able to find the BT device but sadly it wasn't able to connect to it:

2024-02-02 06:05:00 - Bluetooth address is missing. Select address in Setup
2024-02-02 06:06:49 - Bluetooth address is missing. Select address in Setup
2024-02-02 06:18:59 - Bluetooth address is missing. Select address in Setup
2024-02-02 06:22:16 - Connecting
Traceback (most recent call last):
  File "/opt/rd-usb/webapp/backend.py", line 252, in run
    self.retry(self.interface.connect)
  File "/opt/rd-usb/webapp/backend.py", line 357, in retry
    return callback()
  File "/opt/rd-usb/interfaces/wrapper.py", line 34, in connect
    self.get_result(60)
  File "/opt/rd-usb/interfaces/wrapper.py", line 65, in get_result
    raise FatalErrorException(result)
interfaces.interface.FatalErrorException: Traceback (most recent call last):
  File "/opt/rd-usb/interfaces/wrapper.py", line 112, in call
    result = callback()
  File "/opt/rd-usb/interfaces/um.py", line 108, in connect
    raise FatalErrorException("Bluetooth service not found, try to initiate Setup again")
interfaces.interface.FatalErrorException: Bluetooth service not found, try to initiate Setup again

2024-02-02 06:22:20 - Disconnected

Here's my compose:

version: '3.7'
services:
  rd-usb:
    privileged: true
    user: root
    network_mode: host
    container_name: rd-usb
    image: kolinger/rd-usb:latest
    restart: unless-stopped
    security_opt:
      - apparmor=docker-ble
    environment:
      - TZ=Europe/Berlin # change your timeozne (TZ identifier here), for example Europe/London
      - ARGS=
    volumes:
      - /var/run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket
      - /portainer/Files/AppData/Config/rd-usb/data:/opt/rd-usb/data
    # you need something like these examples if you want to use serial port based communication
    # devices:
      # - /dev/ttyUSB0
      # - /dev/rfcomm0

[blu@x~]$ /usr/share/pyshared/lsb_release.py
{'DESCRIPTION': 'Debian GNU/Linux 11 (bullseye)', 'RELEASE': '11', 'CODENAME': 'bullseye', 'ID': 'Debian'}
[]
[blu@x~]$ uname -a
Linux x6.1.21-v8+ #1642 SMP PREEMPT Mon Apr  3 17:24:16 BST 2023 aarch64 GNU/Linux

[kolinger]

If AppArmor doesn't work then comment out security_opt: apparmor lines and it could work if it doesn't then you system is using something else or your AppArmor is just too different. I'm not sure there are too many variables to guess.

Using privileged: true is definitely not a solution or good idea - this gives the container access to everything. This may fix many things but also disables the security of docker completely.

The error you seeing looks like Bluetooth isn't working. It may scan but actual communication is dead. Scanning process is different to the actual communication so it's possible that one parts works and the other doesn't.

[Bluscream]

Yeah, sadly even with the apparmor stuff removed from the compose file:

2024-02-02 06:45:07,861 - INFO - Application is starting...
 * Serving Flask app 'web'
 * Debug mode: off
2024-02-02 06:45:07,939 - INFO - WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on all addresses (0.0.0.0)
 * Running on http://127.0.0.1:5000
 * Running on http://192.168.2.38:5000
2024-02-02 06:45:07,940 - INFO - Press CTRL+C to quit
2024-02-02 06:45:08,542 - INFO - 127.0.0.1 - - [02/Feb/2024 06:45:08] "GET / HTTP/1.1" 200 -
2024-02-02 06:45:08,544 - INFO - Application is available at http://127.0.0.1:5000
2024-02-02 06:45:13,091 - ERROR - <class 'interfaces.interface.FatalErrorException'>
Traceback (most recent call last):
  File "/opt/rd-usb/webapp/backend.py", line 252, in run
    self.retry(self.interface.connect)
  File "/opt/rd-usb/webapp/backend.py", line 357, in retry
    return callback()
  File "/opt/rd-usb/interfaces/wrapper.py", line 34, in connect
    self.get_result(60)
  File "/opt/rd-usb/interfaces/wrapper.py", line 65, in get_result
    raise FatalErrorException(result)
interfaces.interface.FatalErrorException: Traceback (most recent call last):
  File "/opt/rd-usb/interfaces/wrapper.py", line 112, in call
    result = callback()
  File "/opt/rd-usb/interfaces/um.py", line 108, in connect
    raise FatalErrorException("Bluetooth service not found, try to initiate Setup again")
interfaces.interface.FatalErrorException: Bluetooth service not found, try to initiate Setup again

[kolinger]

Do you see some related logs in /var/log/syslog? Do you know if rd-usb works without Docker?

There is also workaround via rfcomm method, where on host you call rfcomm bind with Bluetooth address and this creates serial port, see details in Readme. Then you could pass this serial port into docker and this way you would bypass any Bluetooth docker issues but it's more complicated.

[Bluscream]

Do you see some related logs in /var/log/syslog? Do you know if rd-usb works without Docker?

nothing related to bluetooth or rd-usb

it is connected and paired:

rfcomm and choosing serial in the web ui leads to this:

Traceback (most recent call last):
  File "/opt/rd-usb/webapp/backend.py", line 252, in run
    self.retry(self.interface.connect)
  File "/opt/rd-usb/webapp/backend.py", line 353, in retry
    self.interface.connect()
  File "/opt/rd-usb/interfaces/wrapper.py", line 34, in connect
    self.get_result(60)
  File "/opt/rd-usb/interfaces/wrapper.py", line 67, in get_result
    raise ErrorException(result)
interfaces.wrapper.ErrorException: Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/serial/serialposix.py", line 322, in open
    self.fd = os.open(self.portstr, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK)
FileNotFoundError: [Errno 2] No such file or directory: ''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/rd-usb/interfaces/wrapper.py", line 112, in call
    result = callback()
  File "/opt/rd-usb/interfaces/um.py", line 36, in connect
    self.serial = serial.Serial(port=self.port, baudrate=9600, timeout=self.timeout, write_timeout=0)
  File "/usr/local/lib/python3.7/site-packages/serial/serialutil.py", line 244, in __init__
    self.open()
  File "/usr/local/lib/python3.7/site-packages/serial/serialposix.py", line 325, in open
    raise SerialException(msg.errno, "could not open port {}: {}".format(self._port, msg))
serial.serialutil.SerialException: [Errno 2] could not open port : [Errno 2] No such file or directory: ''

2024-02-02 07:29:33 - Disconnected

[blu@x~]$ sudo rfcomm bind 0 00:16:A6:00:1C:E3

[kolinger]

If you want to use UM34C then you have two choices: 1) rfcomm bind (serial port) 2) direct bluetooth

If you use rfcomm bind, then you need to pass /dev/rfcomm0 or equivalent serial device to docker container and use UM34C Serial and specify /dev/rfcomm0 in GUI. If you want to use direct bluetooth UM34C (without serial) then you shouldn't use rfcomm bind and it shouldn't be paired, UM34C shall be disconnected and rd-usb itself makes direct connection.

[Bluscream]

Does rd-usb require a specific bluetooth service? From the commandline i use bluetoothctl version 5.55

I'm actually trying a external usb bt dongle now instead of the raspberry pi's built in one but idk if the default one i set in bluetoothctl is also the one rd-usb would use

[kolinger]

If you use option 1) rfcomm bind then rd-usb doesn't touch bluetooth and only cares about serial ports, bluetooth side is handled by your system. If you use option 2) direct bluetooth then rd-usb shall do everything on it's own, so you don't use bluetoothctl nor rfcomm at all.

[Bluscream]

If you use option 1) rfcomm bind then rd-usb doesn't touch bluetooth and only cares about serial ports, bluetooth side is handled by your system. If you use option 2) direct bluetooth then rd-usb shall do everything on it's own, so you don't use bluetoothctl nor rfcomm at all.

what i mean is which service it uses under the hood, from what i've read there are different systems for bluetooth just like there are for startup stuff systemd vs init. and apps/library use one or the other

[Bluscream]

Interestingly enough, when i try to manually connect i get this error:

[bluetooth]# connect 00:16:A6:00:1C:E3
Attempting to connect to 00:16:A6:00:1C:E3
[CHG] Device 00:16:A6:00:1C:E3 RSSI: -84
[CHG] Device 00:16:A6:00:1C:E3 Connected: yes
[CHG] Device 00:16:A6:00:1C:E3 ServicesResolved: yes
Failed to connect: org.bluez.Error.NotAvailable

This might be the same thing that rd-usb is failing at but isn't communicating that message in the logs?

[kolinger]

You have systemd or init, you don't have both - but rd-usb doesn't care about either - it doesn't matter how to manage your bluetooth service. Since rd-usb talks to bluetooth stack via dbus connection - this is what /var/run/dbus/system_bus_socket is for. Versions also doesn't matter since UM34C is using very universal part of bluetooth.

Yes error reporting sucks since is no error as such - underlying library returns empty result and I have no idea why, it could mean nearly anything.

BTW: this is what I see when I try to use bluetoothctl on working system:

[bluetooth]# info 00:BA:55:57:C1:85
Device 00:BA:55:57:C1:85 not available
[bluetooth]# connect 00:BA:55:57:C1:85
Device 00:BA:55:57:C1:85 not available

Yet 00:BA:55:57:C1:85 works perfectly in docker. But if I leave bluetoothctl open and connect in docker then bluetoothctl shows UM34C connected and breaks rd-usb in docker. So don't take bluetoothctl too seriously and absolutely don't try to use bluetoothctl or rfcomm in way that it talks with UM34C since this will block communication for any other application including docker. Rule of thumb - if bluetoothctl says anything positive then docker doesn't work, if bluetoothctl says not available then it works perfectly.

[kolinger]

I think the original issue was resolved, thus closing.

[Bluscream]

I think the original issue was resolved, thus closing.

Resolved? I can neither find rd-usb in the official integrations

or in the official addons

or in the third party store

nor do i see a resolution to the feature request for rd-usb: https://community.home-assistant.io/t/bluetooth-usb-energy-monitor/460006/2

[kolinger]

The OP was asking for HA or atlest Docker and the docker was resolved I believe. I don't actually even have idea what would HA integration look like or what functionality it would have either but that doesn't matter since I don't have any capacity to do any HA integration so that's something you or someone else with the knowledge about HA can resolve. I can help if something you need is missing here but I can't help with the HA side.

[Bluscream]

mhmm, thats a bummer :( where's the docker image so i can try myself (needs to be arm32/64)

[kolinger]

The readme has information about it in the Docker section. You did already try it few posts above I think.