search

jens-maus / RaspberryMatic

:house: A feature-rich but lightweight, buildroot-based Linux operating system alternative for your CloudFree CCU3/ELV-Charly 'homematicIP CCU' IoT smarthome central. Running as a pure virtual appliance (ProxmoxVE, Home Assistant, LXC, Docker/OCI, Kubernetes/K8s, etc.) on a dedicated embedded device (RaspberryPi, etc.) or generic x86/ARM hardware.
https://raspberrymatic.de
Apache License 2.0
1.55k stars 192 forks source link

Home Assistant Addon Implementation #1087

Closed angelnu closed 3 years ago

angelnu commented 3 years ago

Describe the solution you'd like Being able to run RaspberryMatic on Home Assistant OS (https://github.com/home-assistant/operating-system) as an installable Addon

cf. https://github.com/home-assistant/addons/issues/1751

Add-on status

Installation/Test HOWTO

!! The following HOWTO is highly experimental. Use at your own risk !!

Please find here a short howto for installing the latest developer versions of Home-Assistant OS for testing the RaspberryMatic HomeAssistant Add-on integration and functionality related to a RPI-RF-MOD or HM-MOD-RPI-PCB connected either via GPIO on a RaspberryPi/Tinkerboard or via a HB-RF-USB-2/HB-RF-ETH.

Installation HOWTO:

  1. Download the latest haos_XXXX-6.0.. file for your hardware platform from https://github.com/home-assistant/operating-system/releases
  2. Perform a fresh install on microSD card (via Etcher) or import fresh OVA in your virtual environment
  3. If you use a RaspberryPi, Tinkerboard or ODROID and want to use a RPI-RF-MOD/HM-MOD-RPI-PCB directly connected to the GPIO bus, make sure to...
    • either mount the flashed sd card locally to be able to access the config.txt file located on the boot partition.
    • or use a SSH Login later on to log in via SSH remotely
    • or connect a HDMI Monitor+keyboard and login in the console
    • On RaspberryPi: edit the /mnt/boot/config.txt file and uncomment the following lines at the bottom: 
      # Uncomment this to enable GPIO support for RPI-RF-MOD/HM-MOD-RPI-PCB
enable_uart=1
dtparam=i2c_arm=on
dtoverlay=miniuart-bt
dtoverlay=rpi-rf-mod
    • On Tinkerboard or ORDOID: edit the /mnt/boot/haos-config.txt file and uncomment the following lines at the bottom: 
      # Uncomment this to enable GPIO support for RPI-RF-MOD/HM-MOD-RPI-PCB
overlays=rpi-rf-mod
  4. Boot-up HAos and wait until basic installation is finished and the system is accessible via http://homeassistant.local:8123/
  5. Switch to Supervisor -> Add-on store and select Repositories by clicking on the three vertical dots on the top right corner.
  6. Add the URL https://github.com/jens-maus/ha-addon-raspmatic as a new experimental add-on store
  7. Scroll down and make sure to install the normal (non-strict) version of RaspberryMatic HA add-on
  8. Enable the sidebar
  9. Make sure you disable protection mode to allow the add-on to access all necessary hardware resources
  10. If you want to use a HB-RF-USB or HB-RF-USB-2 to connect your HomeMatic RF module via USB connect it now.
  11. Start the add-on and monitor the bootup messages where during the Identifying Homematic RF-Hardware... step it should identify the RPI-RF-MOD/HM-MOD-RPI-PCB rf module accordingly.
  12. If you want to use a HB-RF-ETH to connect your Homematic RF-Hardware via Ethernet make sure to follow the documentation here: https://github.com/jens-maus/RaspberryMatic/wiki/Experten-Features#hb-rf-eth-anbindung
  13. Enjoy full HomeMatic/homematicIP dualstack support.
  14. Give some feedback!

Tasks

Latest Screenshot

109651634-12a75d80-7b5f-11eb-9711-cbf448ae7ef7
jens-maus commented 3 years ago

Thank you @angelnu for getting this started! The WIP repository already looks quite promising. One question regarding the future location of that addon comes up immediatly, thought:

Do you think it might be possible to move all this into the main RaspberryMatic repository? If possible, I would like to have all this in the main RaspberryMatic GitHub repository so that it can directly act as a home assistant addon repository. AFAIK all you need is a repository.json file in the top-level directory and then we put, e.g. a "home-assisant-addon" directory where we put all the addon configuration/setup. In my understand this should then be enough to be able to use the RaspberryMatic respository as a home assistant addon repository, right?

And now that you have started to work on this and seem to have got the RaspberryMatic docker booted : What do you mean by "does not work with HA nginx!"?

BTW: I will try to work on the home assistant OS kernel driver integration ASAP and keep you updated in here.

angelnu commented 3 years ago

Do you think it might be possible to move all this into the main RaspberryMatic repository?

@jens-maus - yes, I am intending to do so - the thing is that we need to have this already there during the WIP since I need to put the URLs in different config files and what to test that the remote access works (and not just when you manually check out the add-on to local). Would you consider giving me write access while this is WIP?

In my understand this should then be enough to be able to use the RaspberryMatic respository as a home assistant addon repository, right?

I need to check if HA would get confused when other folders that are not addons are in the github repository. If it works then yes, it would be good to keep it in a single place.

What do you mean by "does not work with HA nginx!"?

The Rega UI does not feel to play well with reverse proxy. Home Assistant integration in the bar required being able to rewrite so the Rega / becomes /raspberrymatic/ . Another area that breaks even without rewrite (tested in my K8S where I have virtual hosts) is when you try to upload a backup file - you do not get back the pop-up to confirm it is about to reset that comes up when I access Rega without reverse proxy.

jens-maus commented 3 years ago

Do you think it might be possible to move all this into the main RaspberryMatic repository?

@jens-maus - yes, I am intending to do so - the thing is that we need to have this already there during the WIP since I need to put the URLs in different config files and what to test that the remote access works (and not just when you manually check out the add-on to local). Would you consider giving me write access while this is WIP?

While I fully trust you (I do!) I really want to keep direct write access to this repository limited as much as possible - especially I am already in the phase of preparing the next release. So can't you simply prepare the necessary major changes (repository.json, etc.) via PRs or maintain these changes in a fork for the time being? Or do you think this would introduce too much hassle or burden on your work? If so, I could indeed reconsider giving you direct write access until the basic addon layout is done.

In my understand this should then be enough to be able to use the RaspberryMatic respository as a home assistant addon repository, right?

I need to check if HA would get confused when other folders that are not addons are in the github repository. If it works then yes, it would be good to keep it in a single place.

Then this should be the first thing todo IMHO. Because if there are really some things that hinder the use of this repository, I could then generate a new "RaspberryMatic-HA-Addon" repository to which I could of course then also grant you direct write access.

What do you mean by "does not work with HA nginx!"?

The Rega UI does not feel to play well with reverse proxy. Home Assistant integration in the bar required being able to rewrite so the Rega / becomes /raspberrymatic/ . Another area that breaks even without rewrite (tested in my K8S where I have virtual hosts) is when you try to upload a backup file - you do not get back the pop-up to confirm it is about to reset that comes up when I access Rega without reverse proxy.

This sounds like some additional lighttpd config tuning within RaspberryMatic could help to actually solve these reverse proxy problems. In fact, I know of many people perfectly using a nginx based reverse proxy to access their CCU/RaspberryMatic from outside. So it should work in principle IMHO.

angelnu commented 3 years ago

While I fully trust you (I do!) I really want to keep direct write access to this repository limited as much as possible - especially I am already in the phase of preparing the next release. So can't you simply prepare the necessary major changes (repository.json, etc.) via PRs or maintain these changes in a fork for the time being? Or do you think this would introduce too much hassle or burden on your work? If so, I could indeed reconsider giving you direct write access until the basic addon layout is done.

I understand it (I leader lager teams in my day job so know how important the maintainer accountability is). Let me find if it gets a blocker - since you are very responsible incorporating PRs it might work out without me having write access to the main repo. Just be prepared for multiple PRs ;-)

Then this should be the first thing todo IMHO. Because if there are really some things that hinder the use of this repository, I could then generate a new "RaspberryMatic-HA-Addon" repository to which I could of course then also grant you direct write access.

Yes, I will try it out first - the only think is that I need to figure out if the 2 docker dev environments (one for the exportroot and another for the HA add-on). Most likely not an issue but it is a first time for me trying.

This sounds like some additional lighttpd config tuning within RaspberryMatic could help to actually solve these reverse proxy problems. In fact, I know of many people perfectly using a nginx based reverse proxy to access their CCU/RaspberryMatic from outside. So it should work in principle IMHO.

Any examples to compare?

I got it to mostly work by:

but as I said the restore CCU backup does not work: you click on the restore (step 3 in the UI) and it does nothing - I debugged in browser and I see the push completing with 200 BUT I do not see anything in the response. When I do it directly I see the response having some data (I do not remember out of my memory cache...).

If you try the HA add-on (it works without HW) then you can see you do not even come to the REGA login page. When you click on the Raspberrymatic add-on in the side bar it sends to http://localhost:7123/local_raspberrymatic (which should be mapped to the REGA /) and then it goes to http://localhost:7123/pages/index.htm?sid=@fGM3q1NqQt@ instead of http://localhost:7123/local_raspberrymatic/pages/index.htm?sid=@fGM3q1NqQt@

jens-maus commented 3 years ago

I understand it (I leader lager teams in my day job so know how important the maintainer accountability is). Let me find if it gets a blocker - since you are very responsible incorporating PRs it might work out without me having write access to the main repo. Just be prepared for multiple PRs ;-)

No problem. I am prepared to receive your PRs and acknowledge them ASAP.

Then this should be the first thing todo IMHO. Because if there are really some things that hinder the use of this repository, I could then generate a new "RaspberryMatic-HA-Addon" repository to which I could of course then also grant you direct write access.

Yes, I will try it out first - the only think is that I need to figure out if the 2 docker dev environments (one for the exportroot and another for the HA add-on). Most likely not an issue but it is a first time for me trying.

For me as well. I do have, thought, less experience with HA and HAos, so we both have to learn a few things here. But I am sure we will manage together to get this going. Perhaps you can write some basic documentation on the docker dev environments and how to build this as I have never uses VSCode here.

This sounds like some additional lighttpd config tuning within RaspberryMatic could help to actually solve these reverse proxy problems. In fact, I know of many people perfectly using a nginx based reverse proxy to access their CCU/RaspberryMatic from outside. So it should work in principle IMHO.

Any examples to compare?

Not really, I am afraid. All I know is, that users are using reverse proxies themselves to access their CCU/RaspberryMatic from the outside and the CloudMatic service is providing the very same to their users. So there must be a way to get this going.

If you try the HA add-on (it works without HW) then you can see you do not even come to the REGA login page. When you click on the Raspberrymatic add-on in the side bar it sends to http://localhost:7123/local_raspberrymatic (which should be mapped to the REGA /) and then it goes to http://localhost:7123/pages/index.htm?sid=@fGM3q1NqQt@ instead of http://localhost:7123/local_raspberrymatic/pages/index.htm?sid=@fGM3q1NqQt@

Can you perhaps provide me a quick howto to get your current state of affairs regarding the HA RaspberryMatic Addon running here on my environment? As said, I do have very less experience with HA and especially regarding development for HA. So I would have to learn first a good bunch of things, I guess. And if you could point me at resources and provide a quick howto and show how I can get your current Addon going, that would be highly appreciated!

EDIT: Ok, I managed to get the Add-on installed at least and the RapberryMatic docker seems to start correctly. Still some things to do, but this is a good starter 👍

angelnu commented 3 years ago

I am glad you got it working - did you see the link to the dev instructions in the Add-on repo? It used Visual Code and a container. I will move this into the dev readme as we merge the repros.

Are you able to repro the problem? You need to enable the nginx access in the add-on settings once you install it -> you then see the icon on the left to access the REGA UI where you will see the problem.

jens-maus commented 3 years ago

Are you able to repro the problem? You need to enable the nginx access in the add-on settings once you install it -> you then see the icon on the left to access the REGA UI where you will see the problem.

Yes, I can see the problem and I partly debugged it already. There is no "nginx", at leaset I can't find any nginx running on the HAos machine. The process listening on 8123 is a python process. Then I tried the "HomeMatic CCU" Addon the HA developers provide themselves and there the OCCU WebUI is perfectly shown. So this does not seem to be a general issue. Perhaps this is a lighttpd conf issue within the RaspberryMatic docker.

angelnu commented 3 years ago

I checked that it is possible to combine the Add-On repo with the Raspberrymatic main repo -> submitted PR.

I also added the dev documentation for building both the Add On and RaspberryMatic inside a container.

Now, for persistence, I did not find a way to map /data which is the folder passed by HA for add on persistency to /usr/local so I will have to add an env variable to tell Raspberrymatic that is running in HA and use that info to bind or symlink /data to /usr/local

EDIT:

jens-maus commented 3 years ago

Hi @angelnu. I am still fighting with the Ingress setup of the home-assistant addon. On all the other HA Add-ons I have found the HA devs use a nginx configuration for creating a reverse proxy to get the ingress support going.

However, as we don't have nginx within the RaspberryMatic docker (and I don't have plans to add it!), I first tried to setup a similar reverse proxy setup with the already existing lighttpd server RaspberryMatic comes with. As I had to learn, the lighttpd server does not allow to modify http responses like nginx does using the sub_filter statements in the nginx config of the homematic occu add-on (see here: https://github.com/home-assistant/addons/blob/master/homematic/rootfs/etc/nginx/nginx.conf). So after have learned that lession, I am currently working on getting a similar reverse proxy implemented using the http-proxy-middleware nodejs class since nodejs v12 already exists in RaspberryMatic.

Here is the current state of the reverse proxy server setup for the ingress proxying:

const express = require('express');
const { createProxyMiddleware } = require('http-proxy-middleware');

const apiProxy = createProxyMiddleware('/', {
  target: 'http://127.0.0.1:80',
  changeOrigin: true, // for vhosted sites,
  logLevel: 'debug',
  selfHandleResponse: true,
  onProxyRes (proxyRes, req, res) {
    console.log(req.headers['x-ingress-path']);
    console.log(proxyRes.statusCode);
    console.log(proxyRes.headers);
    console.log(req.headers);

    if(typeof(proxyRes.headers.location) !== 'undefined') {
      var redirect = proxyRes.headers.location;
      console.log('Received code ' + proxyRes.statusCode + ' from API Server for URL - ' + redirect);
      redirect = req.headers['x-ingress-path'] + redirect;
      console.log('Manipulating header location and redirecting to - ' + redirect);
      proxyRes.headers.location = redirect;
    }

    const bodyChunks = [];
    proxyRes.on('data', (chunk) => {
        bodyChunks.push(chunk);
    });
    proxyRes.on('end', () => {
        const body = Buffer.concat(bodyChunks);

        // forwarding source status
        res.status(proxyRes.statusCode);

        // forwarding source headers
        Object.keys(proxyRes.headers).forEach((key) => {
            res.append(key, proxyRes.headers[key]);
        });

        // modifying html content
        //if (proxyRes.headers['content-type'] && proxyRes.headers['content-type'].includes('text/html'
            let html = body.toString();

            // do whatever you want
            html = html.replace(/\/api\//g, req.headers['x-ingress-path']+'/api/');
            html = html.replace(/\/webui\//g, req.headers['x-ingress-path']+'/webui/');
            html = html.replace(/\/ise\//g, req.headers['x-ingress-path']+'/ise/');
            html = html.replace(/\/pda\//g, req.headers['x-ingress-path']+'/pda/');
            html = html.replace(/\/config\//g, req.headers['x-ingress-path']+'/config/');
            html = html.replace(/\/pages\//g, req.headers['x-ingress-path']+'/pages/');
            html = html.replace(/\/jpages\//g, req.headers['x-ingress-path']+'/jpages/');
            html = html.replace(/\/esp\//g, req.headers['x-ingress-path']+'/esp/');

            res.send(new Buffer.from(html));
        //} else {
        //    res.send(body);
        //}

        res.end();
    });
  },
});

const app = express();
app.use(apiProxy);
app.listen(8099);

While this nodejs-based proxy seem to start well and also seem to serve certain static files (*.png) perfectly fine, I am still getting some 502 errors sporadically here. So can you give this nodejs-based proxy a test and see if you can spot some obvious problem compared to the nginx proxying the homematic occu add-on authors are using? I am probably missing something here. And help would be highly appreciated because I think, if we can get this ingress integration running it would be great for the home-assistant-addon. And starting this nodejs-based reverse proxy would be of course quite easy within RaspberryMatic...

angelnu commented 3 years ago

I can repro the problem and I think I know where it comes:

http://localhost:7123/api/hassio_ingress/tNpfBcSADiKR0wRpRDdtfPT07h5ljGGFb2DVEK9C0ss/api/homematic.cgi

fails. This is because the /api part comes twice in the url. So I think the regular exp match is to greedy. I cannot confirm since it is late - can check tomorrow again.

I am trying to find a way to tag containers without rebuilding... can you ckeck the k8s issue? I have some updates/questions

jens-maus commented 3 years ago

I can repro the problem and I think I know where it comes:

http://localhost:7123/api/hassio_ingress/tNpfBcSADiKR0wRpRDdtfPT07h5ljGGFb2DVEK9C0ss/api/homematic.cgi

fails. This is because the /api part comes twice in the url. So I think the regular exp match is to greedy. I cannot confirm since it is late - can check tomorrow again.

I don't think it is just related to the double /api/ part in the URL. Also this URL fails:

http:/xxxxx:8123/api/hassio_ingress/Y1kh_Az9FKqWQqkvmk14b_CHuapceDYgnTxTyTucKWk/webui/style.cgi

To me, this looks like all *.cgi scripts are causing this exception. Really strange.

jens-maus commented 3 years ago

Ok, I think I got the nodejs-based proxy working now. So the HA ingress is now able to access the RaspberryMatic WebUI here. So I think it is time to test this mit actual rf hardware.

angelnu commented 3 years ago

Cool you got it working! I will give it a try on my side tomorrow.

I am curious how it goes when you propose to PR to the HA code - they are very careful on what code they add there...

jens-maus commented 3 years ago

I am curious how it goes when you propose to PR to the HA code - they are very careful on what code they add there...

You mean for the required kernel modules changes in the HAos to support this RaspberryMatic Add-on? I still hope they will accept it because for them it could be an interesting development/enhancement as well.

But if not (or as an alternative strategy?!?) I already thought about permanently forking the HAos project just for HomeMatic purposes. That would, in principle, (and I am open for that idea!) allow to think about moving towards using HAos as a complete new base system for RaspberryMatic (while calling it "OpenMatic") since it has a better UI, integration, etc. etc. Then we could distribute this HAos-based "OpenMatic" generation with the directly integrated and enabled Add-On so that the CCU WebUI is directly integrated and accessible. But this is just some loose idea currently with no ETA in mind, whatsoever...

angelnu commented 3 years ago

I indeed was thinking in a "plan B" -> they support passing the kernel modules (and I think headers) to the add-on. We could use this to include the modules in the add-on.

I was also thinking on having a common container to load the kernel modules in any host - we would use for HA, K8s, etc. Something like https://github.com/multiarch/qemu-user-static

angelnu commented 3 years ago

@jens-maus - this is the PoC for building and loading the kernel modules via container https://github.com/angelnu/pivccu-modules-container

I plan to use it for my Kubernetes system but it might also come handy for the Home Assistant if we need to keep the sources out of the main project.

We could also use it for the deployer.sh so it avoids downloading each time from apt.

jens-maus commented 3 years ago

I am not quite sure if I understand that correctly yet? What would be the exact process of that additional container? And how could that be actually integrated into the HA add-on? So please elaborate a bit more on the procedure you are suggesting so that I understand all that a bit better.

angelnu commented 3 years ago

The process would be to start the new container before RaspberryMatic using the add-on option kernel_modules: true. This way we would get the kernel modules available into the new container. I have to see if there is an option to also make the kernel sources available to the add-on since it is so limited what folders can be provided to the add-on. Worst case, since this container is privileged, it could just mount the host root disk with the kernel sources itself.

But in general my intention with this extra container is decoupling the task of building and adding the modules to the host. This way instead of depending on the Host being a debian distro the dependency is only to have the kernel sources and docker in the host. This would, for example, enable users of non-debian OSes (NAS, fedora, etc) to load the modules.

jens-maus commented 3 years ago

What does this kernel_modules: true option do in HA? Didn't know it exists. So that would mean that before starting the RaspberryMatic Add-on a user would have to manually start that extra kernel module add-on? Did I get this right? And for the "normal" docker container implementation it would be then the same? A user first would have to manually start this docker container for getting the kernel modules build and installed? And then we expose /lib/modules to the RaspberryMatic docker container so that the container could modprobe the kernel modules itself?

angelnu commented 3 years ago

What does this kernel_modules: true option do in HA?

It is document at https://developers.home-assistant.io/docs/add-ons/configuration

So that would mean that before starting the RaspberryMatic Add-on a user would have to manually start that extra kernel module add-on?

exactly

And for the "normal" docker container implementation it would be then the same?

if with "normal" you mean the "deployer.sh" or other non-HA then yes.

And then we expose /lib/modules to the RaspberryMatic docker container so that the container could modprobe the kernel modules itself?

We would not need to expose the modules to the RaspberryMatic container -> once they are added to the host by running the new "install-modules" container then they are automatically loaded once of the devices is detected.

Please notice that the "install-modules" container needs to be exec at least once when the host kernel is updated to build the modules for the new kernel. I configured it that if the kernel modueles are already installed it does nothing.

UPDATE @jens-maus - have you installed the home assistant on real HW or virtual machine (not as container) - if so could you plese check if you have the kernel sources there in /usr/src or at least the includes to build kernel modules under /lib/modules/*/build ?

If they are there I would continue trying to add the modules via container. If they are not then there is no "plan B" and the only way to add the modules would be at build time of the home assistant.

jens-maus commented 3 years ago

What does this kernel_modules: true option do in HA?

It is document at https://developers.home-assistant.io/docs/add-ons/configuration

That's clear, but the question was more what it actually does? Do you know?

So that would mean that before starting the RaspberryMatic Add-on a user would have to manually start that extra kernel module add-on?

exactly

And for the "normal" docker container implementation it would be then the same?

if with "normal" you mean the "deployer.sh" or other non-HA then yes.

Hmm, I really don't know yet if this is the most smooth way for the RaspberryMatic integration. To me this sounds more like a workaround or hack to get things regarding additional kernel modules going in HAos. To me this would also be somewhat unintuitive that a user would have to install two add-on (one for kernel install and one for the RaspberryMatic installation itself). So I am really unsure if we should go that road or not thing about other possibilities for solving the kernel module installation.

And then we expose /lib/modules to the RaspberryMatic docker container so that the container could modprobe the kernel modules itself?

We would not need to expose the modules to the RaspberryMatic container -> once they are added to the host by running the new "install-modules" container then they are automatically loaded once of the devices is detected.

You mean you would modprobe the modules within the container? I thought that is not possible since within the container you can't run modprobe.

Please notice that the "install-modules" container needs to be exec at least once when the host kernel is updated to build the modules for the new kernel. I configured it that if the kernel modueles are already installed it does nothing.

As said, I am still not sure if such a helper container should be the way to go for the final release...

angelnu commented 3 years ago

That's clear, but the question was more what it actually does? Do you know?

It adds -v /lib/modules:/lib/modules:ro so it gives read-only access in the container to the modules available in the host.

You mean you would modprobe the modules within the container? I thought that is not possible since within the container you can't run modprobe.

It does not seem to be needed -> is enough to install the modules in the host (dkms uses depmod) for them to be automatically loaded once an RF HW is detected. I do not need even a udev rule for that.

But if needed, yes, mounting the /lib/modules AND privileged allows to load modules from within the container. The issue is that the modules must exist on host and we cannot bring them with the container.

As said, I am still not sure if such a helper container should be the way to go for the final release...

We could integrate the dkms solution in the main Raspberrymatic OCI if we want to avoid the extra step. In Kubernetes is very easy to combine multiple OCI containers into a POD so I did not check what HA could do because there are multiple options.

For me the key would be to see if we can get the kernel modules added to HA or if we need to insert them. If the HA kernel sources are not available at runtime (see my UPDATE in the previous post) then the dkms approach would not work.

jens-maus commented 3 years ago

That's clear, but the question was more what it actually does? Do you know?

It adds -v /lib/modules:/lib/modules:ro so it gives read-only access in the container to the modules available in the host.

You mean you would modprobe the modules within the container? I thought that is not possible since within the container you can't run modprobe.

It does not seem to be needed -> is enough to install the modules in the host (dkms uses depmod) for them to be automatically loaded once an RF HW is detected. I do not need even a udev rule for that.

That's clear, but it wouldn't hurt to keep these additional modprobe calls either. Because in the very beginning of your OCI/docker work I thought using modprobe is not possible at all and that's why you/we commented out some of these modprobe calls in the oci platform startup script.

But if needed, yes, mounting the /lib/modules AND privileged allows to load modules from within the container. The issue is that the modules must exist on host and we cannot bring them with the container.

Well, but the privileged access mode brings up quite an annoying warning in the HA WebUI which, if possible, I would like to avoid. or at least get the HA add-on into a stage where it gets the highest possible rating for users to feel confidence that it does not impose any unwanted security threat for them.

As said, I am still not sure if such a helper container should be the way to go for the final release...

We could integrate the dkms solution in the main Raspberrymatic OCI if we want to avoid the extra step. In Kubernetes is very easy to combine multiple OCI containers into a POD so I did not check what HA could do because there are multiple options.

For me the key would be to see if we can get the kernel modules added to HA or if we need to insert them. If the HA kernel sources are not available at runtime (see my UPDATE in the previous post) then the dkms approach would not work.

I checked my ova HAos installation and there are neither kernel sources under /usr/src not under /lib/modules/*/build. Thus, building kernel modules within the docker container does not seem to be a valid option AFAICS.

angelnu commented 3 years ago

Replying from the other thread

But is this really necessary at all in practical terms? Why not simply let the helm chart be updated with the nightly build workflow? IMHO it should be enough to have a daily snapshot rather than having each commit end up in helm chart or docker container updates

I think that is good enough - if anyone wants to manually test a particular container snapshot can manually modify the installed Helm. I will do a PR to disable.

Please note, however, that with switching to self-hosted runners I will have to disable running the workflow execution on pull_requests coming from foreign/forked repositories.

ok, so you will disable the PR. So far we are able to keep the workflow working in the flow it might be good enough -> then you can see, for non trivial fixes, if the test worked in the fork repo.

So this docker container then only partly replaces some steps of the deploy.sh script IMHO. Is this really worth the work? I am not sure.

It is not workable for HA and I do not think we should change deploy.sh just now: it has been stable enough from what I can tell from the feedback. The main usage I see for this is for people running without a Debian based distro.

Replying for this thread

That's clear, but it wouldn't hurt to keep these additional modprobe calls either. Because in the very beginning of your OCI/docker work I thought using modprobe is not possible at all and that's why you/we commented out some of these modprobe calls in the oci platform startup script.

This is a grey line... The containers should not modify the host and that includes loading modules. Even running privileged is not desired since it weakness security. In my work we run containers in dedicated VMs to avoid the security risk since it is not always possible to run without additional privileges.

So I think we should try to avoid modprobes and eventually we should replace the priviledged mode with the minimum required capabilities. But this is a tedious process that take time so IMO once we have an stable base to work on and with more testcases in places.

Well, but the privileged access mode brings up quite an annoying warning in the HA WebUI which, if possible, I would like to avoid. or at least get the HA add-on into a stage where it gets the highest possible rating for users to feel confidence that it does not impose any unwanted security threat for them.

exactly - they use this "gamification" technique to foster to do the right thing without forbiding it. A very clever idea :-)

I checked my ova HAos installation and there are neither kernel sources under /usr/src not under /lib/modules/*/build. Thus, building kernel modules within the docker container does not seem to be a valid option AFAICS.

Yes, as I said above it has no longer usage for HA. So I keep it in my repo for now and we can see if anyone needs it before me doing anything more consumable.

feutl commented 3 years ago

Hi, I am quite interested in getting Homematic migrated to my HA setup. The already existing add-on is just not working properly. In my case, it doesn't work at all, while others seem to have gotten it working with some HM/HMIP devices. I do have a HM-MOD-RPI-PCB modul, right now connected to a RPi 3+ for testing purposes. A RPi 4 is my productive environment which I want to use after this all works ;)

My HM setup is still running on my CCU2 connected to HA using the standard integration. Anyhow, how can I help with what I can provide ;)

jens-maus commented 3 years ago

I am quite interested in getting Homematic migrated to my HA setup. The already existing add-on is just not working properly. In my case, it doesn't work at all, while others seem to have gotten it working with some HM/HMIP devices. I do have a HM-MOD-RPI-PCB modul, right now connected to a RPi 3+ for testing purposes. A RPi 4 is my productive environment which I want to use after this all works ;)

Well, first of all: You probably know that the HAos add-on integration of RaspberryMatic has still beta quality (read here: https://github.com/jens-maus/RaspberryMatic/wiki/Installation-HomeAssistant). Second, the HAos add-on still does not support using a HM-MOD-RPI-PCB or RPI-RF-MOD simply because these modules require special linux kernel modules which have to be available/installed in the base operating system (in case of Home Assistant this is HAos). So using /dev/ttyAMA0 or any other standard serial tty device like other HomeMatic+Home Assistant users are currently trying (e.g. https://gist.github.com/FloThinksPi/a2cc6e0e8d106ca9e6378c8c6c61ee67 or https://community.home-assistant.io/t/can-t-get-homematic-addon-working/275528) will simply not work! So here at RaspberryMatic, we are still working on getting these special kernel modules (https://github.com/alexreinert/piVCCU/tree/master/kernel) directly integrated into Home Assistant OS. However, we are not there yet, thus the HA RaspberryMatic Add-on currently only supports using a HmIP-RFUSB, HM-LGW-O-TW-W-EU, HM-CFG-USB-2, HM-CFG-LAN or HMW-LGW-O-DR-GS-EU to communicate with HomeMatic devices.

feutl commented 3 years ago

Thanks for the feedback, I am still wondering why others in the HA community are successfully using the HM-MOD-RPI-PCB with the HomeMatic CCU Addon on a Raspberry Pi. Perhaps they do not report their failure - but you could be right because the links you provided are also the ones I am contributing too ;)

But my test system is recognizing the module via /dev/ttyAMA0 or /dev/ttyS0 (depending on the boot settings), it seems, but yes you seem to be right, I am stuck again at a point using the "standard" addon.

But your Add-On booted with protection mode disabled shows me that "something" was detected

Updating Homematic RF-Hardware: no GPIO/USB connected RF-hardware found

Anyhow, I am awaiting your addon, I like homematic (homematic IP if supported with just one module would be even better ;) ) and the combination of WiFi devices with the modern interface and power of HA - this is just an amazing combination.

THANKS for your effort upfront and I am counting the days :D

Update Just to verify, using a HB-RF-USB-2 with the HM-MOD-RPI-PCB would work already? Because if, I would order one to test ;)

angelnu commented 3 years ago

@feutl - yes, it works because it does not need extra kernel modules. Please notice that only HM-IP is then possible.

@jens-maus - are you active in the Integrate kernel modules in Home Assistant OS -> @jens-maus TBD?

jens-maus commented 3 years ago

@feutl, @angelnu: Nope, the HB-RF-USB-2 also requires the generic_raw_uart kernel module on top of an additional hb-rf-usb-2 kernel module. So this will also not work in the HAos add-on.

@angelnu I currently do not have a TBD for the kernel module integration. I am currently busy in getting the next RaspberryMatic major version out for next weekend. then after that I will try to work on the kernel module patches for HAos and potentially submit a PR to the HA devs for integration.

feutl commented 3 years ago

@angelnu @jens-maus This means I need to wait for a "kernel patch" anyhow. dam, but ok, I can wait 😄 even if it is really hard

angelnu commented 3 years ago

Sorry, I thought we were referring to HmIP-RFUSB - thus to my comment that it only works with HM IP.

@jens-maus - going after the first release is good. In fact most likely we will not depend on a new Raspberry release for it but a new Home Assistant version.

jens-maus commented 3 years ago

@jens-maus - going after the first release is good. In fact most likely we will not depend on a new Raspberry release for it but a new Home Assistant version.

Yeah, my main motivation to get the current state of affairs out right now rather than anytime later is, that I want to get the Home Assistant community informed that there is a new HomeMatic add-on on the market that will (as planned) replace the internal "HomeMatic CCU" add-on not only in terms of supporting all known homematic rf modules, but also provide significantly more functionality than the limited add-on of their embedded add-on :)

feutl commented 3 years ago

I am quite interested in getting Homematic migrated to my HA setup. The already existing add-on is just not working properly. In my case, it doesn't work at all, while others seem to have gotten it working with some HM/HMIP devices. I do have a HM-MOD-RPI-PCB modul, right now connected to a RPi 3+ for testing purposes. A RPi 4 is my productive environment which I want to use after this all works ;)

Well, first of all: You probably know that the HAos add-on integration of RaspberryMatic has still beta quality (read here: https://github.com/jens-maus/RaspberryMatic/wiki/Installation-HomeAssistant). Second, the HAos add-on still does not support using a HM-MOD-RPI-PCB or RPI-RF-MOD simply because these modules require special linux kernel modules which have to be available/installed in the base operating system (in case of Home Assistant this is HAos). So using /dev/ttyAMA0 or any other standard serial tty device like other HomeMatic+Home Assistant users are currently trying (e.g. https://gist.github.com/FloThinksPi/a2cc6e0e8d106ca9e6378c8c6c61ee67 or https://community.home-assistant.io/t/can-t-get-homematic-addon-working/275528) will simply not work! So here at RaspberryMatic, we are still working on getting these special kernel modules (https://github.com/alexreinert/piVCCU/tree/master/kernel) directly integrated into Home Assistant OS. However, we are not there yet, thus the HA RaspberryMatic Add-on currently only supports using a HmIP-RFUSB, HM-LGW-O-TW-W-EU, HM-CFG-USB-2, HM-CFG-LAN or HMW-LGW-O-DR-GS-EU to communicate with HomeMatic devices.

It seems that sven25 has managed to get it running with the standard integration. For my the Webui is stuck at loading. Now I wonder, why does it work for some with the HM-MOD-RPI-PCB module if it should work? And if it could work (as some have already managed) shouldn't it be possible to use them already with the RaspberryMatic Addon?

Sorry, but I try to understand why we are talking about kernel patches (which could take some time to get into HA) if we could already use 2 modules others are already using successfully. I do understand that those patches are necessary for the long run to support even more modules, but how could others have managed it to get it working wonder

Anyhow, I gonna wait, because it is just too time consuming to get it running, and who knows if it stays in a working state for long, but still try to understand how things are done :D

Update 10.02.2021 I was capable of get it working, the original HA OCCU addon. The Ingress support does not work, but using the Webui with an dedicated port makes it available and I was also capable of "teaching" an HM device to the system.

sotatech commented 3 years ago

Thank you for your work on this add-on for Home Assistant! I have long hoped to integrate my CCU with HA so that I only have one server running. Currently, I run RaspberryMatic with a TinkerBoard S and it integrates well with HA. I have installed it on my test HA server, and it loads perfectly and I can see the UI but of course no devices as I do not have a supported adapter. Three questions for you:

  1. I have a HB-RF-USB-TK USB adapter with the RPI-RF-MOD board which you do not mention, but I think it will need the kernel modules to work with the add-on?
  2. Once the kernel modules are working, this configuration will still support HM and HM-IP?
  3. There is a "well-known" issue with the HA integration that if the CCU is rebooted, the HA server cannot restore the connection to the CCU without itself being rebooted. Will this add-on solve this problem?

If you need me to do any testing for you, I am happy to oblge :)

Pat Rooney

jens-maus commented 3 years ago
  1. I have a HB-RF-USB-TK USB adapter with the RPI-RF-MOD board which you do not mention, but I think it will need the kernel modules to work with the add-on?

Once the mentioned kernel modules are potentially integrated into a future HAos release, then the HB-RF-USB and also the HB-RF-ETH will definitely work, yes.

  1. Once the kernel modules are working, this configuration will still support HM and HM-IP?

Yes, once the kernel modules are up&running the HA RaspberryMatic add-on should come with all the functionality of a standard RaspberryMatic runnning on dedicated hardware or as a separate virtual appliance.

  1. There is a "well-known" issue with the HA integration that if the CCU is rebooted, the HA server cannot restore the connection to the CCU without itself being rebooted. Will this add-on solve this problem?

I actually don't know. We will see how this works out as soon as the current state of affairs is released and as soon as I have started to integrate the kernel modules in HAos. And then let's see if or how the HA developers will accept/answer to my upcoming PR or if it will first take some time to convince them to accept it.

feutl commented 3 years ago 
3\. There is a "well-known" issue with the HA integration that if the CCU is rebooted, the HA server cannot restore the connection to the CCU without itself being rebooted. Will this add-on solve this problem?

I want to add something here, perhaps good for later reference ;) but there is no need to restart HA, but just call a service Homematic Integration homematic.reconnect: Reconnect to CCU/Homegear without restarting Home Assistant (useful when CCU has been restarted)

This can be triggered by RaspberryMatic after a reboot or anytime it is needed. I have a HA automation running, which checks the state of some sensors where I know they get updatess quite often (thermostat) and if the state does not change for some time I call the service to reconnect. Theoretically this could also be done with RaspberryMatic, if it exposes certain values to HA, then the automation can be triggered from HA instead of from RaspberryMatic

sotatech commented 3 years ago

@feutl Yes, I have tried using an automation to call the service, but for me and others this does not work. The good thing is that RaspberryMatic is very stable, so it does not need to be rebooted very often :)

sotatech commented 3 years ago

@jens-maus Thank you for your reply, that is all good news! I will watch your progress with interest.

angelnu commented 3 years ago

This can be triggered by RaspberryMatic after a reboot or anytime it is needed. I have a HA automation running, which checks the state of some sensors where I know they get updatess quite often (thermostat) and if the state does not change for some time I call the service to reconnect. Theoretically this could also be done with RaspberryMatic, if it exposes certain values to HA, then the automation can be triggered from HA instead of from RaspberryMatic

@feutl I used the HA automation to reboot the HA homematic adapter as recommended in the HA instructions but if someone has the code to call the reconnect service from Raspberrymatic after a reboot I would like to have it - I do not like with the HA automation that it depends on a particular device sending updates in a regular base and that it takes some time to react to the reboot (to avoid false positives)

sotatech commented 3 years ago

The instructions for calling the service are here: https://www.home-assistant.io/integrations/homematic/#detecting-lost-connections

angelnu commented 3 years ago

@sotatech - thanks, I had overseen the second version of the automation - it still does not have the CCU directly calling the HA REST API but at least it reacts faster to the reboot.

sotatech commented 3 years ago

If you look at this post, it gives some more information about the problem: https://community.home-assistant.io/t/homematic-states-not-updating-if-controlled-outside-of-ha/188783 In particular, according to @danielperna84 the reconnection solution does not work with HM-IP:

For HmIP it doesn’t work at all. The Server-Code on the CCU from eq3 is broken and doesn’t allow reconnections. So restarting is the only choice.

Also:

Yup. It’s a binary that comes straight from eq3. RaspberryMatic doesn’t modify this. The source code isn’t available as well. So Nobody but eq3 can fix this issue. And for them it isn’t really an issue because within the CCU it’s not a problem. And for tools like ioBroker it’s no problem because the HomeMatic component can be restarted without restarting ioBroker. But as long as Home Assistant doesn’t have a modern integration, a restart is required to reset HomeMatic.

jens-maus commented 3 years ago

If you look at this post, it gives some more information about the problem: https://community.home-assistant.io/t/homematic-states-not-updating-if-controlled-outside-of-ha/188783 In particular, according to @danielperna84 the reconnection solution does not work with HM-IP:

Thanks for the comments and links. Actually, that's the first time I read about such an issue and it AFAIR never been reported to neither me nor the eQ3 developers. And as I have close contact to the eQ3 developers it would be quite easy (with an adequate example case) to get this potential issue fixed in the HmIPServer component. Thus, if @danielperna84 or a HA core developer has more information to share or even a test-case which demonstrates the issue outside of Home Assistant it should be quite straight forward to get this fixed by the eQ3 developers.

sotatech commented 3 years ago

That would be great! If you search the HA forum, you will see the issue comes up quite often and causes a lot of confusion. Daniel is the most active support person for HomeMatic issues on the site. Thanks, again.

Pat

jens-maus commented 3 years ago

That would be great! If you search the HA forum, you will see the issue comes up quite often and causes a lot of confusion. Daniel is the most active support person for HomeMatic issues on the site.

Well, once the new player (RaspberryMatic HA Add-on) is out in town (this weekend) I will try to invest some more time in getting all theses things sorted out step by step, hopefully getting the add-on and all its dependencies (required kernel modules, etc.) into a state that the add-on will run as smooth as a real RaspberryMatic installation on dedicated hardware or dedicated virtual appliance.

sotatech commented 3 years ago

Fantastic, I have a spare Home Assistant server I use for testing so I will set it up on that once it's released.

danielperna84 commented 3 years ago

Actually, that's the first time I read about such an issue

I believe the thread he has mentioned is outdated. Since https://github.com/jens-maus/RaspberryMatic/issues/843 has been fixed, I have assumed the problem has been solved. I myself have never verified it, but I thought with that fix the reconnects are working again for HmIP. Correct me if I'm wrong @sotatech .

sotatech commented 3 years ago

I was not aware of that change. I cannot remember when I last experienced the problem as I now always reboot my Home Assistant server if I rebooted the CCU. I will test this today and report back. Should I still use the reconnect automation with the V_Last_Reboot system variable?

feutl commented 3 years ago

Fantastic, I have a spare Home Assistant server I use for testing so I will set it up on that once it's released.

My testing machine is awaiting patches too :) it already feels a little lonely being turned off and resting 😄

sotatech commented 3 years ago

I have restarted the CCU and here is what I find: I disabled the reconnect automation first and then waited until all the HM-IP devices had reconnected to the CCU. After that, changes to devices in the CCU are not shown in Home Assistant. Changes to these devices in Home Assistant are shown in the CCU. I then called the homematic.reconnect service and at first it looked like nothing had happened as the HA devices did not change their values. However, once I made a change on the CCU, the change was shown on Home Assistant. In conclusion, it seems that we still need to use the automation to reconnect to HomeMatic, but we no longer need to restart Home Assistant, so it's good news!