search

fredlcore / BSB-LAN

LAN/WiFi interface for Boiler-System-Bus (BSB) and Local Process Bus (LPB) and Punkt-zu-Punkt Schnittstelle (PPS) with a Siemens® controller used by Elco®, Brötje® and similar heating systems
222 stars 84 forks source link

[FEATURE REQUEST] Forward all value updates to MQTT #442

Closed escoand closed 4 months ago

escoand commented 2 years ago

Is your feature request related to a problem? Please describe. The current MQTT solution is just transmitting some manual selected parameters regularly to the broker.

Describe the solution you'd like I would like to propose to implement a way to constantly publish all BSB/LSP values to the MQTT broker. Furthermore I would like to add MQTT discovery messages (https://www.home-assistant.io/docs/mqtt/discovery/). With this solution you don't need to add any configuration (beside the MQTT broker) in your smart home system, at least Home Assistant. The overall goal for me is to use BSB-LAN just as a bridge from BSB/LSP to MQTT and do all the view, historize and controlling stuff with Home Assistant.

Describe alternatives you've considered I've implemented a working solution based on ESPHome. But this is currently only listening on BSB, so manly only the "Kesseltemperatur" is updated regularly in my case. https://gist.github.com/escoand/180b6b317705944887c6fcdae3007a16

Describe your own contribution I could adapt my implementation to BSB-LAN. The issue is mainly for discussion the general idea and the concrete implementation.

Additional context This was already initially discussed with @fredlcore and @1coderookie via mail.

mirkolenz commented 1 year ago

Sorry for the long response time, I had (and still have) a lot of work that piled up. I second your suggestion to start with auto-discovery messages for values and switches. I will report back as soon as possible with the necessary text templates.

nielssch commented 6 months ago

@mirkolenz I fully support your approach to have BSB-LAN publish sensors with auto discovery so that those can be easily added in homeassistant. Did you progress on this matter?

fredlcore commented 6 months ago

@nielssch: Since there hasn't been any activity for almost a year, and you also didn't get a reply in over two weeks now, I don't see that anyone else but you is still (actively) interested in this matter. As I mentioned before, I assume the easiest way would be to write a conversion script that takes the information from BSB_LAN_custom_defs.h and generates a yaml file for HomeAssistant that can then be easily imported. This has two main advantages: It will not add code to BSB-LAN that will only be used by some part of the user base and even there basically only once when parameters are being set up. The other advantage is that this yaml will contain all parameters that BSB_LAN_custom_defs.h contains, and not just the up to 20 that you define in BSB-LAN.

So in case you don't have any experience in providing a solution in code to be added into BSB-LAN, I could offer you to write such a conversion script, but you would need to provide me with templates such as the ones mentioned in https://github.com/fredlcore/BSB-LAN/issues/442#issuecomment-1362751154 - for every type that HA supports (switches, sensors etc.) and for every value type that HA supports (temperature, humidity etc.). Then I could come up with something, but since I'm otherwise neither using HA nor MQTT, most of these examples above contain too little information for me to act upon them and include them in BSB-LAN.

mirkolenz commented 6 months ago

Hi @nielssch and @fredlcore! First of all, sorry for my late reply, I am currently a bit overwhelmed by my workload and did not see your post.

As for the actual question: I sadly did not make any substantial progress on that matter. My setup currently works fine for my needs and creating said message templates turned out to be more work than initially anticipated. I am still in principle interested in this feature, but sadly do not have the capacity to progress here. If this changes in the future I will report back, but I do not expect this to happen for the foreseeable future. Sorry for these bad news!

fredlcore commented 6 months ago

Thanks, @mirkolenz, that is of course understandable, and while this would be nice to have, most people get it up and running without auto-discovery in no time usually. I'll keep this issue open in case someone else would like to help out here, but it's not urgent. I was trying to have a look at the HomeAssistant documentation to figure out about the structure of the auto-discovery messages, but unfortunately the documentation leaves quite some things to be wanted. While it talks about binary sensors and sensors that transmit several parameters, it doesn't mention anything about multi-option (i.e. "dropdown") parameters. One last question that I would have at this point is whether it would be easier for users if parameters would be advertised via auto-discovery in read-only mode (because then we would not have to think about how to set these parameters), or whether changing these would make it more difficult / less intuitive compared to not having auto-discovery in the first place. Let's take comfort temperature setpoint as an (not the best) example: If auto-discovery would advertise this as a sensor that is read-only, at some point the user would want to change it and modify this sensor to be writeable. Is this easily possible or would it be more intuitive if the user sets up the template from scratch in the first place? Provided of course that the auto-discovery doesn't overwrite any of these changes...

mirkolenz commented 6 months ago

A read-only discovery is an interesting idea, but I share the same doubts like you: Users will at one point change some value and this will not be possible. They would then need to create their BSB-LAN MQTT configuration from scratch. This may lead to lots of bug reports in this repo.

fredlcore commented 6 months ago

Thanks for clarifying this. So it's either a full implementation or none at all. Could you post your BSB-LAN related yaml configuration here? If it is a separate file (i.e. separate from other devices that you may be using in HomeAssistant), I could try and generate this yaml configuration in the same process that I generate the BSB_LAN_custom_defs.h file (whether users are using HomeAssistant or not). Then users could just drop in this file into HomeAssistant and it would work from the outset, right?

mirkolenz commented 6 months ago

Sure, here is my complete configuration:

# Units: https://github.com/home-assistant/core/blob/4e29bdf7159eb3f3f2383d7568cfd032adb3ce4d/homeassistant/const.py#L474

mqtt:
  button:
    # https://community.home-assistant.io/t/bsb-lan-integration/113501/80
    # https://1coderookie.github.io/BSB-LPB-LAN/kap06.html
    - name: Boiler hot water temperature push
      unique_id: a07bd605-3288-4cda-805f-a4a4e3cb4966
      command_topic: "bsblan"
      payload_press: "S1603=1"
      availability_topic: "bsblan/status"
      icon: mdi:water-boiler

    - name: Boiler presence reduced
      unique_id: 12495024-0d95-421d-9148-3a95bbe25676
      command_topic: "bsblan"
      payload_press: "S701=1"
      availability_topic: "bsblan/status"
      icon: mdi:home-floor-0

    - name: Boiler presence comfort
      unique_id: 38283088-4d8e-4486-9607-09dcb61960db
      command_topic: "bsblan"
      payload_press: "S701=2"
      availability_topic: "bsblan/status"
      icon: mdi:home-account

  number:
    - name: Boiler comfort setpoint
      unique_id: 069a4bc0-e476-4392-8eec-6171202e9ac3
      state_topic: "bsblan/710"
      command_topic: "bsblan"
      availability_topic: "bsblan/status"
      min: 19.0
      max: 22.0
      step: 0.5
      mode: slider
      unit_of_measurement: °C
      device_class: temperature
      command_template: "S710={{value}}"
      value_template: "{{value}}"
      icon: mdi:cog

    - name: Boiler away setpoint
      unique_id: 6db1cb24-4e62-4a3d-9115-d92ef1eadfdd
      state_topic: "bsblan/712"
      command_topic: "bsblan"
      availability_topic: "bsblan/status"
      min: 16.0
      max: 19.0
      step: 0.5
      mode: slider
      unit_of_measurement: °C
      device_class: temperature
      command_template: "S712={{value}}"
      value_template: "{{value}}"
      icon: mdi:cog

    - name: Boiler changeover temperature
      unique_id: dac63f1c-37dd-4837-a93b-0234c600dc14
      state_topic: "bsblan/730"
      command_topic: "bsblan"
      availability_topic: "bsblan/status"
      min: 10.0
      max: 20.0
      step: 0.5
      mode: slider
      unit_of_measurement: °C
      device_class: temperature
      command_template: "S730={{value}}"
      value_template: "{{value}}"
      icon: mdi:sun-snowflake-variant

  switch:
    - name: Boiler hot water
      unique_id: 30d66077-1ea6-4c9a-a1a3-6871d1133525
      state_topic: "bsblan/1600"
      state_on: "1 - On"
      state_off: "0 - Off"
      command_topic: "bsblan"
      payload_on: "S1600=1"
      payload_off: "S1600=0"
      availability_topic: "bsblan/status"
      icon: mdi:water-boiler

  select:
    - name: Boiler operating mode
      unique_id: 05a2b97a-bf56-419e-a3a6-ca9a019e0aeb
      state_topic: "bsblan/700"
      command_topic: "bsblan"
      availability_topic: "bsblan/status"
      # Options: {0: 'Protection', 1: 'Automatic', 2: 'Reduced', 3: 'Comfort'}
      # Example value: 1 - Automatic
      value_template: >
        {% set mapping = {0: 'Eco', 1: 'Automatic', 2: 'Away', 3: 'Comfort'} %}
        {% set idx = value.split() | first | int %}
        {{ mapping[idx] }}
      # Example payload: S700=1
      command_template: >
        {% set mapping = {'Eco': 0, 'Automatic': 1, 'Away': 2, 'Comfort': 3} %}
        S700={{ mapping[value] }}
      options:
        - Automatic
        - Comfort
        - Away
        - Eco
      icon: mdi:list-box

  sensor:
    - unique_id: 93ee89e4-3486-478d-a2ff-20fe38d1e0d9
      force_update: true
      availability_topic: "bsblan/status"
      state_topic: "bsblan/8310"
      name: Boiler inlet temperature
      value_template: "{{iif(value == '---', '', value)}}"
      unit_of_measurement: °C
      device_class: temperature
      icon: mdi:thermometer-chevron-up

    - unique_id: 608d2ae9-8fe1-4458-8e90-e66f72599ea5
      force_update: true
      availability_topic: "bsblan/status"
      state_topic: "bsblan/8314"
      name: Boiler return temperature
      value_template: "{{iif(value == '---', '', value)}}"
      unit_of_measurement: °C
      device_class: temperature
      icon: mdi:thermometer-chevron-down

    - unique_id: 4ed49365-2156-4cbd-be49-29e057241c28
      force_update: true
      availability_topic: "bsblan/status"
      state_topic: "bsblan/8744"
      name: Boiler heating circuit setpoint
      value_template: "{{iif(value in ['0', '0.0', '---'], '', value)}}"
      unit_of_measurement: °C
      device_class: temperature
      icon: mdi:home-thermometer

    - unique_id: 9a6d9eb4-f032-429a-9bb6-9dccad3ef261
      force_update: true
      availability_topic: "bsblan/status"
      state_topic: "bsblan/8311"
      name: Boiler inlet setpoint
      value_template: "{{iif(value == '---', '', value)}}"
      unit_of_measurement: °C
      device_class: temperature
      icon: mdi:home-thermometer

    - unique_id: 781cd591-8354-4e8c-beec-baf31bfbfdc0
      force_update: true
      availability_topic: "bsblan/status"
      state_topic: "bsblan/8308"
      name: Boiler pump speed
      value_template: "{{iif(value == '---', '0.0', value)}}"
      unit_of_measurement: "%"
      device_class: power_factor
      icon: mdi:pump

    - unique_id: 22403a60-4e3a-4435-a3fb-c69a2e14ca24
      force_update: true
      availability_topic: "bsblan/status"
      state_topic: "bsblan/8325"
      name: Boiler fan speed
      value_template: "{{iif(value == '---', '0.0', value)}}"
      unit_of_measurement: "%"
      device_class: power_factor
      icon: mdi:fan

    - unique_id: 144ecdae-4b87-4728-bef5-02b003a53adc
      force_update: true
      availability_topic: "bsblan/status"
      state_topic: "bsblan/8326"
      name: Boiler modulation
      value_template: "{{iif(value == '---', '0.0', value)}}"
      unit_of_measurement: "%"
      device_class: power_factor
      icon: mdi:fire

    - unique_id: 0cc22638-1c11-47a4-a319-88df4cf5d1c2
      force_update: true
      availability_topic: "bsblan/status"
      state_topic: "bsblan/8329"
      name: Boiler ionization current
      value_template: "{{iif(value == '---', '0.0', value) | float / 1000}}"
      unit_of_measurement: "mA"
      device_class: current
      icon: mdi:sine-wave

    - unique_id: 7309d048-5339-4f5c-b18e-e8caba786412
      force_update: true
      availability_topic: "bsblan/status"
      state_topic: "bsblan/8327"
      name: Boiler water pressure
      value_template: "{{iif(value == '---', '0.0', value)}}"
      unit_of_measurement: bar
      device_class: pressure
      icon: mdi:gauge

    - unique_id: ec3e856a-8ef8-4bfd-9a77-0eb315639e7c
      force_update: true
      availability_topic: "bsblan/status"
      state_topic: "bsblan/8700"
      name: Boiler outside temperature
      value_template: "{{iif(value == '---', '', value)}}"
      unit_of_measurement: °C
      device_class: temperature
      icon: mdi:sun-thermometer

    - unique_id: 7af0ef9c-3aa8-4ff5-8ce9-29402d77a7a2
      force_update: true
      availability_topic: "bsblan/status"
      state_topic: "bsblan/8830"
      name: Boiler hot water temperature
      value_template: "{{iif(value == '---', '', value)}}"
      unit_of_measurement: °C
      device_class: temperature
      icon: mdi:water-boiler

Correct, users could include such an auto-generated file in their home assistant configuration. There is a special !input statement to include other yaml files in the file system. Also, home assistant allows to use so-called packages which are merged internally, so it would even be possible to create separate files and let them be combined by home assistant.

mirkolenz commented 6 months ago

In addition to the raw sensors, I have the following automation:

id: ac6e5103-a68f-4719-a5d5-6e88aad911f1
description: ""
mode: single
alias: BSB-LAN
variables:
  params:
    - 700
    - 710
    - 712
    - 730
    - 1600
trigger:
  - platform: state
    id: refresh
    entity_id: input_button.boiler_refresh_parameters
  - platform: homeassistant
    id: refresh
    event: start
  - platform: event
    id: refresh
    event_type: event_mqtt_reloaded
  - platform: time
    id: refresh
    at: 03:00:00
  - platform: state
    id: enable_heating
    entity_id: switch.enable_heating
    from: "off"
    to: "on"
  - platform: state
    id: disable_heating
    entity_id: switch.enable_heating
    from: "on"
    to: "off"
condition: []
action:
  - choose:
      - conditions:
          - condition: trigger
            id: refresh
        sequence:
          - repeat:
              count: "{{ params | count }}"
              sequence:
                - service: mqtt.publish
                  data:
                    payload: "{{ params[repeat.index - 1] }}"
                    topic: bsblan
                - delay:
                    hours: 0
                    minutes: 0
                    seconds: 2
                    milliseconds: 0
      - conditions:
          - condition: trigger
            id: enable_heating
        sequence:
          - service: select.select_option
            target:
              entity_id: select.boiler_operating_mode
            data:
              option: Automatic
      - conditions:
          - condition: trigger
            id: disable_heating
        sequence:
          - service: select.select_option
            target:
              entity_id: select.boiler_operating_mode
            data:
              option: Eco

It fetches the status of seldomly used values (e.g., operating mode) on demand. This enables me to include more than 20 BSB-LAN entities in my home assistant configuration.

hvorragend commented 6 months ago

You don't need an automation for this. This can all be solved via the climate device.

mirkolenz commented 6 months ago

Interesting, never thought of this. I am currently not sure how I would represent my switches for hot water and the heating mode selection via a climate entity. If I understand it correctly, I would need to set up multiple of them, is that true?

fredlcore commented 6 months ago

@mirkolenz: Thank you! I hope you can help me with a few questions to understand this fully:

button:

The parameters you listed under this section are all parameters that do not give feedback, i.e. can only be "pushed". Would button also be applicable for general on/off or yes/no buttons? If so, how would I transmit the information which refers to "on"/"yes" etc. and which one to "off"/"no"...?

unique_id: a07bd605-3288-4cda-805f-a4a4e3cb4966

This ID is unique to the BSB-LAN parameter, right? Could I just use the parameter number here? Or does it have to follow a certain pattern?

command_topic: "bsblan"

Is this the same as MQTTTopicPrefix in BSB_LAN_config.h (which defaults to "BSB-LAN")?

icon: mdi:water-boiler

Can this (easily) be changed via the UI later on? If so, then I'd just omit this. Otherwise, what I could do is match a certain unit type (i.e. "temperature" to a certain icon. But that would result in the same icon for comfort temperature setpoint as well as buffer temperature. If that's ok, it would be great if you could point me to or provide me a list for each unit type.

state_topic: "bsblan/710" command_topic: "bsblan"

Again, is this the MQTTTopicPrefix?

mode: slider

Can this be generally used for all parameters in the number section?

device_class: temperature

What kind of device classes do exist in HA and what is this attribute used for? And is it required? Matching dozens of units of measurement to a specific device_class seems to me the most work at first glance...

state_on: "1 - On"

Should the numerical value come before the concrete description (here: On)? And could the switch type be used for other binary parameters as well (sich as Yes/No) and I would still use state_on for "Yes" for example?

force_update: true

Which kind of parameters need this?

value_template: "{{iif(value == '---', '', value)}}" value_template: "{{iif(value in ['0', '0.0', '---'], '', value)}}" value_template: "{{iif(value == '---', '0.0', value) | float / 1000}}"

When would I use any of these value templates? And why do you have to divide the mA value by 1000? And last but not least: Is it really iif with two i?

If you could help me setteling these questions, I don't think it'll be a big thing to write a conversion script that outputs a suitable yaml file for HA use. Do you have an example for !input at hand that I could include? Or would it be easier to create a "package" for this (and if so, do you know how or could point me to the right direction)?

I think then I can get to work and settle this for good :)...

mirkolenz commented 6 months ago

I will try to answer all of these points:

The parameters you listed under this section are all parameters that do not give feedback, i.e. can only be "pushed". Would button also be applicable for general on/off or yes/no buttons? If so, how would I transmit the information which refers to "on"/"yes" etc. and which one to "off"/"no"...?

As you said, a button can only send some message and has no state, so a button makes sense to send the "DHW push" or activate comfort/reduced temperature when in "auto" mode. However, for all other parameters that actually have a state (like "DHW enabled"), a switch entity would be more appropriate.

This ID is unique to the BSB-LAN parameter, right? Could I just use the parameter number here? Or does it have to follow a certain pattern?

The unique id is not strictly necessary, but Home Assistant requires it in order to enable changes to the entity from the UI (e.g., renaming, changing the icon). This can be any string as long as it stays consistent throughout the entire lifetime of the entity. The KNX integration for instance creates some ID based on the defined group addresses, so you could use something like bsblan_$PARAMETER_ID.

Is this the same as MQTTTopicPrefix in BSB_LAN_config.h (which defaults to "BSB-LAN")?

Exactly

Can this (easily) be changed via the UI later on? If so, then I'd just omit this. Otherwise, what I could do is match a certain unit type (i.e. "temperature" to a certain icon. But that would result in the same icon for comfort temperature setpoint as well as buffer temperature. If that's ok, it would be great if you could point me to or provide me a list for each unit type.

This can be changed via the UI if a unique id is created for the entity. Some sane default would be nice. Here are all units of measurement currently available in HA:

Again, is this the MQTTTopicPrefix?

Yes

Can this be generally used for all parameters in the number section?

Yeah, in principle you can use it for any number. But a slider only makes sense if you also include a minimum, maximum, and a step value. Otherwise, the user risks entering some unsupported value just by sliding. I don't know if you can easily extract the value range from the BSB-LAN bus system. If so, a slider would certainly make sense for numbers.

What kind of device classes do exist in HA and what is this attribute used for? And is it required? Matching dozens of units of measurement to a specific device_class seems to me the most work at first glance...

If I recall correctly, the device class makes sure that users can choose their preferred unit from the UI (e.g., celsius/fahrenheit). All classes are listed here: https://developers.home-assistant.io/docs/core/entity/sensor/#available-device-classes

Should the numerical value come before the concrete description (here: On)? And could the switch type be used for other binary parameters as well (sich as Yes/No) and I would still use state_on for "Yes" for example?

This is basically the raw MQTT value provided by BSB-LAN. This value of course depends on the MQTT representation defined in the settings. I use plain values, so in case JSON is used, the corresponding template needs to be defined together with the Home Assistant MQTT entity.

Which kind of parameters need this?

I would say all sensors should have this. Otherwise, HA does not record a change if the value is the same. In case of the burner modulation, that would lead to the graph in HA showing a steady increase, whereas in reality it is a sudden rise and then a fall again. For continuous sensors like outside temperature it could however even make sense to set this to false.

Which kind of parameters need this?

The value template is needed in cases where the raw representation of the MQTT message does not correspond to some internal state of the entity. This is for instance always the case if using the JSON MQTT mode: You would need to parse the message first.

In case of the plain MQTT mode (which I use), this is needed if I have defined a numerical sensor with BSB-LAN sending the value --- to indicate that something is off (to differentiate from it being on with 0 percent of its capacity). IIRC this is needed for the modulation: In case the burner is not active, --- is sent by BSB-LAN which cannot be parsed by the MQTT integration of HA, so this first needs to be converted to some other value. The empty string used in the first two examples leads to no change being recorded by HA, whereas the value 0 in the last one records an actual zero. Using an empty string may not be desirable for every use case and sometimes leads to bugs (e.g., in the summer where there are no frequent changes to some value), but it worked well enough for me.

And why do you have to divide the mA value by 1000?

The last time I worked on these sensors, HA only supported A measurements, no milli-A. So I set the unit to A and divided the value by 1000. It seems that this is supported now however.

And last but not least: Is it really iif with two i?

Yes, it really is iif with to i. It is a special shorthand function: https://www.home-assistant.io/docs/configuration/templating/#immediate-if-iif

Do you have an example for !input at hand that I could include? Or would it be easier to create a "package" for this (and if so, do you know how or could point me to the right direction)?

I think the most straightforward way (both for implementation and user documentation) would be to generate a package: https://www.home-assistant.io/docs/configuration/packages

You would just set all MQTT parameters as needed and the user puts the following snippet in his/her config:

homeassistant:
  packages:
    bsblan: !include auto_generated_bsblan.yaml

This way, you will never be at risk of interfering with a user's own config.

I hope this answers at least some of the questions you may have. If anything is still unclear/sketchy feel free to come back :)

fredlcore commented 6 months ago

Thanks a lot! I hope with a few clarifications, I could get to work:

Again, thanks a lot, and I hope that if we can find general rules for force_update and value_template, this could work :)...

EDIT: Und noch eine Frage zur Logistik: Es können bei manchen Reglern durchaus an die 1000 Parameter sein, die da ausgegeben werden. Wie würde HA die denn dann sortieren? Würde es Sinn machen, vor den Klartext die Parameternummer und ggf. die Kategorie zu schreiben? Denn der eigentliche Parameter heißt oft nur "Komfort-Temperatur", egal ob in HK1, 2 oder 3...

mirkolenz commented 6 months ago

Silders: Theoretically, I could get the min/max values from the heater, the problem is that some of them "follow" a different parameter: For example you cannot set comfort temperature (710) setpoint below reduced temperature setpoint (712). So for those parameters where a min/max value cannot be safely determined, I just omit the mode: slider line or should I choose a different mode?

I think the default mode (i.e., if the mode is omitted) is slider. On alternative is box (which is just a text field). I agree that in such cases it may be best to use a regular input field.

force_update: You say that every parameter should have this, but then you say that ouside temperature should set it to false. I'm not able to determine based on the parameter alone which values would fall in one or the other category. Do you have a suggestion how to proceed here?

If in doubt, I would propose to set force_update to true for every parameter. The only negative thing is that it causes more data to be written, but since each value is so small, it does not really matter. Having it disabled however may lead to surprising statistics for parameters like the modulation.

value_template: Here it's also not clear to me which solution I should choose: empty string or zero? I can only set it for all parameters, not individualy. I may not even be able to determine which parameter can be "disabled" (use the '---').

In that case, my recommendation is to set the value to 0, because otherwise users may think that the integration stopped working because the statistics will seem as if they are no longer updated (e.g., in the summer). You can also safely add this value template to all parameters: If some of them never send the value ---, then the value template will just do nothing.

EDIT: Und noch eine Frage zur Logistik: Es können bei manchen Reglern durchaus an die 1000 Parameter sein, die da ausgegeben werden. Wie würde HA die denn dann sortieren? Würde es Sinn machen, vor den Klartext die Parameternummer und ggf. die Kategorie zu schreiben? Denn der eigentliche Parameter heißt oft nur "Komfort-Temperatur", egal ob in HK1, 2 oder 3...

Die Nummer würde ich nicht anzeigen, das wird das Dashboard recht unübersichtlich machen. Die Kategorie hingegen finde ich sinnvoll. Aber macht es überhaupt Sinn, alle Parameter zu exportieren? Man müsste doch nach meinem Verständnis noch immer die jeweiligen Werte in BSB-LAN an MQTT exportieren und ist hier doch weiterhin auf die 20 Parameter beschränkt, oder?

I hope this information is helpful and again sorry that I could not come up with the auto-discovery templates...

BTW: I found another nice overview of all device classes with their units of measurement: https://www.home-assistant.io/integrations/number/

fredlcore commented 6 months ago

Thanks for the clarifications (and sorry for everyone else for switching to German last night :) )... As for the parameter numbers and category names used as descriptions as well as the number of parameters: Maybe I don't understand HA well enough, but can I just use parameters in HA that are actively pushed by BSB-LAN? Of course, for the most used parameters, that is what I'd want, but if at some point I want to query a seldom used parameter, couldn't I do this with HA as well? If it's a requirement that the parameters are actively pushed from BSB-LAN, then of course the number is only limited to 20. But if I generate such a file in the context of creating the BSB_LAN_custom_defs.h, I wouldn't know which parameters the user would like to use. Then they would have to edit the generated file and remove all the hundreds of parameters they are not going to use. That is probably an error-prone process (you delete one line too much, the file no longer works, people come to me to solve the problem -> not good :) ). If they can remove all unwanted devices from the UI, then it's probably better, but I guess doing that for hundreds of parameters is no fun either.

An alternative would be to not do the conversion when processing the dump, but from the already generated BSB_LAN_custom_defs.h file. Then people would have to run a script and could provide the parameters they want to be part of the HA configuration file. That would require users to install a cross-platform script language such as Perl or Python to run this script. Not sure if that means that the auto-detection would be easier to handle after all :(...

mirkolenz commented 6 months ago

That is a good question…

As far as HA is concerned, it will display every parameter that is configured via the MQTT integration. If there are entities that are not written to by BSB-LAN, HA will display the entity status as unavailable.

I just did some more research and found that HA offers the option enabled_by_default for MQTT entities. If set to true (default), they are added to the user's dashboard, if set to false, they are disabled and thus hidden.

So exporting all available entities with the option enabled_by_default: false should work since the user can then just enable all of the entities that are published by BSB-LAN.

fredlcore commented 6 months ago

Thanks! And this enablement can be done in the UI or would users have to go through the config file and remove all these lines? And just to be sure: We're talking about the yaml file, not the MQTT auto-discovery (because you mention the MQTT integration, and I'm not quite sure what this refers to here...)

mirkolenz commented 6 months ago

Exactly, users can handle all of this from the UI. So even non-technical users would be able to achieve this.

And yes, this all refers to the yaml configuration and not the auto-discovery. Both options are possible by the MQTT integration, it just maps MQTT messages to Home Assistant entities 😄

fredlcore commented 6 months ago

Ok, great, then I'll give it a go and let's see how it turns out in the end - thank you!

jbaudoux commented 5 months ago

I'm looking forward on the first point of this issue which is publishing in MQTT the messages that are transmitted on the bus. Currently, you need to list the parameter to monitor and this will query the bus on a regular interval. However, there are several info messages that are collected by this BSB-LAN but they are not forwarded to MQTT. Has someone already started a PR on this matter? Otherwise, can I make such?

fredlcore commented 5 months ago

No, no one has started working on this matter yet. Please explain how you would approach this matter first before starting to implement something, because I don't see the benefit yet. Those data that BSB-LAN collects from broadcast bus messages are usually all mapped to some parameter already. Information that is not yet mapped to a parameter (off the cuff, I'm not quite sure which one that would be) would require a different form of sending data because right now, everything is based on sending parameter-based data. For just one or two pieces of data, I would not approve extensive code changes.

jbaudoux commented 5 months ago

Here is what I understand looking at the code. I may have missed something... When a telegram is transmitted on the bus, 2 functions are called: printTelegram and LogTelegram.

decoded messages

Currently printTelegram decodes the telegram according to the cmd table and print the result on Serial. First, you need a Serial connection to see the messages and you need to be close to the device which is not convenient in my case. My suggestion would be to allow to also send this in MQTT using the current parameter format : BSB-LAN/<param> <payload>.

As mqtt_sendtoBroker in mqtt_handler already relies on decodedTelegram, little change is required to this function. It would require to move the query() out of this method (btw that would be more in line with its name).

Thanks to this, you don't need to poll for the parameters. As soon as there is a change, this is notified to the MQTT broken. It is then up the the MQTT clients to subscribe to what they are interested in.

For the many like me using home assistant (but this is also valid for other clients), you would see in real time on the dashboard the changes you have subscribed to.

raw messages

Currently LogTelegram allows to log raw telegram but only on a SD card. My suggestion would be to allow to also send this in MQTT using a new topic format : BSB-LAN/<bus type>/<from>/<to> <payload>

The existing switch (logTelegram) would allow to choose which telegram to send in MQTT. The function would then also support CF_LOGMODE_MQTT

As this is more for debug, I would always send a plain text payload with a new small function mqtt_log_to_broker in mqtt_handler

fredlcore commented 5 months ago

I'd generally be in favor of abstracting log functionality so that it becomes transparent whether you log to SD card, MQTT or UDP. Since broadcast telegrams are not a reliable source of data, they should only be logged in addition to regular logging intervals because broadcast telegrams will not be caught by BSB-LAN in case it is doing something else in that very moment (such as printing a website or sending MQTT data to the broker. For example, if you want to log the burner status, and you missed the telegram that switches from off to on, you may only catch the next telegram which notifies that the burner is "off" - without you knowing that it was "on" in the meantime. That's why logging these parameters in regular intervals is a necessity. You might still miss changes back in forth that are happening in a timeframe shorter than the log interval, but in practice that is tolerable or can be remedied with a shorter log interval if necessary. Home automation systems such as FHEM can be configured to ignore data collected that is identical to the previously collected state.

Based on this, the detection of a broadcast telegram should check log_parameters[] whether the detected telegram is part of the log parameter list, and if so, set a flag that will execute MQTT and/or UDP logging towards the end of the loop function. To do that, the log_now variable could be evaluated. It is currently already used in the MQTT and UDP branches, so that would require only minimal changes in code. You would only have to specify that only one specific variable would have to be logged, not all of the log_parameters array. Also, the log interval should not be reset. The only drawback of this would be that another query/answer telegram pair would have to go on the bus, but that would be tolerable since it would prevent major code changes.

As for logging raw telegrams, I don't see the need to do that via MQTT. Already now you can just set debug settings from serial to telnet and have access to all raw data. For the usualy debugging cases, I need more than just the telegrams but also information about what else is going on. So adding a debug channel that does not contain all relevant information for me is not feasible.

jbaudoux commented 5 months ago

Indeed the code is currently not very thread safe but there is technically no reason for an ESP micro-controller to miss any message on the bus (arduino is less reliable on this). Besides from changing anything, we can rely on some facts:

I don't see the point of restricting the list of forwarded messages. It is up to the client to subscribe to what he wants. If it is to reduce network traffic, maybe an exclusion list of parameter but I would not even do that as the parameter could then also be dropped from your local BSB_LAN_custom_defs.h.

For logging, I would not touch log_now as this forcing the logging of all polled telegrams. I would 1/ move the call to printTelegram at the begging of LogTelegram instead of always repeating the 2 calls in the code 2/ after the call to printTelegram, decodedTelegram is populated. Then if MQTT is enabled, I would call mqtt_sendtoBroker that already relies on decodedTelegram. As said before, It would require to move the query() out of this function. 3/ continue with the existing logging to SD card.

For raw telegram, this is indeed not very useful and we can stick to existing solution.

fredlcore commented 5 months ago

What do you mean that you don't see the point in restricting the list of forwarded messages? Of course the client subscribes to those parameters that he wants. But if you want to know the burner status, then obviously, you'd want to log the burner status anyway and not risk relying only on broadcast messages.

Also, I mentioned that because log_now logs all parameters, it would have to be limited. The easiest way could be to set log_now with the parameter that needs to be logged (as parameter 1 is never going to be logged).

I will not support a solution that will only cater for MQTT and not UDP. Also, I'm not going to support a solution that requires major changes in code that hasn't been in demand for half a decade and will only affect the logging of 2-3 parameters.

fredlcore commented 5 months ago

Ok, finally I got an understanding of the concrete requirements for the MQTT autodiscovery by watching this video: https://www.youtube.com/watch?v=VHiCtZqllU8&feature=youtu.be So I managed to create a BSB-LAN device with several sensor entities, but I didn't manage to have any of them writeable. Or maybe I'm looking at the wrong place, but on the dashboard, I can only see the current value of each entity and cannot set a new one. This is the JSON I'm using to create the entitiy:

{
  "name":"710 - Komforttemperatur",
  "unique_id":"710-255-255",
  "state_topic":"BSB-LAN/710",
  "command_topic": "BSB-LAN",
  "unit_of_measurement":"°C",
  "device_class": "temperature",
  "command_template": "S710={{value}}",
  "value_template": "{{value}}",
  "mode": "slider",
  "device": {
    "name": "BSB-LAN",
    "identifiers": "BSB-LAN-DEADBEEF"
  }
}

Do you have any idea what I need to change?

jbaudoux commented 5 months ago

Do you have any idea what I need to change?

Check what is send to the broker: mosquitto_sub -h <mqtt_broker> -v -t 'BSB-LAN/#'

Telnet to your device to see in the log if the mqtt message has been received

Check if you enabled in the config to set that parameter

fredlcore commented 5 months ago

The latter seems to be the issue because when I change the topic to publish the device from sensor to climate, I can set the temperature and MQTT messages are sent from HA to the broker. But a climate device is not really suitable because most temperature parameters do not have a "mode" to be set etc. All I'm going to offer is a sensor that needs to be writeable (I know that sensors aren't but whatever the equivalent would be).

fredlcore commented 5 months ago

I changed the config topic from sensor to number and used this JSON structure:

{
  "name":"710 - Komforttemperatur",
  "unique_id":"710-255-255",
  "state_topic":"BSB-LAN/710",
  "unit_of_measurement":"°C",
  "device_class": "temperature",
  "command_template": "S710={{value}}",
  "value_template": "{{value}}",
  "mode": "slider",
  "device": {
    "name": "BSB-LAN",
    "identifiers": "BSB-LAN-DEADBEEF"
  }
}

It does not work, unless I switch from homeassistant/number/710/config to homeassistant/sensor/710/config

jbaudoux commented 5 months ago

Have you tried (you need to translate values to german...)

mqtt:
  climate:
    - name: "Thermostat 1 radiateurs"
      unique_id: bsb_lan_thermostat_1
      availability_topic: "BSB-LAN/status"
      payload_on: 1
      payload_off: 0
      modes:
        - auto
        - heat
        - cool
        - "off"
      mode_state_topic: "BSB-LAN/700"
      mode_state_template: >-
        {% set values = { '0 - Mode protection':'off', '1 - Automatique':'auto', '2 - Réduit':'cool', '3 - Confort':'heat'} %}
        {{ values[value] if value in values.keys() else 'off' }}
      mode_command_topic: "BSB-LAN"
      mode_command_template: >-
        {% set values = { 'off':'S700=0', 'auto':'S700=1', 'cool':'S700=2', 'heat':'S700=3'} %}
        {{ values[value] if value in values.keys() else '0' }}
      current_temperature_topic: "BSB-LAN/8740"
      current_temperature_template: >-
        {% if value == '---' %}
            {{ 'None' }}
        {% else %}
            {{ value }}
        {% endif %}
      min_temp: 13
      max_temp: 24
      temp_step: 0.5
      temperature_high_state_topic: "BSB-LAN/710"
      temperature_high_command_topic: "BSB-LAN"
      temperature_high_command_template: "{{'S710='+ (value| string)}}"
      temperature_low_state_topic: "BSB-LAN/712"
      temperature_low_command_topic: "BSB-LAN"
      temperature_low_command_template: "{{'S712='+ (value| string)}}"
      device:
        {
          identifiers: "BSB-LAN-DEADBEEF"],
          name: "BSB-LAN",
        }

You need to add the entity, do not try to add a card

fredlcore commented 5 months ago

But a climate device is not really suitable because most temperature parameters do not have a "mode" to be set etc.

mqtt: climate:

So since I'm explicitly not looking for a climate device, I don't know how this is going to help me. I need a generic (numerical) device where I can have some (graphical) way to change the values. Everything else won't work because it requires more information than I can provide in the process...

fredlcore commented 5 months ago

I found this one here in the HA forums: https://community.home-assistant.io/t/mqtt-discovery-for-custom-device-properties/260941 Someone seems to have the same problem as I do:

However, I couldn’t find a way how to discover simple properties like a list of states, e.g. [“left”, “right”, “stop”], which can be selected and read back via HA a numeric input There are a number of device types supported by MQTT discovery, which comprise properties very close to my needs. HVAC devices support modes, though limited to [“auto”, “off”, “cool”, “heat”, “dry”, “fan_only”], but essentially this is what I need, just for custom values. Lights support a brightness property - but I don’t need a complete light device, just a numeric input with a custom range, lets say: 1…42. I can’t figure out how to do that, and I can’t believe it’s not possible.

To which someone else replies:

You can’t use it to discover what are known as ‘helpers’ such as input_datetime, input_select, input_text, input_number, timers, etc.

So does that mean that if I want to advertise writeable parameters such as heat curve/slope values or percentage values, I have to "force" these into a climate device or a slider? Ok, this thread was from 2020, but is it still not possible to have something like in the webinterface of BSB-LAN where it shows the current value and lets you enter a new one to be set? @mirkolenz, how do you deal with these in JSON? Or have you only used yaml and therefore didn't encounter this problem? If there is no solution to this, I'll be forced to only advertise read-only parameters, with the exception of switches maybe...

fredlcore commented 5 months ago

OTOH, it says here that number should be available also for MQTT autodiscovery: https://www.home-assistant.io/integrations/number.mqtt/

fredlcore commented 5 months ago

Ok, command_topic needs to be set with the number integration. Step by step ;)...

fredlcore commented 5 months ago

Ok, I've made quite some progress now, but one thing puzzles me: When I use a switch, I can set its value correctly via an MQTT message, and if I toggle the switch in the webinterface, it sends the correct MQTT message out. But after it has done so, it falls back to its previous position. Any ideas anyone?

{
  "name":"726 - Heizkennlinien-Adaption",
  "unique_id":"726-255-255",
  "state_topic":"BSB-LAN/726",
  "state_on": "1 - On",
  "state_off": "0 - Off",
  "command_topic":"BSB-LAN",
  "payload_on": "S726=1",
  "payload_off": "S726=0",
  "device": {
    "name": "BSB-LAN",
    "identifiers": "BSB-LAN-DEADBEEF"
  }
}
fredlcore commented 5 months ago

Not sure, but could it be that a switch expects the device to return its current status? At least I see in mqtt_handler.h that after receiving a SET command via MQTT, BSB-LAN returns the current state to the broker. So this problem might go away when I implement this into BSB-LAN. But if someone could confirm this behaviour by HomeAssistant or MQTT in general, then I could put this matter to rest...

mirkolenz commented 5 months ago

Hi! Great to see that you already made some progress! Your assumption is correct: The MQTT integration expects to receive a message containing the new value, otherwise it will return to its previous state. Maybe you can adjust it using the optimistic parameter. In my own setup, BSB-LAN responds with the new value so it was never an issue for me.

I remember seeing this strange behavior as well if BSB-LAN does not have the writable setting enabled. So could it also be that the message was not properly processed by BSB-LAN?

fredlcore commented 5 months ago

Thanks for confirming this - then it should not be a problem because currently, I'm just sending JSON structures directly via mosquitto_pub, so obviously, there is no response from BSB-LAN yet. But once this will be implemented into BSB-LAN, the same mechanisms will kick in, so it should be fine then. My plan now is as follows:

What I'm still not sure about is whether to send only those parameters set up as logging parameters - or send all available parameters, which can be up to 1.000 or more. Doing this during every boot might be quite time-consuming and worrying for the user if the webinterface is not reachable after a minute or so. But I'll run some tests to see how it behaves.

As an alternative to only send logged MQTT parameters, I'm thinking of always sending queried parameters also via MQTT if MQTT is enabled. Would that make sense? Because if it's possible to send a URL-based query from HA to BSB-LAN, you could select a selection of parameters independent from the log parameters that would be queried and then sent to the MQTT broker. This way, you could also easily implement different log intervals for different parameters. So you could query parameters such as 700, 1000 etc. just every couple of hours, and other parameters every few minutes or so. All this in addition to those parameters that you log via MQTT via BSB-LAN's logging functionality.

mirkolenz commented 5 months ago

IIRC parameter states can already be requested via MQTT from BSB-LAN with the current version. In other words, publishing a message on the main BSB-LAN topic with the parameter number as the payload will trigger BSB-LAN to read the value from the boiler and publish it in the correct topic. I set up an automation that does this once a day for some parameters that I did not add to the logging parameters.

fredlcore commented 5 months ago

Yes, you can publish parameter queries as well as SET commands by publishing the corresponding messages, but is that as easy as sending one URL with all the desired parameters? Don't get me wrong, I'd prefer not to add this functionality, but since I don't use MQTT myself, I'm not sure if it's as easily possible to query a number of parameters (maybe even from different device IDs) as it would be to send a URL like this: http://bsb-lan.local/700/710/8700!1/8000-8009 and have all these parmeters sent to the broker. If it is, then great :) - then I can focus on generating these JSON messages...

mirkolenz commented 5 months ago

Ok, that totally makes sense. What about adding support for publishing a list pf parameters (separated by commas or slashes?) to the main bsblan topic and then read and publish their values? In that case, it should work nicely in MQTT itself without needing to resort to another protocol.

fredlcore commented 5 months ago

Ok, let me think about this. Makes more sense, but would be unrelated to the things to be done for the auto-detection right now. Could you please open up a new feature request for that and explain how it can be done at the moment and how it should be possible to do in the future? Thanks!

mirkolenz commented 5 months ago

Ok, I thought this was the answer you were looking for 😄 I will open a new issue once I get around tuning my smarthome setup again.

jbaudoux commented 5 months ago

Yes, you can publish parameter queries as well as SET commands by publishing the corresponding messages, but is that as easy as sending one URL with all the desired parameters? Don't get me wrong, I'd prefer not to add this functionality, but since I don't use MQTT myself, I'm not sure if it's as easily possible to query a number of parameters (maybe even from different device IDs) as it would be to send a URL like this: http://bsb-lan.local/700/710/8700!1/8000-8009 and have all these parmeters sent to the broker. If it is, then great :) - then I can focus on generating these JSON messages...

This looks weird to me. As you said, you can just make many queries. One query to get or set a value, and one response with the state. It's simple. Why building complex multi parameters messages? I don't find it great :/ Also don't forget that the topic serves as routing and not as query string. If you want to achieve such, the list of parameters should be in the payload, not in the topic. But again, it will still be weird as one parameter like 700 is an element of the routing. Proper topics are like this:

You start from generic location to more precise. Then inside the boiler, you route to bsb bus. And on the bsb bus, you route to 700. All in lower case letters. BSB-LAN can be the MQTT client id but should not be used as a proper topic. Those are best practice.

You can then subscribe to whatever happens in the basement, or on the boiler, or only bsb bus, or a specific parameter or a specific command. If you want to subscribe to all query commands for the boiler bsb bus, then you can subscribe to basement/boiler/bsb/+/get

fredlcore commented 5 months ago

@mirkolenz: The thing is that while your suggestion makes more sense since it doesn't require switching to a different protocol, it means that I have to rework the MQTT message parser. Whereas my suggestion - which was to add a call to the MQTT publish function whenever a parameter is set/queried via HTTP - would have only required adding this call in two or three places. But as I said, you are right that yours is the cleaner solution, but it's a different "Baustelle" for me ;)...

@jbaudoux: You didn't seem to understand the idea. http://bsb-lan.local/700/710/8700!1/8000-8009 is not an MQTT topic but a URL. The idea was to have quick and easy way to tell BSB-LAN to publish a number of updated parameter values. Other than that, your suggestions may be "clean", but would break compatibility with hundreds of installations. Making such changes would require a significant benefit in order to offset the expected amount of support questions I would have to face.

jbaudoux commented 5 months ago

@fredlcore When you said send that to the broker, I thought you where talking about MQTT broker... Can you open another issue for non MQTT related discussions?

There is no need for breaking changes. If you expose a few config parameters, you get there. Your current config:

Alternative config:

Parameter and command is inverted from my example but doesn't matter. I have no intention in doing that. Just wanted to share some insights so that any further improvement on MQTT doesn't derive from good practices. We can close this parenthesis :)

fredlcore commented 5 months ago

@jbaudoux: Why not just use the current MQTTTopicPrefix variable and set it to basement/boiler/bsb/state/set? That's possible now already. But once we change the payload sent to a topic, all BSB-LAN users who are currently having configurations such as payload_on = 'S726=1' would have to change their configurations for maybe dozens of parameters. That could be done, but not without a significant benefit.

fredlcore commented 4 months ago

Ok, I think I'm almost done, but I need your help because this seems to be a Home Assistant issue: With the most recent version, there is now a function mqtt_auto_detect which takes true (send "create" messages) or false (send "remove", i.e. empty, messages). To test it, you have to enable #define CUSTOM_COMMANDS in BSB_LAN_config.h and add these lines to BSB_LAN_custom_setup.h:

mqtt_connect();
mqtt_send_discovery(REMOVE);
mqtt_send_discovery(CREATE);

It works for almost all my parameters, except those with temperatures. The JSON seems to be valind, for example this one for parameter 712:

 {   "name":"712 - Raumtemperatur-Reduziertsollwert",   "unique_id":"712-90-239",   "state_topic":"BSB-LAN/712","icon": "mdi:numeric",   "unit_of_measurement":"°C",   "command_topic":"BSB-LAN",   "command_template": "S712!0={{value}}",   "device": {     "name":"BSB-LAN",     "identifiers":"BSB-LAN-AC67B24657F4"   } }

But none of these temperature parameters show up. Whereas parameter 720 shows up with this JSON structure:

 {   "name":"720 - Heizkennlinien-Steilheit",   "unique_id":"720-90-239",   "state_topic":"BSB-LAN/720","icon": "mdi:numeric",   "unit_of_measurement":"",   "command_topic":"BSB-LAN",   "command_template": "S720!0={{value}}",   "device": {     "name":"BSB-LAN",     "identifiers":"BSB-LAN-AC67B24657F4"   } }

The topics are also almost the same: homeassistant/text/712_90_239/config and homeassistant/text/720_90_239/config

So I'm counting on you Home Assistant experts here to run some tests with your setups and let me know what is the (hopefully final) problem here...

mirkolenz commented 4 months ago

Great news, thank you for your work! I am currently busy working on my PhD and heavily rely on automations built on BSB-LAN parameters in my Home Assistant config, so I do not want to risk breaking my setup and possibly debugging the issues for a few hours. I could test the implementation some time in the summer, but I don't know if that is already too late...