MQTT publish and statestream error after core 2023.4 (.1)update

frigo99 commented 1 year ago

The problem

hi, I use automations to publish some sensor data on MQTT(using mosquito add on) automation.yaml

`- id: '1673992974258' alias: MQTT - GridWatts(gridactivepowerfreeds=-1*grid_active_power) description: '' trigger:

platform: state entity_id: sensor.gridactivepowerfreeds condition: [] action:
service: mqtt.publish data_template: payload: '{{trigger.to_state.state}}' topic: Inverter/GridWatts mode: single`

and also sending directly some values configuration.yaml

`#MQTT - Publish, send data from power meter/inverter/calculated values mqtt_statestream: base_topic: Inverter publish_attributes: true publish_timestamps: true include: entity_globs:

sensor.grid_active_power mqtt: broker: 192.168.0.102 username: mqtt-xxxx password: xxxx`

After update to latest core revision(2023.4.1) mqtt publish service is not working, also when I look into a new automation/action/call service, I cannot find anymore "mqtt.publish" option.

before was possible to select several mqtt services in automation action call service dropdown.

now is not existing anymore these options

The integration MQTT core mosquito in settings/integrations is shown as "! Not loaded"...I tried to resubmit/re-configure. seems no success.

In the MQTT - explorer add on of course there are no visible publishing action.

Thank you in advance for your kind support!

What version of Home Assistant Core has the issue?

2023.4.1

What was the last working version of Home Assistant Core?

2023.3.6

What type of installation are you running?

Home Assistant OS

Integration causing the issue

MQTT(publish and statestream), Automation

Link to integration documentation on our website

No response

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

Logger: homeassistant.config
Source: config.py:882
First occurred: 1:11:28 PM (1 occurrences)
Last logged: 1:11:28 PM

Invalid config for [mqtt]: The 'broker' option near /config/configuration.yaml:116 has been removed, please remove it from your configuration for dictionary value @ data['mqtt']. Got {'broker': '192.168.0.102', 'username': 'mqtt-xxxx', 'password': 'xxxxx', 'sensor': [{'name': 'freeds_pwm', 'state_topic': 'freeds_23dc/pwm', 'unit_of_measurement': '%', 'device_class': 'power'}, {'name': 'freeds_wsolar', 'state_topic': 'freeds_23dc/wsolar', 'unit_of_measurement': 'W', 'device_class': 'power'}, {'name': 'freeds_wgrid', 'state_topic': 'freeds_23dc/wgrid', 'unit_of_measurement': 'W', 'device_class': 'power'}, {'name': 'freeds_wtogrid', 'state_topic': 'freeds_23dc/wtogrid', .... (See /config/configuration.yaml, line 116). Please check the docs at https://www.home-assistant.io/integrations/mqtt

Logger: homeassistant.setup
Source: setup.py:221
First occurred: 1:11:28 PM (1 occurrences)
Last logged: 1:11:28 PM

Setup failed for mqtt: Invalid config.

Logger: homeassistant.setup
Source: setup.py:203
First occurred: 1:11:28 PM (1 occurrences)
Last logged: 1:11:28 PM

Unable to set up dependencies of mqtt_statestream. Setup failed for dependencies: mqtt

Logger: homeassistant.setup
Source: setup.py:205
First occurred: 1:11:28 PM (1 occurrences)
Last logged: 1:11:28 PM

Setup failed for mqtt_statestream: (DependencyError(...), 'Could not setup dependencies: mqtt')

Logger: homeassistant.components.automation.mqtt_pvwattstotal_active_power
Source: helpers/script.py:410
Integration: Automation (documentation, issues)
First occurred: 1:12:18 PM (479 occurrences)
Last logged: 1:39:08 PM

MQTT - PvWattsTotal(active_power): Error executing script. Service not found for call_service at pos 1: Unable to find service mqtt.publish

Additional information

I got a notification: "Invalid config The following integrations and platforms could not be set up:

mqtt (Show logs) mqtt_statestream (Show logs) Please check your config and logs."

home-assistant[bot] commented 1 year ago

mqtt_statestream documentation mqtt_statestream source

home-assistant[bot] commented 1 year ago

Hey there @emontnemery, @jbouwh, mind taking a look at this issue as it has been labeled with an integration (mqtt) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `mqtt` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign mqtt` Removes the current integration label and assignees on the issue, add the integration domain after the command.

_{^{(message by CodeOwnersMention)}}

mqtt documentation mqtt source _{^{(message by IssueLinks)}}

frigo99 commented 1 year ago

what I forgot to mention that after rolling back to 2023.3.6 core backup everything is working fine....

jbouwh commented 1 year ago

Did previously ignore that you still have mqtt broker settings in your configuration.yaml. Those settings were removed. See the release notes breaking changes section. You need to remove any broker settings. The have been already migrated to the entry in previous releases. You can check the current broker setting via the UI.

AnanasAPod commented 1 year ago

Same problem here. Rolling back to 2023.3.0 worked fine.

jbouwh commented 1 year ago

Same problem here. Rolling back to 2023.3.0 worked fine.

Did you check configuration.yaml?

AnanasAPod commented 1 year ago

Here is my yaml. Have nothing regarding mqtt in it.

# Loads default set of integrations. Do not remove.
default_config:

# Load frontend themes from the themes folder
frontend:
  themes: !include_dir_merge_named themes

# Text to speech
tts:
  - platform: google_translate

automation: !include automations.yaml
script: !include scripts.yaml
scene: !include scenes.yaml

homeassistant:
  name: Home
  unit_system: metric
  # etc
  customize:
    sensor.tasmota_sml_total_kwh:
      state_class: total_increasing
      device_class: energy
      unit_of_measurement: kWh
      unit_prefix: k
    sensor.tasmota_sml_export_total_kwh:
      state_class: total_increasing
      device_class: energy
      unit_of_measurement: kWh
      unit_prefix: k
    sensor.tasmota_sml_curr_w:
      device_class: SensorDeviceClass.POWER
      unit_of_measurement: W

panel_iframe:
  router:
    title: "FritzBox"
    url: "http://192.168.178.1"
    icon: "mdi:dns"
    require_admin: true
  portainer:
    title: "Portainer"
    url: "http://192.168.178.2:9000/#!/2/docker/containers"
    icon: "mdi:dns"
  elecpow:
    title: "PV-Anlage"
    url: "http://192.168.178.50/index.htm"
    icon: "mdi:solar-power-variant"
  nodered:
    title: "Node-RED"
    url: "http://192.168.178.4:1880"
    icon: "mdi:nodejs"

spotcast:
  sp_dc: $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
  sp_key: $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
  country: DE #optional, added in 3.6.24

jbouwh commented 1 year ago

Are you seeing Invalid config for [mqtt]: The 'broker' option near /config/configuration.yaml:116 has been removed, please remove it from your configuration in your logs or something similar?

jbouwh commented 1 year ago

Here is my yaml. Have nothing regarding mqtt in it.

# Loads default set of integrations. Do not remove.
default_config:

# Load frontend themes from the themes folder
frontend:
  themes: !include_dir_merge_named themes

# Text to speech
tts:
  - platform: google_translate

automation: !include automations.yaml
script: !include scripts.yaml
scene: !include scenes.yaml

homeassistant:
  name: Home
  unit_system: metric
  # etc
  customize:
    sensor.tasmota_sml_total_kwh:
      state_class: total_increasing
      device_class: energy
      unit_of_measurement: kWh
      unit_prefix: k
    sensor.tasmota_sml_export_total_kwh:
      state_class: total_increasing
      device_class: energy
      unit_of_measurement: kWh
      unit_prefix: k
    sensor.tasmota_sml_curr_w:
      device_class: SensorDeviceClass.POWER
      unit_of_measurement: W

panel_iframe:
  router:
    title: "FritzBox"
    url: "http://192.168.178.1"
    icon: "mdi:dns"
    require_admin: true
  portainer:
    title: "Portainer"
    url: "http://192.168.178.2:9000/#!/2/docker/containers"
    icon: "mdi:dns"
  elecpow:
    title: "PV-Anlage"
    url: "http://192.168.178.50/index.htm"
    icon: "mdi:solar-power-variant"
  nodered:
    title: "Node-RED"
    url: "http://192.168.178.4:1880"
    icon: "mdi:nodejs"

spotcast:
  sp_dc: $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
  sp_key: $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
  country: DE #optional, added in 3.6.24

Tasmota leans on MQTT and triggers installation. You should check your MQTT config entry settings.

AnanasAPod commented 1 year ago

Here the logs. I don't have that much experience. I am a Beginner.

Home Assistant Core 2023.4.1

2023-04-08 13:38:15.523 WARNING (SyncWorker_1) [homeassistant.loader] We found a custom integration hacs which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2023-04-08 13:38:15.524 WARNING (SyncWorker_1) [homeassistant.loader] We found a custom integration spotcast which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2023-04-08 13:38:15.525 WARNING (SyncWorker_1) [homeassistant.loader] We found a custom integration climate_group which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2023-04-08 13:38:15.525 WARNING (SyncWorker_1) [homeassistant.loader] We found a custom integration better_thermostat which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2023-04-08 13:38:16.747 WARNING (Recorder) [homeassistant.components.recorder.util] The system could not validate that the sqlite3 database at //config/home-assistant_v2.db was shutdown cleanly
2023-04-08 13:38:16.780 WARNING (Recorder) [homeassistant.components.recorder.util] Ended unfinished session (id=37 from 2023-04-08 13:23:00.170046)
2023-04-08 13:38:21.037 ERROR (MainThread) [homeassistant.config_entries] Error setting up entry Pi-Hole for pi_hole
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/config_entries.py", line 383, in async_setup
result = await component.async_setup_entry(hass, self)
File "/usr/src/homeassistant/homeassistant/components/pi_hole/__init__.py", line 97, in async_setup_entry
await er.async_migrate_entries(hass, entry.entry_id, update_unique_id)
File "/usr/src/homeassistant/homeassistant/helpers/entity_registry.py", line 1213, in async_migrate_entries
ent_reg.async_update_entity(entry.entity_id, **updates)
File "/usr/src/homeassistant/homeassistant/helpers/entity_registry.py", line 886, in async_update_entity
return self._async_update_entity(
File "/usr/src/homeassistant/homeassistant/helpers/entity_registry.py", line 832, in _async_update_entity
raise ValueError(
ValueError: Unique id '8675aa2d5e20dc44b9f008f07ae0fb28/ads_percentage_today' is already in use by 'sensor.pi_hole_ads_percentage_blocked_today'
2023-04-08 13:38:29.029 WARNING (MainThread) [homeassistant.components.sensor] Sensor sensor.tasmota_sml_server_id has device class None, state class None and unit thus indicating it has a numeric value; however, it has the non-numeric value: 0a01484c59020012fce5 (<class 'str'>); Please update your configuration if your entity is manually configured, otherwise create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+tasmota%22
2023-04-08 13:38:51.100 WARNING (MainThread) [homeassistant.components.sensor] Setup of sensor platform fritz is taking over 10 seconds.
2023-04-08 13:38:59.122 WARNING (Recorder) [homeassistant.components.recorder.migration] Dropping index `ix_events_context_id` from table `events`. Note: this can take several minutes on large databases and slow computers. Please be patient!
2023-04-08 13:38:59.703 WARNING (Recorder) [homeassistant.components.recorder.migration] Dropping index `ix_states_context_id` from table `states`. Note: this can take several minutes on large databases and slow computers. Please be patient!
2023-04-08 13:39:02.019 WARNING (Recorder) [homeassistant.components.recorder.migration] Dropping index `ix_states_entity_id_last_updated_ts` from table `states`. Note: this can take several minutes on large databases and slow computers. Please be patient!
2023-04-08 13:39:23.673 WARNING (MainThread) [homeassistant.components.sensor] Sensor sensor.battery_voltage has device class battery, state class measurement and unit V thus indicating it has a numeric value; however, it has the non-numeric value: 3,49 (<class 'str'>); Please update your configuration if your entity is manually configured, otherwise create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+mqtt%22
2023-04-08 13:39:23.679 WARNING (MainThread) [homeassistant.components.sensor] Sensor sensor.battery_charge has device class battery, state class measurement and unit % thus indicating it has a numeric value; however, it has the non-numeric value: 40,5 (<class 'str'>); Please update your configuration if your entity is manually configured, otherwise create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+mqtt%22

Home Assistant Core 2023.3.0

2023-04-08 13:50:14.173 WARNING (SyncWorker_3) [homeassistant.loader] We found a custom integration hacs which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2023-04-08 13:50:14.174 WARNING (SyncWorker_3) [homeassistant.loader] We found a custom integration spotcast which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2023-04-08 13:50:14.174 WARNING (SyncWorker_3) [homeassistant.loader] We found a custom integration climate_group which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2023-04-08 13:50:14.175 WARNING (SyncWorker_3) [homeassistant.loader] We found a custom integration better_thermostat which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2023-04-08 13:50:20.417 ERROR (MainThread) [homeassistant.config_entries] Migration handler not found for entry SML Consumption (Total) for utility_meter
2023-04-08 13:50:28.007 WARNING (MainThread) [homeassistant.components.sensor] Sensor sensor.tasmota_sml_server_id has device class None, state class None and unit thus indicating it has a numeric value; however, it has the non-numeric value: 0a01484c59020012fce5 (<class 'str'>); Please update your configuration if your entity is manually configured, otherwise create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+tasmota%22
2023-04-08 13:50:43.952 WARNING (Recorder) [homeassistant.components.recorder.migration] Database is about to upgrade from schema version: 41 to: 35
2023-04-08 13:50:53.658 WARNING (MainThread) [homeassistant.components.sensor] Sensor sensor.battery_voltage has device class battery, state class measurement and unit V thus indicating it has a numeric value; however, it has the non-numeric value: 3,49 (<class 'str'>); Please update your configuration if your entity is manually configured, otherwise create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+mqtt%22
2023-04-08 13:50:53.665 WARNING (MainThread) [homeassistant.components.sensor] Sensor sensor.battery_charge has device class battery, state class measurement and unit % thus indicating it has a numeric value; however, it has the non-numeric value: 40,5 (<class 'str'>); Please update your configuration if your entity is manually configured, otherwise create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+mqtt%22

jbouwh commented 1 year ago

I am not seeing any MQTT logging. You should have MQTT set up under Settings->Integrations. If the settings are incorrect you can correct them or you can remove the entry and set up a new one.

jbouwh commented 1 year ago

2023-04-08 13:50:53.658 WARNING (MainThread) [homeassistant.components.sensor] Sensor sensor.battery_voltage has device class battery, state class measurement and unit V thus indicating it has a numeric value; however, it has the non-numeric value: 3,49 (<class 'str'>); Please update your configuration if your entity is manually configured, otherwise create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+mqtt%22
2023-04-08 13:50:53.665 WARNING (MainThread) [homeassistant.components.sensor] Sensor sensor.battery_charge has device class battery, state class measurement and unit % thus indicating it has a numeric value; however, it has the non-numeric value: 40,5 (<class 'str'>); Please update your configuration if your entity is manually configured, otherwise create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+mqtt%22

It seem MQTT is running correctly. Besides of some issues with some sensors. You can address those at the contributer of the integration that supplies the entities.

AnanasAPod commented 1 year ago

2023-04-08 13:50:53.658 WARNING (MainThread) [homeassistant.components.sensor] Sensor sensor.battery_voltage has device class battery, state class measurement and unit V thus indicating it has a numeric value; however, it has the non-numeric value: 3,49 (<class 'str'>); Please update your configuration if your entity is manually configured, otherwise create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+mqtt%22
2023-04-08 13:50:53.665 WARNING (MainThread) [homeassistant.components.sensor] Sensor sensor.battery_charge has device class battery, state class measurement and unit % thus indicating it has a numeric value; however, it has the non-numeric value: 40,5 (<class 'str'>); Please update your configuration if your entity is manually configured, otherwise create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+mqtt%22

It seem MQTT is running correctly. Besides of some issues with some sensors. You can address those at the contributer of the integration that supplies the entities.

Exactly in the older version 2023.03 it works perfectly. I made the logs from 2023.04 and 2023.03 above. I can easily switch running in Docker.

frigo99 commented 1 year ago

thanks for the fast reaction! removing all mqtt relating setup/commands from configuration.yaml and a restart seems was the solution for me. publishing is working through automation/action/call service/mqtt.publish. also MQTT core mosquito is OK. now I have to setup a MQTT readout to take some datas and bring for further processing in HA, can you please recommend simple solutions please? best regards!

jbouwh commented 1 year ago

thanks for the fast reaction! removing all mqtt relating setup/commands from configuration.yaml and a restart seems was the solution for me. publishing is working through automation/action/call service/mqtt.publish. also MQTT core mosquito is OK. now I have to setup a MQTT readout to take some datas and bring for further processing in HA, can you please recommend simple solutions please? best regards!

Nice your issue was solved. Using state_stream and trigger might not be the best approach. I would us a template sensor for this. See: https://www.home-assistant.io/integrations/template/

For for more help and info use https://community.home-assistant.io/

home-assistant / core