Duplicated entities after upgrading ESPHome

[smoki3]

The problem

After upgrading to 2023.12 I have duplicated entities in home assistant.

Look like it has something todo with the "ä,ü,ö,ß" symbols. Before the "ü" get converted to an "u" now its replay by a "_"

entity name before 2023.12:

sensor.zisterne_fullstand

entity name after 2023.12:

sensor.zisterne_f_llstand

Issue for HA: https://github.com/home-assistant/core/issues/106159

Which version of ESPHome has the issue?

2023.12

What type of installation are you using?

Home Assistant Add-on

Which version of Home Assistant has the issue?

2023.12

What platform are you using?

ESP8266

Board

No response

Component causing the issue

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

[MarcoL1975]

Have the same problem, all my original sensors etc are renamed to xxxx_2! Did a rollback to previous version!

[Plawasan]

I assume this has to do with https://github.com/esphome/esphome/pull/5810 .. luckily I upgraded only 1 of my 20+ ESPHome devices before I noticed it..

Example where this was truly a breaking change:

button:
  - platform: template
    name: "${esp_name} Tilt 66%" 
    id: button_tilt_66

Original entity name was button.cover_office_tilt_66, new entity is button.cover_office_tilt_66_2

Initially I saw both sets of entities as active and enabled, now after a while the old set is the usual "no longer provided by the integration" but can't be deleted without deleting the entire device and adding it back again, which the also fixes the entity naming issue

[TripelJ]

I also have the same issue. I don't have any special charakters in my name. I unfortunately updated all mine. But interestingly it is only some devices that have add a xxx_2 entity for all on the device, other devices some stayed where only a few changed but not all. Also have some that didn't change at all. Do you say it's fixed if you roll back ? Of course my important one my Power (Glow) devices is the one most fucked up with all the renames so doesn't track anything right now.

[MarcoL1975]

I also have the same issue. I don't have any special charakters in my name. I unfortunately updated all mine. But interestingly it is only some devices that have add a xxx_2 entity for all on the device, other devices some stayed where only a few changed but not all. Also have some that didn't change at all. Do you say it's fixed if you roll back ? Of course my important one my Power (Glow) devices is the one most fucked up with all the renames so doesn't track anything right now.

For me the rollback worked yes!

[TripelJ]

I also have the same issue. I don't have any special charakters in my name. I unfortunately updated all mine. But interestingly it is only some devices that have add a xxx_2 entity for all on the device, other devices some stayed where only a few changed but not all. Also have some that didn't change at all. Do you say it's fixed if you roll back ? Of course my important one my Power (Glow) devices is the one most fucked up with all the renames so doesn't track anything right now.

For me the rollback worked yes!

Haven't tried that before, but I need to find out then. Thanks.

[Plawasan]

I also have the same issue. I don't have any special charakters in my name. I unfortunately updated all mine. But interestingly it is only some devices that have add a xxx_2 entity for all on the device, other devices some stayed where only a few changed but not all. Also have some that didn't change at all. Do you say it's fixed if you roll back ? Of course my important one my Power (Glow) devices is the one most fucked up with all the renames so doesn't track anything right now.

For me the rollback worked yes!

Haven't tried that before, but I need to find out then. Thanks.

In my experience, if you haven't previously changed any of the default entity names in HA then you can also remove the device, add it back in and it should have the right entity names (as long as the issue is duplication)

[MarcoL1975]

I also have the same issue. I don't have any special charakters in my name. I unfortunately updated all mine. But interestingly it is only some devices that have add a xxx_2 entity for all on the device, other devices some stayed where only a few changed but not all. Also have some that didn't change at all. Do you say it's fixed if you roll back ? Of course my important one my Power (Glow) devices is the one most fucked up with all the renames so doesn't track anything right now.

For me the rollback worked yes!

Haven't tried that before, but I need to find out then. Thanks.

In HA go to backups, there you should find the backup for the update of ESPHome add-on. You can restore this backup to previous ESPHome version.

[dvb6666]

Same problem. All entities were duplicated (with suffix "_2") after upgrading to 2023.12.0:

[TripelJ]

I also have the same issue. I don't have any special charakters in my name. I unfortunately updated all mine. But interestingly it is only some devices that have add a xxx_2 entity for all on the device, other devices some stayed where only a few changed but not all. Also have some that didn't change at all. Do you say it's fixed if you roll back ? Of course my important one my Power (Glow) devices is the one most fucked up with all the renames so doesn't track anything right now.

For me the rollback worked yes!

Haven't tried that before, but I need to find out then. Thanks.

In HA go to backups, there you should find the backup for the update of ESPHome add-on. You can restore this backup to previous ESPHome version.

Ohh yeah of course. Thx. And then just build and install previous firmware on the device.

[shammysha]

sanitize() lost "-" in allowed chars

[TripelJ]

I also have the same issue. I don't have any special charakters in my name. I unfortunately updated all mine. But interestingly it is only some devices that have add a xxx_2 entity for all on the device, other devices some stayed where only a few changed but not all. Also have some that didn't change at all. Do you say it's fixed if you roll back ? Of course my important one my Power (Glow) devices is the one most fucked up with all the renames so doesn't track anything right now.

For me the rollback worked yes!

Haven't tried that before, but I need to find out then. Thanks.

In HA go to backups, there you should find the backup for the update of ESPHome add-on. You can restore this backup to previous ESPHome version.

Can confirm rolling back fixed it for me on those that was an issue. I need to learn not to jump on the first release even though it might be exciting :) or have a test dedicated device. One thing though is this coming in from the add on it self or after the devices get's the upgraded version rolled out.?

[OGB1952]

Same issue here, my HA is currently just displaying dozens of entity unavailable messages, with many automations broken as a consequence. In my case the problem seems to impact every entity connected with an ESHPHome instance running on ESP32C3 devices. At least for me, it's not impacting ESP82266 based devices for some reason. The database looks a bit screwed up, as there are dozens of duplicated entities, with the subsequent loss of history for them.

[Arnsteinl]

I somehow seem to have esp home update without my knowledge, and breaking a lot of HA. Any idea how do disable. It seems like every other update breaks a lot of HA.

Edit: Found it. The last update broke a lot of my HA. Its will take days to fix all the dashboards and automation that is broken......

[sheaur]

Same problem here, it's because of the special characters (á, é, í, ó, ú or ñ)

[TripelJ]

Same problem here, it's because of the special characters (á, é, í, ó, ú or ñ)

I don't have any special characters in mine at all, and it still happened. Maybe the regex didn't work as expected and created new entities despite not having any reason too. Should the new 12.1 fix these issues already if I am reading right?

[sheaur]

Problem with 2023.12.1 version

[TripelJ]

Problem with 2023.12.1 version

Thanks. I'll hold on from updating / testing more for now then.

[TripelJ]

I somehow seem to have esp home update without my knowledge, and breaking a lot of HA. Any idea how do disable. It seems like every other update breaks a lot of HA.

Edit: Found it. The last update broke a lot of my HA. Its will take days to fix all the dashboards and automation that is broken......

Don't! It's a waste of your time. Locate the backup and roll back to 11.6 and update all. And things should work again. It did for me and others here. Just gathering the overview on where to change it all gave me a headache. :)

[OGB1952]

I'm in the process of rolling back to 11.6. I did try the 12.1 update but it just lost all the entities that had been impacted by the changes in 12.0 . . . Hopefully I'll be able to recover them, if not it looks like I have to start again with the devices that are impacted by this. Still, I've learned a valuable lesson - NEVER be too quick off the mark to install updates. Wait a week or two and let others break things first!

[TillFleisch]

I too, can confirm that 2023.12 (currently running 12.3) broke entity naming in HA for all components with (german) umlauts in them. After looking at some, they now seem to have the suffix _2 added to their previous name. I.e. light.kuche_insel is now a new entity named light.kuche_insel_2.

[TripelJ]

I too, can confirm that 2023.12 (currently running 12.3) broke entity naming in HA for all components with (german) umlauts in them. After looking at some, they now seem to have the suffix _2 added to their previous name. I.e. light.kuche_insel is now a new entity named light.kuche_insel_2.

If I understand you correctly then even on 12.3 you still get the _2 entity. ?
I had no special characters in mine and they still change just to _2 so if its a thing that is unavoidable when updating I will wait til I have more time to go and fix everthing afterwards.

[TillFleisch]

I've installed the updates pretty quickly after one another, therefore I don't know the exact intermediate states. On 12.3 I had renamed entities as you've described. I am now in the process of rolling back to 2023.11.x and I have also seen at least one entity with suffix _3.

I have not restarted home assistant with the changed entity names, maybe that would have changed things for the better.

[OGB1952]

Same here, ESPHome seems to have automatically updated to 12.2 this morning and broken dozens of entity names again. I reverted back to 11.6 and updated all and got it back, but still have 91 orphaned entities (all those that were given new names that no longer function). What I'm not sure about is why ESPHome automatically updated. In the past I've always had to manually update, but this morning I didn't choose to update to 12.2, it just happened - first I knew of it was when I had lots of "entity not available" messages again (as happened yesterday with 12.0).

[TripelJ]

Same here, ESPHome seems to have automatically updated to 12.2 this morning and broken dozens of entity names again. I reverted back to 11.6 and updated all and got it back, but still have 91 orphaned entities (all those that were given new names that no longer function). What I'm not sure about is why ESPHome automatically updated. In the past I've always had to manually update, but this morning I didn't choose to update to 12.2, it just happened - first I knew of it was when I had lots of "entity not available" messages again (as happened yesterday with 12.0).

Go into your add-on setting for ESPHome. Possibly the check mark for auto update is ticked. Mine was also always ticked. Not anymore.

[Jpsy]

Same here, affecting all entities of my 23 MCUs. Reverting to 11.6 fixes it. Have to recompile and OTA them all manually, as I cannot do it from the Updates list (as it's "down dates").

[JDIacobbo]

I was under the impression that this PR fixed the issue but updating again today to 2023.12.3 again broke my entities. I'm willing to provide any information that may be helpful in troubleshooting.

[sheaur]

Reverting to 11.6, problem with special characters (á, é, í, ó, ú, ñ, etc.)

[OGB1952]

I was under the impression that this PR fixed the issue but updating again today to 2023.12.3 again broke my entities. I'm willing to provide any information that may be helpful in troubleshooting.

I was under the same impression, so updated to 12.3 only for HA to break again as it did yesterday. I think the problem is with entity unique_id's being changed. It looks as if the changes introduced to deal with some special characters in entity names are causing entities to be duplicated because the changes create new unique_id's. If I've understood things correctly (and there is every chance I have not!) then HA uses the unique_id to keep track of entities. When the unique_id changes HA assumes this is a new entity, one not used by any integrations.

[OGB1952]

From what I have been able to understand, this was a planned change and it was known that it would seriously break a lot of HA installations. It wasn't clear to me when reading the release notes just how much havoc this would cause. I cannot find any easy way to fix it, seems the options are to just stay on 11.6 or to manually edit dozens of entries and lose all the data associated with every broken entity. It looks as if the intention was to tidy up entity naming with regard to special characters, which seems like a sound idea.
What I can't quite understand is why every affected entity unique_id had to be changed as well - that seems to be key as to why this causes so much havoc. If the unique_id had been retained and just the entity name itself had been tidied up I don't think this would have caused the massive problems it has.

[TripelJ]

From what I have been able to understand, this was a planned change and it was known that it would seriously break a lot of HA installations. It wasn't clear to me when reading the release notes just how much havoc this would cause.

I cannot find any easy way to fix it, seems the options are to just stay on 11.6 or to manually edit dozens of entries and lose all the data associated with every broken entity. It looks as if the intention was to tidy up entity naming with regard to special characters, which seems like a sound idea.

What I can't quite understand is why every affected entity unique_id had to be changed as well - that seems to be key as to why this causes so much havoc. If the unique_id had been retained and just the entity name itself had been tidied up I don't think this would have caused the massive problems it has.

I agree that clean up is a good thing. As mentioned a few times this happens despite of special characters. If this would have only happened in such a case I would have been fine. As I have done a lot to make sure my entities names are correct and no unsupported characters. So it shouldn't be touching mine but still does for some reason. I am not a regex expert but seems like it maybe catches more than it should.

[OGB1952]

Given that it looks like I need to rebuild HA pretty much in its entirety, does anyone have any advice as to how to do this with the least damage to the underlying database, please? I have 91 entities that are now orphaned and inaccessible, although they show on the entity list. I could re-write everything and create a new HA installation, but that's several days work and would mean losing lots of data, right at the end of the year when I would really like to pull some statistics off for things like energy usage etc.

[Jpsy]

This won't be fixed. It is an intentional breaking change. All the details are discussed here: https://github.com/home-assistant/core/issues/106159

The good news is that it is a really simple fix. DO NOT rename all of your entities! Instead:

• Compile your ESPHome node with 2023.12.1 or higher, so it shows double entities. Do NOT use 2023.12.0 as there is another code change from .0 to .1 that will break your entities again.
• Remove the ESPHome node (Settings -> Devices and services -> ESPHome -> 3-dot-menu of node -> Delete).
• Wait for HA autodiscovery to find the node again. It will be displayed in Devices and services. Or restart HA to force immediate discovery.
• Add the node again (click "Configure" under the discovered node).

The history of the entities will NOT GET LOST by this process. Make sure you have your encryption key at hand, in case you will be asked for it when re-adding the node. Most often you won't.

Hot tip: You may want to test the whole process with only one less important ESPhome node before you go all in with all your nodes.

[ssieb]

To be clear, any duplicates that were caused by having a - in the name will be fixed, but not the ones caused by having accented or other special characters. If you already have duplicated entities, then remove the device from the integrations list and add it again. Everything should be retained. (At least the ones due to -, I'm not sure about the others.)

[TripelJ]

To be clear, any duplicates that were caused by having a - in the name will be fixed, but not the ones caused by having accented or other special characters. If you already have duplicated entities, then remove the device from the integrations list and add it again. Everything should be retained. (At least the ones due to -, I'm not sure about the others.)

I am just trying to understand here it's good that there is a work around and it's was a breaking change to fix special characters etc. but is '_' underscore also a special character. Don't think so. Entities always been added with underscore so my e.g was light.kitchen_ceiling_light Became light.kitchen_ceiling_light_2

Where is the need to fix anything here. ?

[TillFleisch]

(At least the ones due to -, I'm not sure about the others.)

I have removed, updated and added back some nodes and everything seems to be unchanged, including entities with umlauts in them (ä,ö,ü,ß).

A couple of words on which Actions to take (for affected users) in the release notes would have gone a long way.

I think we can consider this issue (and the one in the HA repo) closed.

[Jpsy]

I have removed, updated and added back some nodes and everything seems to be unchanged, including entities with umlauts in them (ä,ö,ü,ß).

"unchanged" as in "working again" or as in "still not fixed"?

[TillFleisch]

"unchanged" as in everything is working as it used to on 2023.11 and history within home assistant is persistent. Pardon my wording.

*Note: I did have to enable the 'disabled by default' sensors I did end up using. This is to be expected after removing the node from HA.

[Jpsy]

light.kitchen_ceiling_light Became light.kitchen_ceiling_light_2

Where is the need to fix anything here. ?

The breaking change is not about entity_id but about unique_id. The generation of the unique_ids has changed. You can find the unique_ids of any ESPhome node and its entities by going to Settings -> Devices and services -> ESPHome -> 3-dot-menu of node -> Download diagnistics and checking the resulting file.

In my nodes some of the unique_ids are cryptic, like e8db84dc3cad-wifisignal, containing a condensed form of the MAC address of the node (e8.db.84.dc.3c.ad) and the platform of the sensor (wifi_signal). The corresponding entity_id is "sensor.rauchmelder_garage_ug_ruckwand_wifi_signal".

Others are clearly generated from the node name and the entity name, like smoke-ug-garage-hintensensorrauchmelder_garage_ug_r_ckwand__batterie for a node called "smoke-ug-garage-hinten" having a sensor with name "Rauchmelder Garage UG Rückwand, Batterie" and entity_id "sensor.rauchmelder_garage_ug_ruckwand_batterie".

Nodes can have both forms of unique_ids for their entities. They seem to be mixed and matched with now recognizable rule.

[TripelJ]

light.kitchen_ceiling_light Became light.kitchen_ceiling_light_2

Where is the need to fix anything here. ?

The breaking change is not about entity_id but about unique_id.

The generation of the unique_ids has changed.

You can find the unique_ids of any ESPhome node and its entities by going to Settings -> Devices and services -> ESPHome -> 3-dot-menu of node -> Download diagnistics and checking the resulting file.

In my nodes some of the unique_ids are cryptic, like e8db84dc3cad-wifisignal, containing a condensed form of the MAC address of the node (e8.db.84.dc.3c.ad) and the platform of the sensor (wifi_signal). The corresponding entity_id is "sensor.rauchmelder_garage_ug_ruckwand_wifi_signal".

Others are clearly generated from the node name and the entity name, like smoke-ug-garage-hintensensorrauchmelder_garage_ug_r_ckwand__batterie for a node called "smoke-ug-garage-hinten" having a sensor with name "Rauchmelder Garage UG Rückwand, Batterie" and entity_id "sensor.rauchmelder_garage_ug_ruckwand_batterie".

Nodes can have both forms of unique_ids for their entities. They seem to be mixed and matched with now recognizable rule.

Thanks for elaborating. Will try to re-add my devices.

[OGB1952]

After a few hours work I now have a working HA system again. I followed the advice to delete the config for each affected ESPHome device, then restart HA so that it discovers the deleted devices again. This worked well to get back the 91 affected entities into a semi-working state, but I was left with 91 duplicates and several automations that were still broken, as they were using defunct entities. Only fix was to manually edit each entity entry, after having gone through 182 entities to work out the 91 that were valid versus the 91 that weren't. Took a fair time, but I'm pretty sure I've caught all the rogue duplicates now. For now I've just hidden the non-functioning duplicate entities, I don't want to delete them until I'm 100% sure that I've not made an error somewhere. FWIW, every affected entity for me had a degree symbol, °, which was replaced with an underscore, _ character. Also, every affected entity had "esp32c3" appended to the front of the entity name (immediately after "sensor." or "switch."). Not at all sure why that happened, it doesn't seem to be documented AFAICS.

[Jpsy]

I followed the advice to delete the config for each affected ESPHome device, then restart HA so that it discovers the deleted devices again. This worked well to get back the 91 affected entities into a semi-working state, but I was left with 91 duplicates and several automations that were still broken, as they were using defunct entities.

I did this for 23 Nodes with about 500 entities and it worked for all. But I know that there was an additional change to the unique_id sanitization code in 2023.12.1. So for anybody on 2023.12.0 I recommend to restore a backup to 2023.11, recompile once, and then directly update to 2023.12.3 or higher.

[TripelJ]

I followed the advice to delete the config for each affected ESPHome device, then restart HA so that it discovers the deleted devices again. This worked well to get back the 91 affected entities into a semi-working state, but I was left with 91 duplicates and several automations that were still broken, as they were using defunct entities.

I did this for 23 Nodes with about 500 entities and it worked for all. But I know that there was an additional change to the unique_id sanitization code in 2023.12.1. So for anybody on 2023.12.0 I recommend to restore a backup to 2023.11, recompile once, and then directly update to 2023.12.3 or higher.

Do you remove them from both the integration and ESPHome add-on, or just the integration?

[Jpsy]

Read my instructions above: https://github.com/esphome/issues/issues/5254#issuecomment-1868254856

[OGB1952]

As above, the delete config and restore probably fixes cases where the only change from 12.x was the added underscore character but in my case all the affected entities had also been renamed with an added "esp32c3" at the front (after the "switch." or "sensor." at the start). Fixing this needed a manual approach, going through every affected automation that was showing an "entity not available" message and changing from the now non-functioning entity to the new one with the added "esp32c3" prefix.
Not hard to do, but a bit time consuming if you have a lot of messed up entities (I had 91). AFAICS HA is now working normally (I.m now on 12.3), has been for a few hours now and nothing has yet thrown a wobbly, so I'm hopeful I've correctly identified and edited every corrupted entity.

[pumo2019]

This won't be fixed. It is an intentional breaking change. All the details are discussed here: home-assistant/core#106159

The good news is that it is a really simple fix. DO NOT rename all of your entities! Instead:

* Compile your ESPHome node with 2023.12.1 or higher, so it shows double entities. Do NOT use 2023.12.0 as there is another code change from .0 to .1 that will break your entities again.

* Remove the ESPHome node (Settings -> Devices and services -> ESPHome -> 3-dot-menu of node -> Delete).

* Wait for HA autodiscovery to find the node again. It will be displayed in Devices and services. Or restart HA to force immediate discovery.

* Add the node again (click "Configure" under the discovered node).

The history of the entities will NOT GET LOST by this process. Make sure you have your encryption key at hand, in case you will be asked for it when re-adding the node. Most often you won't.

Hot tip: You may want to test the whole process with only one less important ESPhome node before you go all in with all your nodes.

I did this, still _2 so should I rename those back?

• upgraded esphome 11.6 -> 12.5
• removed all (Settings -> Devices and services -> ESPHome -> 3-dot-menu of node -> Delete)
• added all again from discovery
• upgraded all firmwares with ota. I have one what counts daily electricity consumption, what is problematic. at least it didnt reset all older consumptions to zero from energy page, but I have now sensor.talon_paivan_kokonaiskulutus and sensor.talon_paivan_kokonaiskulutus_2

I added this new one (_2) beside of old to energy dashboard my comption counting in esphome: name is in finnish Daily comsumption of house.

Total daily energy

• platform: total_daily_energy name: "Talon päivän kokonaiskulutus" power_id: esp_watts_kok unit_of_measurement: kWh accuracy_decimals: 3
  icon: mdi:counter

edit: nevermind I will return to 11.6 and downgrade firmwares...

[Jpsy]

You did not do it in the right order. You must FIRST recompile with the 12.x version (your "upgrade firmwares"), THEN remove and re-add.

[pumo2019]

ok thanks. I will test it edit: and it worked, thanks @Jpsy

[pumo2019]

one problem even this works now. I lost all electricity prices before yesterdays upgrade. I use "Use an entity with current price" this daily consumption.
it counts prices after yesterdays upgrade.

[dflvunoooooo]

As above, the delete config and restore probably fixes cases where the only change from 12.x was the added underscore character but in my case all the affected entities had also been renamed with an added "esp32c3" at the front (after the "switch." or "sensor." at the start). Fixing this needed a manual approach, going through every affected automation that was showing an "entity not available" message and changing from the now non-functioning entity to the new one with the added "esp32c3" prefix. Not hard to do, but a bit time consuming if you have a lot of messed up entities (I had 91). AFAICS HA is now working normally (I.m now on 12.3), has been for a few hours now and nothing has yet thrown a wobbly, so I'm hopeful I've correctly identified and edited every corrupted entity.

I have the same problem. I followed @Jpsy instructions, but this did not help.

My esphome code looke like this:

substitutions:
  device_name: "esp32-kueche-arbeitsplattenbel"
  device_description: "ESP32 Arbeitsplattenbeleuchtung"
  ort: "Küche"
  friendly_name: "${ort} ${device_description}"
…
esphome:
  name: ${device_name}
  friendly_name: ${friendly_name}
…
light:
  - platform: cwww
    name: "${ort} Arbeitsplattenlicht ${device_description}"
…

The created light entity is called light.esp32_kueche_arbeitsplattenbel_k_che_arbeitsplattenlicht_esp32_arbeitsplattenbeleuchtung. Before I updated it was light.kuche_arbeitsplattenlicht_esp32_arbeitsplattenbeleuchtung. I don't know why the device description is added at the beginning and the ü from the ort is not transformed correctly.

Edit: I am on version 12.5. Edit2: After the steps in the instructions, I now have the esphome: name in front of every single entity name, even those with normal names! Ah!