State does not persist through restarts

[tankdeer]

The state of the sensor does not persist if Home Assistant is restarted. I am replacing a old sensor of the same name, so there is old state history that pulls in instead. The previous (broken) sensor stopped at 797 GB. The current value is 136 GB. This is accurately updated during the next scheduled update, but if I restart HA between updates, it reverts back to the old value. You can see in the history graph here. The two spikes near the end show what happens if I restart

[thor0215]

I would need more details on how the sensor was originally setup. I did reuse a previous rest based sensor and I'm not seeing this issue. I did fully disable that rest based sensor yaml config before testing the addon, but my states are holding the correct value, even if the addon is stopped and not set to auto start. I would stop the addon, disable and then delete the old xfinity sensor you have. Once that is gone, reboot HA and then start the addon. Then see if you still have the same issue.

[tankdeer]

Ok, with that guidance I thought I had fixed it. I did not, however I was able to get a little more insight into what is happening.

The original sensor was an MQTT sensor. My first thought was there must be a retained topic with the old values - and there was. That's where the 797GB was coming from. However that was only masking the actual problem. I deleted that topic, and now as expected it does not show the old value. Worse though, the sensor itself is deleted.

It seems like this is what was happening before, the new sensor from your addon was getting deleted, but because I had a retained MQTT topic, it was getting recreated with the old info. Now with that gone, so is the sensor:

If I restart the add-on, the sensor gets recreated without issue. Same if I wait for the next refresh cycle. But until then, there's nothing

[thor0215]

Can you check on a few things?

1. Go to Developer Tools -> States and search for xfinity_usage and click on the entity. Then check if this is set in the attributes: "state_class: measurement" If it is then go to Step 2

2. Go to Developer Tools -> Statistics and search for xfinity_usage. Let me know if this exists or not. If it does exist go to step 3

3. In your HA configuration is Recorder enabled? Go to this link to see the HA documentation, https://www.home-assistant.io/integrations/recorder

[tankdeer]

Everything looks fine before rebooting:

Of course after rebooting (and clearing front-end cache since it likes to stick around otherwise) everything is gone

Recorder is enabled using mostly default settings using MariaDb and 21 days purge keep days. There are a handful of exclusions but nothing that applies to this sensor. (A couple of non-sensor domains and a handful of individual entities)

[tankdeer]

Updated to 0.0.7 and not sure I understand the expected functionality. The submitted PR claims to write the state to disk during shutdown. I checked the container path and appears that it does in fact do that:

root@989f231b-xfinity-usage:/config# ls -la
total 12
drwxr-xr-x 2 root root 4096 May  7 14:59 .
drwxr-xr-x 1 root root 4096 May  7 14:58 ..
-rw-r--r-- 1 root root 1158 May  7 14:59 .sensor-backup
root@989f231b-xfinity-usage:/config# cat .sensor-backup
{"attributes": {"policy_name": "1.2 Terabyte Data Plan", "start_date": "05/01/2024", "end_date": "05/31/2024", "home_usage": 149, "wifi_usage": 0, "total_usage": 149, "allowable_usage": 1229, "unit_of_measure": "GB", "display_usage": true, "devices": [{"id": "1C:93:7C:A4:B7:EF", "usage": 154, "policyName": "XI Connect More"}], "additional_blocks_used": 0, "additional_cost_per_block": 10, "additional_units_per_block": 50, "additional_block_size": 50, "additional_included": 0, "additional_used": 0, "additional_percent_used": 0.0, "additional_remaining": 0, "billable_overage": 0, "overage_charges": 0.0, "overage_used": 0, "current_credit_amount": 0, "max_credit_amount": 0, "maximum_overage_charge": 100, "policy": "limited", "courtesy_used": 1, "courtesy_remaining": 1, "courtesy_allowed": 1, "courtesy_months": ["07/2022"], "in_paid_overage": false, "remaining_usage": 1080, "friendly_name": "Xfinity Usage", "unit_of_measurement": "GB", "device_class": "data_size", "state_class": "measurement", "icon": "mdi:network", "internet_download_speeds_Mbps": 300, "internet_upload_speeds_Mbps": 20, "tier_of_service": "Connect More Internet"}, "state": 149}

However the data is not restored during startup as I would expect it to be. And the README seems to imply that is expected?

Can you please clarify the expected behavior? Shouldn't this data be restored from disk on startup? If not I don't understand its purpose.

Thanks again

[thor0215]

The sensor will be created/updated after reboot once the addon is started. The backup sensor file is to update the sensor upon addon startup before the addon tries to get an update from Xfinity. I've updated the README to explain that there is a known limitation that the sensor will be unavailable if you restart Home Assistant. This is caused by the way Home Assistant handles sensors which are not backed up by an entity, but instead come from an add-on or AppDaemon.

The addon sensor entity does not actually exist in the HA core.entities file like an integration entity would. Do you have the addon set to load on startup?

[thor0215]

One more thing I would try fixing the Recorder issue with the sensor.

[tankdeer]

Interesting. So you're saying the addon would need to be restarted in order for the state to be restored from backup? Wouldn't that just update it from xfinity anyways?

I do have the addon set to load on start, but my understanding is that since it's a separate docker container, it would actually stay running the whole time, and not restart the container. This is supported by the log file entries (The timestamps in there are a big help - thanks for that too). My understanding of how that works has always been if you have it disabled, the container is shut down with HA, and not restarted. If you have it enabled, then it just keeps it running (as opposed to restarting it)

I don't show any recorder issues - that only shows when the entity is missing, but corrects itself when it is recreated.

[thor0215]

If you do not set the addon to "Start on boot", then it will never run after a HA "Reboot System". You are correct that the addon does not restart if you do a "HA Restart System". I understand the issue better now and will push a blueprint to help with the creation of an Automation to restart the addon when HA restarts.

[tankdeer]

How do other addons handle this? I have not yet encountered a situation where entities created via an addon are lost during a restart. Perhaps this is why so many use a ____2mqtt pattern...

[thor0215]

Yes using MQTT would solve this issue, but I liked the simplicity of this in comparison to the work involved to setup MQTT. I also didn't want to create separate entities for each attribute and I couldn't figure out how to push a single entity with multiple attributes into MQTT.

[tankdeer]

I understand the desire to keep it simple.

For attributes on an MQTT entity you either provide a json_attributes or json_attributes_template key in the discovery payload with a dictionary of attributes. Or you can provide a json_attributes_topic to a different topic to subscribe to specifically for attributes. It takes a little fiddling at first but in the end isn't too difficult. I've seen the latter most often, probably because it makes json serialization a bit easier. eg:

/path/to/sensor/discovery/topic
{
"name": "Xfinity Usage",
"json_attributes_topic": "/path/to/attributes/topic",
"other": "keys"
}

/path/to/attributes/topic
{
"attribute_key1": "attr_value1",
"attribute_key2": "attr_value2",
"etc": "etc"
}

[thor0215]

Thanks for the tip, maybe someday I'll look into that.