search

bramstroker / homeassistant-powercalc

Custom component to calculate estimated power consumption of lights and other appliances
MIT License
956 stars 252 forks source link

Failed to download after updating HomeAssistant #2526

Open brianegge opened 5 days ago

brianegge commented 5 days ago

System Health details

System Information

version core-2024.9.3
installation_type Home Assistant OS
dev false
hassio true
docker true
user root
virtualenv false
python_version 3.12.4
os_name Linux
os_version 6.6.46-haos
arch x86_64
timezone America/New_York
config_dir /config
Home Assistant Community Store GitHub API | ok -- | -- GitHub Content | ok GitHub Web | ok HACS Data | ok GitHub API Calls Remaining | 5000 Installed Version | 2.0.1 Stage | running Available Repositories | 1425 Downloaded Repositories | 29
Home Assistant Cloud logged_in | true -- | -- subscription_expiration | February 8, 2025 at 7:00 PM relayer_connected | true relayer_region | us-east-1 remote_enabled | true remote_connected | true alexa_enabled | true google_enabled | true remote_server | us-east-1-1.ui.nabu.casa certificate_status | ready instance_id | 831317b31a3e4274bd53f16d096aa4ab can_reach_cert_server | ok can_reach_cloud_auth | ok can_reach_cloud | ok
Home Assistant Supervisor host_os | Home Assistant OS 13.1 -- | -- update_channel | stable supervisor_version | supervisor-2024.09.1 agent_version | 1.6.0 docker_version | 26.1.4 disk_total | 468.7 GB disk_used | 17.0 GB healthy | true supported | true host_connectivity | true supervisor_connectivity | true ntp_synchronized | true virtualization | board | generic-x86-64 supervisor_api | ok version_api | ok installed_addons | Matter Server (6.5.1), Mosquitto broker (6.4.1), Spotify Connect (0.13.0), Govee to MQTT Bridge (2024.07.13-82ddc6e9), File editor (5.8.0), Studio Code Server (5.15.0)
Dashboards dashboards | 4 -- | -- resources | 17 views | 20 mode | storage
Recorder oldest_recorder_run | September 15, 2024 at 12:27 PM -- | -- current_recorder_run | September 24, 2024 at 10:58 AM estimated_db_size | 923.92 MiB database_engine | sqlite database_version | 3.45.3
Sonoff version | 3.8.1 (ffa7e22) -- | -- cloud_online | 1 / 1 local_online | 1 / 1
Spotify api_endpoint_reachable | ok -- | --

Checklist

Describe the issue

Download is failing after upgrading HAOS. I'm not sure why it's downloading as I didn't add any new devices, and it's not falling back to a cached value. I tried restarting and I get the same problem.

Reproduction steps

  1. Upgrade to core-2024.9.3
  2. restart

Debug logs

This error originated from a custom integration.

Logger: custom_components.powercalc.power_profile.loader.remote
Source: custom_components/powercalc/power_profile/loader/remote.py:258
integration: Powercalc (documentation, issues)
First occurred: 10:58:21 AM (2 occurrences)
Last logged: 10:58:24 AM

Failed to download, retrying... (Attempt 2 of 3)
Failed to download, retrying... (Attempt 3 of 3)

Diagnostics dump or YAML config

No response

bramstroker commented 5 days ago

Powercalc downloads JSON file containing full library meta data each startup, so it knows which profiles there are. If there are new ones or existing once have an update. Apparently this fails for you. Could you try accessing yourself? https://api.powercalc.nl/library. I don't experience any issues with it.

It falls back to last save library.json on your local instance after 3 retries, so you should not experience any issues except this error in the logs.

brianegge commented 5 days ago

Sorry, I had trouble figuring out how to get the detailed logs from HAOS. Got it now and see why I'm the only one with the issue!

2024-09-24 15:22:38.098 ERROR (MainThread) [homeassistant.setup] Error during setup of component powercalc
Traceback (most recent call last):
  File "/config/custom_components/powercalc/power_profile/loader/remote.py", line 250, in download_with_retry
    return await callback()
           ^^^^^^^^^^^^^^^^
  File "/config/custom_components/powercalc/power_profile/loader/remote.py", line 282, in download_profile
    raise ProfileDownloadError(f"Failed to download profile: {manufacturer}/{model}")
custom_components.powercalc.power_profile.error.ProfileDownloadError: Failed to download profile: Epson/ET-3760 Series

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/config/custom_components/powercalc/power_profile/loader/remote.py", line 170, in _download_profile_with_retry
    await self.download_with_retry(callback)
  File "/config/custom_components/powercalc/power_profile/loader/remote.py", line 255, in download_with_retry
    raise ProfileDownloadError(f"Failed to download even after {max_retries} retries, falling back to local copy") from e
custom_components.powercalc.power_profile.error.ProfileDownloadError: Failed to download even after 3 retries, falling back to local copy

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/setup.py", line 416, in _async_setup_component
    result = await task
             ^^^^^^^^^^
  File "/config/custom_components/powercalc/__init__.py", line 239, in async_setup
    await discovery_manager.start_discovery()
  File "/config/custom_components/powercalc/discovery.py", line 77, in start_discovery
    power_profile = await self.get_power_profile(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/powercalc/discovery.py", line 101, in get_power_profile
    self.power_profiles[entity_id] = await get_power_profile(
                                     ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/powercalc/power_profile/factory.py", line 42, in get_power_profile
    profile = await library.get_profile(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/powercalc/power_profile/library.py", line 103, in get_profile
    profile = await self.create_power_profile(model_info, custom_directory)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/powercalc/power_profile/library.py", line 129, in create_power_profile
    json_data, directory = await self._load_model_data(manufacturer, model, custom_directory)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/powercalc/power_profile/library.py", line 162, in _load_model_data
    result = await loader.load_model(manufacturer, model)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/powercalc/power_profile/loader/composite.py", line 39, in load_model
    result = await loader.load_model(manufacturer, model)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/powercalc/power_profile/loader/remote.py", line 137, in load_model
    await self._download_profile_with_retry(manufacturer, model, storage_path, model_path)
  File "/config/custom_components/powercalc/power_profile/loader/remote.py", line 174, in _download_profile_with_retry
    await self.hass.async_add_executor_job(shutil.rmtree, storage_path)
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/shutil.py", line 759, in rmtree
    _rmtree_safe_fd(stack, onexc)
  File "/usr/local/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
    onexc(func, path, err)
  File "/usr/local/lib/python3.12/shutil.py", line 669, in _rmtree_safe_fd
    orig_st = os.lstat(name, dir_fd=dirfd)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/config/.storage/powercalc_profiles/Epson/ET-3760 Series'

It's found my Epson printer, and my integration must have broken the download.

bramstroker commented 5 days ago

Could you also enable debug logging? https://docs.powercalc.nl/troubleshooting/debug-logging/ That should log some more and hopefully provide some more insight in the exact reason why the Epson profile cannot be downloaded.

bramstroker commented 5 days ago

Might be some casing issue somewhere. As this API call is working:

https://api.powercalc.nl/download/epson/ET-3760%20Series

But this doesn't:

https://api.powercalc.nl/download/Epson/ET-3760%20Series

Looking at your error logs it's trying the second one. custom_components.powercalc.power_profile.error.ProfileDownloadError: Failed to download profile: Epson/ET-3760 Series

Need to have a further look into this upcoming days when I have some more time.

bramstroker commented 5 days ago

I have made one small change. https://github.com/bramstroker/homeassistant-powercalc/commit/dcd2e315b06422e51977ca8fbb50f54edbd4585c

Could you try again by rebooting HA?

brianegge commented 4 days ago

Thank you! Yes, it’s loading fine now. I first I could write a model validator as part of the PR actions. I’m getting the duplicate device, which others have reported. I don’t know the fix. My printer has five entities: four ink levels and the printer status.  image

This is the sensor of interest:And here is an entity to ignore:Seems like I need to match on device_class==enum

image

brianegge commented 4 days ago

Also, I see the local model downloader does a lower() on the manufacturer. I was thinking the remote should do the same.

+++ b/custom_components/powercalc/power_profile/loader/remote.py
@@ -129,12 +129,12 @@ class RemoteLoader(Loader):
         retry_count: int = 0,
     ) -> tuple[dict, str] | None:
         """Load a model, downloading it if necessary, with retry logic."""
-        model_info = self._get_model_info(manufacturer, model)
-        storage_path = self.get_storage_path(manufacturer, model)
+        model_info = self._get_model_info(manufacturer.lower(), model)
+        storage_path = self.get_storage_path(manufacturer.lower(), model)
         model_path = os.path.join(storage_path, "model.json")

         if await self._needs_update(model_info, model_path, force_update):
-            await self._download_profile_with_retry(manufacturer, model, storage_path, model_path)
+            await self._download_profile_with_retry(manufacturer.lower(), model, storage_path, model_path)

         try:
             json_data = await self._load_model_json(model_path)
bramstroker commented 4 days ago

Thank you! Yes, it’s loading fine now. I first I could write a model validator as part of the PR actions. I’m getting the duplicate device, which others have reported. I don’t know the fix. My printer has five entities: four ink levels and the printer status.  image This is the sensor of interest:And here is an entity to ignore:Seems like I need to match on device_class==enum image

I see, need to develop some mechanism to indicate which entity should be used for discovery. Needs to be done somewhere here: https://github.com/bramstroker/homeassistant-powercalc/blob/master/custom_components/powercalc/discovery.py#L147-L151. And some extension on model.json fields to configure this. Currently it will only look at the entity domain, for printer this is expected to be sensor.. So in this scenario that would be 5 entities and 5 discoveries. For other devices like switches and lights this was never an issue because it was only 1 entity per device.

bramstroker commented 4 days ago

Also, I see the local model downloader does a lower() on the manufacturer. I was thinking the remote should do the same.

+++ b/custom_components/powercalc/power_profile/loader/remote.py
@@ -129,12 +129,12 @@ class RemoteLoader(Loader):
         retry_count: int = 0,
     ) -> tuple[dict, str] | None:
         """Load a model, downloading it if necessary, with retry logic."""
-        model_info = self._get_model_info(manufacturer, model)
-        storage_path = self.get_storage_path(manufacturer, model)
+        model_info = self._get_model_info(manufacturer.lower(), model)
+        storage_path = self.get_storage_path(manufacturer.lower(), model)
         model_path = os.path.join(storage_path, "model.json")

         if await self._needs_update(model_info, model_path, force_update):
-            await self._download_profile_with_retry(manufacturer, model, storage_path, model_path)
+            await self._download_profile_with_retry(manufacturer.lower(), model, storage_path, model_path)

         try:
             json_data = await self._load_model_json(model_path)

Yeah, we can make this change as all the manufacturer directories in profile_library directory are lower cased. I initially made the decision to do that in the beginning of the project. So for consistency reasons we will keep that.