search

sockless-coding / panasonic_cc

Panasonic Comfort Cloud - Home Assistant Component
MIT License
137 stars 35 forks source link

Frequency of cloud API calls - should we try to reduce the load on the Comfort Cloud servers? #152

Open jandersonhill opened 8 months ago

jandersonhill commented 8 months ago

It's dawned on some of us that Panasonic might be baulking at the sheer number of calls being made to their cloud API by Homebridge (my own experience in the IoT space makes me acutely aware that an increased API calls increases the cost of running the platform).

As things stand, anyone who uses their Comfort Cloud app probably makes a few dozen API calls a day - whereas anyone using Homebridge is likely making thousands of API calls a day (mostly Homekit just getting updates on whether the AC is on or not!). Multiply this by the number of customers using Homebridge, the Homebridge numbers likely stand out like a sore thumb and could be what is leading to Panasonic attempting to block such projects.

We suggest reducing the frequency of status checks to bring us more in line with someone using their app by checking the current status every ten minutes (rather than every ten seconds, which I suspect Panasonic never planned for). It's true that anyone using the Comfort Cloud app alongside Homebridge will see a delay before Homebridge updates, but that has to be better than Panasonic blocking all Homebridge activity altogether.

We're proposing to change the default frequency of calls to the Comfort Cloud API for the Homebridge Panasonic AC plugin to ten minutes, but allow people to tweak it (probably between 1 minute and 60 minutes). We'll then look at the feedback.

Our hope is that a community-wide approach to solving this issue will be sufficient to save such projects from being blocked outright, but we wanted to get the thoughts of others involved as it will take all of us to change how we think about this to reduce the load on the Comfort Cloud servers. What does everyone think?

mkz212 commented 8 months ago

We want to ask you what frequency of air conditioning status refresh you have set?

There is no official API to support plugins. For the API that is, it is designed to support the official Panasonic application. How many queries does such an application generate per day? Let's assume that someone opens it even 10 times a day is 10. Now let's assume that someone has 5 air conditioning so it is 50.

And now how much plug, depending on the frequency? Interval - API calls

No wonder Panasonic can lock API for plugins.

Our suggestion is that users have a choice of frequency from 1 minutes to 60 minutes (or even form 5 minutes), and by default (i.e. recommended value) would be 10 minutes.

And second suggestion, in case of failure login / read values - add delay - if error is 429 (panasonic 24 h ban) delay 8h, if error is 401 (wrong username / password) delay 8h, in other cases 1h delay.

More frequent refresh would result in too much daily number of requests to the Panasonic server, which could result in an account lock for 24 hours, or even a complete API lock.

This is a question for everyone and in particular for the author of the plugin: @sockless-coding . What do you think about it?

P.S.: The proposed solutions have already been introduced (or are currently introducing) in the Homebridge plugin. But if it would make sense and Panasonic would stop blocking / changing the API, the change must happen everywhere.

codyc1515 commented 8 months ago

I would rather not poll it at all. An option to disable polling would be welcomed. I never use the remote or the app, so there's no way for the data to get out of sync. Moreover, my heat pump doesn't have the indoors sensor and I don't use the outdoors sensor so that data is of no use to me either. I might work on a PR to add this.

mkz212 commented 8 months ago

I would rather not poll it at all. An option to disable polling would be welcomed. I never use the remote or the app, so there's no way for the data to get out of sync. Moreover, my heat pump doesn't have the indoors sensor and I don't use the outdoors sensor so that data is of no use to me either. I might work on a PR to add this.

Option to completely disabling the poll is also a good idea! But limiting the frequency of polling to a minimum of 5 minutes (co it could be disabled, 5 minutes, 10 minutes etc.) also gives a lot.

so it should be: disabled, 5 minutes, 10 minutes, etc.

codyc1515 commented 8 months ago

It looks like the integration should be using Coordinated, single API poll for data for all entities but I cannot see any references to code that looks similar. This probably explains why there are some warnings in the log about things taking a while to complete (because it's not fully asynchronous).

So, I'm not sure how often it's polling (maybe it's every minute). I would think we could at least start by reducing that to every 15 minutes.

jandersonhill commented 8 months ago

I think someone told me it was currently polling more frequently than once a minute (something like every ten seconds), but either way, I'd love to get these down to a level acceptable to Panasonic. Certainly, reducing the frequency made the polling problem go away for some of those experiencing issues.

mkz212 commented 8 months ago

In addition, it is worth adding the option of excluding individual devices.

Someone may have air conditioning from different houses on 1 Comfort Cloud account, and want to add only a part to one house.

codyc1515 commented 2 months ago

I have a single device and got banned by Cloudflare for hitting the API too often. Now the official app isn’t even working. How often is this polling? Can we disable polling?

IMG_0111 IMG_0112

sockless-coding commented 2 months ago

@codyc1515 the time delta between updates is 60s. The integrations used two throttles, one for the normal status and another one for energy.

mkz212 commented 2 months ago

@sockless-coding

I suggest 10 minutes when device is turned on and 60 minutes when devices is turned off.

sockless-coding commented 2 months ago

I'm not sure the amount of polling is the sole cause of the suspicious activity error. I have 2 HA instances running and 2 CC apps running. Do you have different credentials for HA and the APP? If not make sure you do, and is the app set to a different language then English? I think it's more likely something like multiple request from the same origin with different fingerprints causing the suspicious activity

mkz212 commented 2 months ago

I'm not sure the amount of polling is the sole cause of the suspicious activity error. I have 2 HA instances running and 2 CC apps running. Do you have different credentials for HA and the APP? If not make sure you do, and is the app set to a different language then English? I think it's more likely something like multiple request from the same origin with different fingerprints causing the suspicious activity

No, I don't mean account blocking. I mean Panasonic is constantly changing the api. There is a high probability that this is the cause of a high frequency of request to the server.

codyc1515 commented 2 months ago

I'm not sure the amount of polling is the sole cause of the suspicious activity error. I have 2 HA instances running and 2 CC apps running. Do you have different credentials for HA and the APP? If not make sure you do, and is the app set to a different language then English? I think it's more likely something like multiple request from the same origin with different fingerprints causing the suspicious activity

I found the cause on my side which was an automation that ran every 5 seconds which was updating the mode of Climate entities. Strangely, HA does not trust the state and so sends an update every time.

andrioid commented 2 months ago

I have 2 heat pumps and am also getting the 429 cloudflare error in HA. I'd much rather have it be slow to update, than being locked out. It seems Panasonic is getting more aggressive in blocking this.

No automations or anything that would bug their API aside from the polling here.

I can recommend having a separate account for HA and the Cloud Comfort app - that way you're not completely locked out when this happens.

mkz212 commented 2 months ago

Maybe let's start with the basic question - how often do you need to have the state of the device updated?

I think it's enough:

I think refreshing every few seconds is a strong exaggeration.

See how frequency affects the number of requests to the server (Interval - API calls per day):

And that's just for one device! Considering that the plugins use the API designed for mobile apps (iOS / Android) - I guess no one refreshes the application 8640 times a day? Or 1440? I believe that refreshing every 10 minutes when device is on and 60 minutes when device is off is the absolute maximum (not more often). In this case it will be about 50 request to the server per day.

DHT-Xavier commented 2 months ago

Failed to get any data from panasonic server with the following error

This error originated from a custom integration.

Logger: custom_components.panasonic_cc.panasonic
Source: custom_components/panasonic_cc/panasonic.py:86
integration: Panasonic Comfort Cloud (documentation, issues)
First occurred: 00:08:24 (126 occurrences)
Last logged: 03:42:39

Error updating device 4282ab987d1c583fc5773280357e0bc74a448412215600a70c9442004bb5e906
Error updating device febb623875e16bd08cfeeded58bbe79cf58c3cd10f01857d24460c71017ff175
Traceback (most recent call last):
  File "/config/custom_components/panasonic_cc/panasonic.py", line 86, in do_update
    data = await self._api.get_device(self.id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/panasonic_cc/pcomfortcloud/apiclient.py", line 123, in get_device
    json_response = await self.execute_get(self._get_device_status_url(device_guid), "get_device", 200)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/panasonic_cc/pcomfortcloud/panasonicsession.py", line 131, in execute_get
    check_response(response, function_description, expected_status_code)
  File "/config/custom_components/panasonic_cc/pcomfortcloud/panasonicsession.py", line 25, in check_response
    raise exceptions.ResponseError(
custom_components.panasonic_cc.pcomfortcloud.exceptions.ResponseError: (get_device: Expected status code 200, received: 500: <bound method ClientResponse.text of <ClientResponse(https://accsmart.panasonic.com/deviceStatus/CW-HU70AA%xxxxxxxxx) [500 Internal Server Error]>
<CIMultiDictProxy('Content-Type': 'application/json', 'Content-Length': '53', 'Connection': 'keep-alive', 'Date': 'Wed, 03 Jul 2024 16:08:24 GMT', 'x-amzn-RequestId': 'dab7ad76-8aa4-4721-af7e-491a8fd0460b', 'x-amz-apigw-id': 'aV-ZHEHEtjMEibw=', 'X-Amzn-Trace-Id': 'Root=1-6685776d-1a38ac8617345193450f8252;Parent=55b83b6d4de88507;Sampled=0;lineage=4d740725:0|b407fd35:0', 'X-Cache': 'Error from cloudfront', 'Via': '1.1 acfb1cd944f7bc93838f5d68a454df62.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'HKG62-C2', 'Alt-Svc': 'h3=":443"; ma=86400', 'X-Amz-Cf-Id': 'qujUXF53V1VpBDABN2tRMhbbReJZwB_y4KyXonj1hJTHOwotssJ2fw==', 'X-XSS-Protection': '1; report=1;mode=block', 'X-Frame-Options': 'DENY', 'Content-Security-Policy': "script-src 'self' 'unsafe-inline'", 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains')>
>

It work again if I changed the route to accsmart.panasonic.com using another WAN (IP blocked due to calling frequency ?)

sockless-coding commented 2 months ago

@xavier-wong-hk Panasonic is currently having some server issues. It seems to be affecting getting and setting device information, and it's not limited to the integration, the App has the same issues.

DHT-Xavier commented 2 months ago

@sockless-coding I can still control the climate with the App using cellular but not wifi while HA is not working. Thats why I try to route the traffic to another WAN

Update: I just restored the route and it is working now

sockless-coding commented 1 month ago

v1.1.0 supports setting the update interval for both device and energy. Buttons for manual updates are also available.

mkz212 commented 1 month ago

v1.1.0 supports setting the update interval for both device and energy. Buttons for manual updates are also available.

It's good to hear! I believe that if traffic on Panasonic servers will be limited to absolute necessary minimum, they will stop fighting like this and changing the API. 👍