Usage of Homematic IP API (according to customer support)

[maxigs]

Not a direct issue per-se but i thought it would be interesting to share:

I have been in contact with the customer support of Homematic in the last few days. At the start unrelated to the API (question about notifications in the Homematic app), but it quickly turned into a discussion about my API usage (via home-assistant), based on this library.

Originally, in the beginning of the year, when i switched to Homematic IP (Cloud), i asked the support about using it with non-approved third party integrations, and they seemed ok with it. No official support, but also no problem with it, and they have server side meassures in place to prevent it to cause propblems (the well known rate limit). So all good.

Now, with my contact this week, it turned into something less favourable. Aparently they had a pretty high increase in API traffic and have to start limiting things further. They did not tell me to stop using it, but strongly encuraged me to think about alternative ways, to avoid interruptions. It's a bit sad, because the homematic devices work incredibly well and easy to set up, especially thanks to this library. Not sure if there is any way to "fix" this, but i would hate to see it break completely.

Note : I currently use it as read only, to display values in home-assistant, not for any updates into the homematic system. But i do have a pretty high number of devices (~50).

[hahn-th]

Hey, thank you for your insights.

It think this is related to an error in this library in april (?) where all of us were banned by HmIP because of an error. I fixed the error. This was my first contact with this library. But there are more libraries around like ioBroker Integration, which provides basically the same functionallity.

I think i should integrate something like not try to reconnect every few milliseconds, when an error occured. Because this is when we get blocked by HmIP because of high API traffic.

If they warned you so strongly, maybe they are working in a fix which makes access by a third party lib impossible. Hopefully not! I mean i can unterstand them. There are a few libraries which are spaming their servers when errors occuring and creating load on their infrastructure. I would not like that either. On the other hand, for them it's a free opertunity to make their products more available.

What did they say about button presses?

[maxigs]

I think i should integrate something like not try to reconnect every few milliseconds

If this is the case it would indeed be very bad. Normally clients should have some logic to give the server a break in such cases. Typically what's called exponential backoff.

The last message i received did not really sound like they would kill it. Not sure if there is anything in the works to further/stronger limit the API usage. There are a couple of alternative implementations/systems, like the local CCU. I'm not sure about the differences though, their cloud service was working perfectly fine and i put all my heating schedules there. But i sill like to have the data in home-assistant for display and analysis.

What did they say about button presses?

This was no topic for me, as i have none. I just mentioned it, as the docs state button presses are causing a lot traffic which might acuse the ip blocking.

[hahn-th]

This was no topic for me, as i have none. I just mentioned it, as the docs state button presses are causing a lot traffic which might acuse the ip blocking.

Ah okay. I mixed something up. This is about pushing buttons in HomeAssistant multiple times. I think it is not my responsiblity to avoid that. But indeed its a point.

Normally clients should have some logic to give the server a break in such cases. Typically what's called exponential backoff.

I gonna have a look at that! If you are firm with python, feel free to create a push request 😄

[maxigs]

I gonna have a look at that! If you are firm with python, feel free to create a push request

Unfortunately my python is very basic, but i will have a look. Need to figure out how the connection handling works here in the first place.

[hahn-th]

It should be function api_call in class AsyncConnection. If you have discord: agonist#6159

[maxigs]

Just had a quick look over the project. Not 100% sure but i think the issue is here:

https://github.com/hahn-th/homematicip-rest-api/blob/master/homematicip/home.py#L779

This is doing a immediate re-connect when the websocket fails, no waiting time.

The AsyncConnection seems to be used further down, or directly by usig the CLI to make calls – but those are one-off exectutions. The Home class seems to be what is taking care of a persistent connection, used in upstream systems (like home-assistant).

This seems to be a good example of how exponential backoff in python can look like: https://keestalkstech.com/2021/03/python-utility-function-retry-with-exponential-backoff/

I will try to test it. Might take a few days though, until i manage to get it set up and running for testing though.

[josvo]

Hi

I just found this issue. Looks like some kind of (IP?) Rate limiting is already in place. When I do a basic query for my temperature sensors, I get an error from time to time and the query fails. The error did not lead me to the right direction

  File "/usr/local/lib/python3.9/site-packages/homematicip/home.py", line 349, in _get_devices
    for id_, raw in json_state["devices"].items():
TypeError: string indices must be integers

However I got a message in the Homeatic Android App, saying that rate limiting is in place and I have to wait 15 minutes before trying again.

A stub of my query (I have about 10 devices in total):

    ...
    global home
    home.get_current_state()
        for g in home.groups:
            if g.groupType == "META":
                for device in g.devices:
                    if isinstance(device, TemperatureHumiditySensorDisplay):
                        write_temperaturehumiditysensor(g.label, device)
                    elif isinstance(device, TemperatureHumiditySensorOutdoor):
                        write_temperaturehumiditysensoroutdoor(g.label, device)
            time.sleep(300)

This is reproducible at the moment.

[hahn-th]

How often do you loop thru your code? Looks like the json_state string contains wrong data.

[josvo]

How often do you loop thru your code?

Once every 5 min and then I check the values for every device (about 10 devices). I'm logging the current temperatures and humidity to a database.

Looks like the json_state string contains wrong data.

That's the problem. The json_state string is empty (empty string), I checked in debug mode. I get no answer from the API. I contacted the customer support and ask what exactly the limits are and how I could avoid being limited.

[h3tz]

Same here I just can request every 5-10min all devices to avoid touching limits. That's a mess.

[tschissler]

I also find it annoying that there is this limitation. I think other vendors are managing it pretty well to allow reading data more frequently. I personally would also prefer if I could just read the data directly from my access point but did not find a way to do so yet.

[helje5]

AFAIK the access point is just a bridge to the cloud service, it doesn't run the app server. To have something local, you need to use a CCU3, which you can query as often as you like (but using different protocols as of today).

[tschissler]

Thanks for the quick feedback. I was fearing that I have to buy a CCU3 to properly handle my scenario. I really like the approach Shelly (https://www.shelly.cloud/) is taking here :-) Maybe I should with over to their platform ;-) Still great what you folks are doing here.

[h3tz]

I am also using Shelly plugs to measure the solar power input, but only for the solar panels. For other devices, i am still using homematic and the 5min availability to keep track of them. I am thinking about using Wireshark to observ the homematic AP conversation. Maybe there is something like a black channel they are using because just using the app 5-second poll is not a problem.

I am just using this one for shellyPlugs: https://github.com/Jan200101/ShellyPy

[helje5]

I would guess that they (the app) uses a WebSocket connection to get value updates from the server. The AP is probably just a proxy (or VPN) and has not a lot of own logic.

[tschissler]

I am just using this one for shellyPlugs: https://github.com/Jan200101/ShellyPy

What I really like about Shelly is that they have a public documentation about the API https://shelly-api-docs.shelly.cloud/ That makes it easy and simple to communicate with the device from various technologies.

[jensenb]

Does anyone have any idea what the rate limits are for the REST API? I just started implementing a custom script to dump the values from my hmip devices into influxdb and was surprised by the throttling messages. I dropped the polling rate to 5 mins and this seems to be ok, but running at every 2 mins I was still getting the throttling messages. And really in terms of service usage, polling for updates every 2 mins should not be so stressful for their infrastructure.

The only statement I found from the company on the terms of the API usage is here: https://www.eq-3.de/service/faq/throttling.html

[hahn-th]

I checked my home assistant data points, and they are getting new states about every 2 1/2 minutes. Depends on the devices. Maybe you have to poll more irregular. For example a random value between 120 and 300 seconds. If i would try to find automatic polls, i would search for regularities.

This one confirmes my guesses: "Die Grenzen wurden so eingerichtet, dass eine interaktive Nutzung – beispielsweise mit den Homematic IP Smartphone Apps – nicht behindert wird."

On the other hand: do you really need that much data points? Isnt it maybe enough to have datapoints every 10 minutes? The more you stress the api out, the higher the possibility is, eq3 is hardening their rules.

[h3tz]

I observed that just pulling (without any interaction from another device like Android app Homematic) 2.5 min is possible. Setting actors is only possible every 5min. Both values got stabilized over the last 12 months. Everything less then 2.5 and 5 is not working properly 24/7.

Sending emails to the supplier about this topic ends up in nirvana.

Still, it will be interesting to do a Wireshark capture because the app itself is able to push and pull every 5 seconds.

[jensenb]

The strange thing for me is that the throttling exception does not appear to be applied consistently. I have had the exception applied after 30 mins of gathering data, but yesterday it kicked in only after 7 hours. I think the variable in my setup is that I have the mobile app installed on 2 devices, so I have no idea with which frequency they are grabbing data from the HMIP cloud.

[h3tz]

This one confirmes my guesses: "Die Grenzen wurden so eingerichtet, dass eine interaktive Nutzung – beispielsweise mit den Homematic IP Smartphone Apps – nicht behindert wird."

@jensenb

[jensenb]

I am using the synchronous REST API at the moment, which means I am creating a new connection to the HMIP cloud with every request.

Thinking about how EQ-3 formulated that message and that they have rate limit checks on the TCP level, I am wondering if using the web sockets API would be the way to go. It would mean at least fewer TCP connections from my local network, which I assume would mean less throttling.

[h3tz]

i already tried to use the same connection opened once with the same issues. Also i think its time to close this topic.

[hahn-th]

Do you just read values every few minutes? If this is true, you can import this library into your code and use the persistent connection to get updates as soon they are happening. So you will not have any troubles with the polling-rate.

[hahn-th]

I am talking about the Home class. See the implementation in home-assistant: https://github.com/home-assistant/core/blob/dev/homeassistant/components/homematicip_cloud/hap.py

Especially this function, where the Home class is instantiated and the callback methods getting assigned: https://github.com/home-assistant/core/blob/f43f0c4bcc070664519d14485cca1a462014a597/homeassistant/components/homematicip_cloud/hap.py#L236

In your code you can handle changes in the callback method.

[jensenb]

In my case I only experimented with the synchronous API. I will take a look at web sockets based API and see if it changes things for me.

I have been running an experiment for the last day. I poll updates every 5 mins. I get blacklisted anywhere from every 30 mins to every 90 mins. It doesn't seem to be very consistent.

i already tried to use the same connection opened once with the same issues. Also i think its time to close this topic.

Why? I don't think there has been a definite resolution on how to ensure you do not get throttled.

[hahn-th]

I will create a boilerplate and publish it on a new project. You can use this project to modify your application. After that, the problems are gone.

[h3tz]

Let me know the fork and i can give you support after initial check in,

[jensenb]

@hahn-th Are you creating a code example of how to use the asyncio API to passively listen for updates (instead of actively polling) ?

[jensenb]

I did a first scan of the code for using a web socket to listen for updates yesterday. I found the code quite complicated. There are 3 different different libraries used depending which part of the API you use: websockets, websocket-client, and aiohttp. And one variant uses asncio, the other uses threads(!). I am still working through how user code gets notified about incoming events.

[phavel]

I am still working through how user code gets notified about incoming events.

A simple tutorial is here: https://github.com/hahn-th/homematicip-rest-api#events I am working with it and it works great. Thanks for the library!

[hahn-th]

Here is it. Error Handling is missing and other stuff. But it should make clear how to use the library. https://github.com/hahn-th/homematicip-rest-api-boilerplate

[h3tz]

Here is it. Error Handling is missing and other stuff. But it should make clear how to use the library. https://github.com/hahn-th/homematicip-rest-api-boilerplate

It's working like charm. For me, with this solution of @hahn-th the issue is solved. Many Thanks!

[jensenb]

Thanks for putting this together! Will have a look tonight.

On Thu, Nov 24, 2022, 12:55 hahn-th @.***> wrote:

Here is it. Error Handling is missing and other stuff. But it should make clear how to use the library. https://github.com/hahn-th/homematicip-rest-api-boilerplate

— Reply to this email directly, view it on GitHub https://github.com/hahn-th/homematicip-rest-api/issues/460#issuecomment-1326348561, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFP5FZVOGBMJCC4BMXZFITWJ5JSLANCNFSM6AAAAAAQX3QATU . You are receiving this because you were mentioned.Message ID: @.***>

[jensenb]

@hahn-th This is exactly what I am looking for. Consuming updates from the various devices as they stream in, no active polling. I was totally unaware that the HMIP cloud streams out data to clients.

A simple tutorial is here: https://github.com/hahn-th/homematicip-rest-api#events

Shame on me for not reading the docs!

[jensenb]

That having been said, I do find the whole callback interface a bit clunky. Especially the fact that you have to keep the main thread indefinitely sleeping (or otherwise blocked). I am wondering if it wouldn't make more sense to offer a simplified version for clients that just want to consume updates. Something like:

home = Home()
home.set_auth_token(config.auth_token)
home.init(config.access_point)

async for update_info in home.updates():
    # Do something with the update_info here

What do you think? I could put together a draft PR to see what such an interface would look like.

[hahn-th]

Try not do block the main thread and see what happens :) You can simplify it by not using this handler class. But you need the EventHandler function.

[jensenb]

Try not do block the main thread and see what happens :) I did and I got the exact result I was expecting. When the main thread exits, the program exits. As it should be.

But this means you have to keep the main thread alive, but unoccupied. But since the background thread is not exposed via the API, you cannot have the main thread simply wait indefinitely on the web sockets thread. So repeated calls to sleep or similar. Pointless code that should not be necessary.

Which is why I think a more explicit API would be preferable, like i proposed in the earlier comment.

[jensenb]

For what it is worth, the web socket seems very stable, I have it running now for over 24 hours without an issue. Updates seem to propagate within seconds.

To anyone who is still interested in pursing the polling issue: 10 mins seems the shortest interval that the homematic cloud allows repeated requests without throttling.

[hahn-th]

You can use the async version and use your own loop. Like for example home assistant does it.

[hahn-th]

Try not do block the main thread and see what happens :) I did and I got the exact result I was expecting. When the main thread exits, the program exits. As it should be.

But this means you have to keep the main thread alive, but unoccupied. But since the background thread is not exposed via the API, you cannot have the main thread simply wait indefinitely on the web sockets thread. So repeated calls to sleep or similar. Pointless code that should not be necessary.

Which is why I think a more explicit API would be preferable, like i proposed in the earlier comment.

To be honest, i do not know exactly, what you mean. As i know websockets, its a default behaviour to keep them open and wait for updates. So if updates are populated thru websocket, you can handle them in your event handler function in an own thread. The main thread is just keeping the app alive. I don't want to open a connection and get current state, wait for some changes, quit the application and restart it a few mintutes later again.

of course, if you want to create a draft pr you can do that. Maybe things getting clear 😄 But keep in mind, that the application is used by people in different ways. Changing the behaviour is always critical. Do you have discord? Its maybe more easily to talk there.

[jensenb]

To be honest, i do not know exactly, what you mean. As i know websockets, its a default behaviour to keep them open and wait for updates. So if updates are populated thru websocket, you can handle them in your event handler function in an own thread.

I agree with you up to this point

The main thread is just keeping the app alive.

I disagree on this point. I think using callbacks introduces a layer of indirection that make it hard to follow the program flow. I think it is better practice to have the program flow more explicit. Instead of having the socket issue callback (and the clunky logic that is associated with it), why not just let the user code read updates directly as they come in, blocking explicitly when there is nothing to read. Let me make illustrate this point with an example from the AIOHttp library used by this library:

async with session.ws_connect('http://example.org/ws') as ws:
    async for msg in ws:
        if msg.type == aiohttp.WSMsgType.TEXT:
            if msg.data == 'close cmd':
                await ws.close()
                break
            else:
                await ws.send_str(msg.data + '/answer')
        elif msg.type == aiohttp.WSMsgType.ERROR:
            break

See how clear the program flow is? Translated to this library, I imagine a user facing API like:

async for update in home.next_update():
    # Do something with the update_info here
    if update["Type"] == EventType.GROUP_CHANGED:
        print("Group Changed Event")
        # Do something with the info here
   elif update["Type"] == EventType.DEVICE_CHANGED:
        # Do something with the device changed here

This is getting off topic of the original issue, but since I took the time to review this library I thought I would make a proposal for a simplification for users that are only interested in reading streaming updates.

[hahn-th]

I get it :) And i like it! And yes, its slightly offtopic. Contact me at homematicip-rest-api@thomas-hahn.org

[barryvdh]

I suddenly also get the 15-minute timeout when using HomeAssistant, see https://github.com/home-assistant/core/issues/104783

Is this something that has been more strict in recent days, or does that mean I probably use an invalid token or something?

[maxigs]

I haven't checked home-assistant logs (or done anything with the integration in months), but i did notice some laggy behavior lately in the official homematic-ip app. Could be a sign of my account being rate-limited. :/