The api blocks all requests (403) with the user-agent "pyIndego" [develop]

[FBruynbroeck]

Since yesterday (May 31, 2023), all requests return a 403 error (pyIndego==3.0.1). This problem is mentioned in the source code by @sander1988: https://github.com/jm-73/pyIndego/blob/4606ed1f18daaac2205c35cdd38499a18242ed19/pyIndego/const.py#L22-L27

I've tested this by simply changing the user-agent to another random string and it works.

Maybe generate a fake random user-agent instead of a hard-coded name used by everyone?

[sander1988]

Good find! I see the same issue here since this morning.

I think it's some kind of auto-learning security tool of Microsoft that sees this agent as a security risk now. I will try to apply a quick workaround.

[sander1988]

A fix is in the repo ; version v2.2.2 of this library. I'm getting data again.

Looking at the logs in my instance, I might it see why it was blocked. It was doing too many requests (multiple a second). This should not happen. The question is was this the reason of the 403 or the results of the 403. I'm not sure.

With this fix it should prevent the issue from affecting all users when there is something wrong with the HA instance of one user.

[kimzeuner]

Hpw can i get the new version or edit the code manually ? Thanks in advance!

[FBruynbroeck]

@sander1988 Maybe related to https://github.com/jm-73/Indego/issues/133 I'm implementing the map in HA and this feature calls the API very often. I'm trying to limit the calls (don't call when the mower is docked, limit the frame interval to 5 seconds,...) If there are several of us testing this, the Azure proxy probably didn't like it. Especially for tests performed before these commits: https://github.com/FBruynbroeck/Indego/commit/34adbd53d110fc3e57d643cadb74e00ae90ac9b0 https://github.com/FBruynbroeck/Indego/commit/7a1caf49f981cef34cfa11b492d6d8d3c1e2e8ba

Having a unique/random user-agent per instance seems to me to be a good solution. Thanks for this correction (I was busy doing it but you were faster than me 😅).

[LarsLautrup]

What do I need to get the Indego integration in Home Assistant going again? I have no DEFAULT_HEADER and User-Agent settings in the const.py file in custom_components\indego

(I am a complete amateur and I dont really know what I'm doing, :-) )

[parkkralle]

What do I need to get the Indego integration in Home Assistant going again? I have no DEFAULT_HEADER and User-Agent settings in the const.py file in custom_components\indego

(I am a complete amateur and I dont really know what I'm doing, :-) )

Same problem here. How can we fix this in the Home Assistant integration?

[jm-73]

This is one of the reasons that I havent implemented the map function in the integration. When the integration first was developed I expermeneted with the map. But in order to get meaningful updates I had to hammer the api to get realtime data. I then came to the conclusion that doing this, and maybe 100's or 100's of other HA users would not end well. I feared that Bosch would investigare and put up measures in order to get rid of the traffic.

I now fear that we are there. Maybe widespread use of the new map functions are sabotaging for us all?

[FBruynbroeck]

@jm-73 We need to find the right balance.

Modifying the user-agent is already a good thing.

Regarding updating the map, I respect Bosch's wishes by limiting the stream refresh to once every 5 seconds, similar to their mobile application (and once every 10 seconds for the preview).

There will be no map update if the mower is docked (as it is not relevant). Background updates will not occur if the map is not displayed from a web browser or mobile app.

As I mentioned before, I believe the issue we encountered here is related to an older version of my fork, where I had not yet implemented these limitations. At that time, the stream refresh rate was 4 API calls per second (which was too high), or even more if the map was displayed multiple times (opening multiple navigation tabs, using the mobile application simultaneously, etc.). We were all experiencing issues because we were using the same user-agent. However, this is just a speculation.

I am continuously improving the map update to minimize the number of calls as much as possible. The next step will be to store the map to avoid calling the "/map" route after each refresh and only call "/state" (with forceRefresh) to retrieve the accurate position.

I have tested my latest modifications, and the results are promising. I displayed the stream throughout the mowing duration. I make API calls every 5 seconds to retrieve the mower's current position (only when the mower is running). No calls are made while the mower is docked. If multiple streams are displayed, the API calls are limited to a maximum of once every 5 seconds. With this approach, it should not cause any disruptions.

[FBruynbroeck]

@LarsLautrup @parkkralle While waiting for a future release on PyPI, you can install the patch by @sander1988 in the following way:

pip3 install --upgrade https://github.com/sander1988/pyIndego/archive/c270bf5cf2d9a549c33a960523f8211aec997fac.zip

(Be careful to use the pip of the Python version running your HA instance)

And on the Indego plugin side, modify the version of pyIndego in the manifest.json file:

"requirements": ["pyIndego==2.2.2"],

[sytchi]

@LarsLautrup @parkkralle While waiting for a future release on PyPI, you can install the patch by @sander1988 in the following way:

pip3 install --upgrade https://github.com/sander1988/pyIndego/archive/c270bf5cf2d9a549c33a960523f8211aec997fac.zip

(Be careful to use the pip of the Python version running your HA instance)

Is there a way to achieve that running home assistant in hass.io ?

[sander1988]

Is there a way to achieve that running home assistant in hass.io ?

Changing the requirements line in the manifest of the component should work in all cases; also hass.io : https://github.com/sander1988/Indego/blob/master/custom_components/indego/manifest.json#L8

You should restart HA after changing this line in the manifest.json ; HA will auto install the requirements when loading the component.

[parkkralle]

Is there a way to achieve that running home assistant in hass.io ?

Changing the requirements line in the manifest of the component should work in all cases; also hass.io : https://github.com/sander1988/Indego/blob/master/custom_components/indego/manifest.json#L8

You should restart HA after changing this line in the manifest.json ; HA will auto install the requirements when loading the component.

It works. Thank you so much!

[jm-73]

@sander1988 If you have solved the user-agent-problem, could you do a pull request so I can build a working release?

[clfberlin]

Unfortunately changing the manifest.json to "requirements": ["pyIndego==2.2.2"], doesn't help me with the first installation. It then gives me a "500 Internal Server Error Server got itself in trouble". With the previous setting it came back with the error that no Indego mowers connected to that account.

[FBruynbroeck]

@clfberlin try this:

"requirements": ["git+https://github.com/sander1988/pyIndego.git#pyIndego==2.2.2"],

[clfberlin]

Thanks! That did the trick!

[spanier106]

"Keine Mäher in diesem Bosch Indego-Konto gefunden"

@clfberlinVersuche dies:

"requirements": ["git+https://github.com/sander1988/pyIndego.git#pyIndego==2.2.2"],

wo genau soll ich das eingeben? Im Terminal geht es bei mir nicht. Sorry blutiger anfänger

___-

Danke Funktioniert

[LarsLautrup]

In the Manifest.json file - in the custom_components\indego folder.

[spanier106]

In der Datei Manifest.json – im Ordner „custom_components\indego“.

danke!! Funktioniert

[jm-73]

@sander1988 If you have solved the user-agent-problem, could you do a pull request so I can build a working release?

@sander1988 ???

[sander1988]

@sander1988 If you have solved the user-agent-problem, could you do a pull request so I can build a working release?

@sander1988 ???

I'm on a trip and no have no access to a laptop atm. The (iOS) app is quite limited; it only lets me reply. Will create it when I'm back.

[sander1988]

@jm-73 I have just created the PR (https://github.com/jm-73/pyIndego/pull/120)

Please note that there might be still a 403 loop issue in the current and this version. I will try to debug that when I have time.

[sander1988]

Please note that there might be still a 403 loop issue in the current and this version. I will try to debug that when I have time.

I have just added some fixes regarding 4xx error handling to prevent loops. Version 2.2.3 on my develop branch. I will be testing it the next couple of days and merge when no errors appear in the logs.

The current PR can still be merged as it contains the user agent fix.

[SmarthomeAddicted]

This is one of the reasons that I havent implemented the map function in the integration. When the integration first was developed I expermeneted with the map. But in order to get meaningful updates I had to hammer the api to get realtime data. I then came to the conclusion that doing this, and maybe 100's or 100's of other HA users would not end well. I feared that Bosch would investigare and put up measures in order to get rid of the traffic.

I now fear that we are there. Maybe widespread use of the new map functions are sabotaging for us all?

so if I understand that right the suspect was that due too to many request for the maps and the high amount of data Bosch get rid of the user-agent pyindego ? If yes I found a solution and integrate in some own version of my integration where the map it just download once and after every change of position X and Y value will be updated in the map. But I'm not sure if the is some feature that should be implemented here. But I think that would be the best way that we don't stress the Bosch servers with our map update requests every minute.
I made and explanation to set this up in the video below ( unfortunately in German)

https://www.youtube.com/watch?v=-sOSOXz-Vv0&t=2s

[jm-73]

How do you get the ipdates for the map? Push or pull? And how often? Or is the position collected in the same way that the indego integration waits for data update from the Bosch API server?

[sander1988]

I have just added some fixes regarding 4xx error handling to prevent loops. Version 2.2.3 on my develop branch. I will be testing it the next couple of days and merge when no errors appear in the logs.

I have just created a pull request ; it's ok to merge for beta. I have been running this version for a couple of days without issues. I saw the Bosch servers giving me a 403 a couple of times (which is expected behavior every now and then), this time is was handled correctly by our integration. So I will call it fixed ;-)

@jm-73 - Note that I have justed synced with your repo. So we are using the same versioning again to keep this clear and easy to track. This new beta should be 3.1.1

[sander1988]

But I think that would be the best way that we don't stress the Bosch servers with our map update requests every minute.

@SmarthomeAddicted - I think I have found your status-update code: https://github.com/SmarthomeAddicted/Indego-Integration/blob/e25557c91afed237f6df45ddfa0f6bd9944db10a/indego/__init__.py#L1212 I'm I right?

I think the map is an useful feature. My first thoughts after looking at the code are below. I hope it's helpful to find the right balance between functionality and server resources/calls.

1. You should only update the position when it's the very first time the integration is loaded or when map_update_available is True. That's under the assumption that map_update_available becomes True when the position changes (I have not tested this but I expect this behavior). This way you will only fire some additional calls to their servers when the mower is actually mowing and at the rate Bosch find acceptable (as they can control the rate on which map_update_available becomes True).

2. And I also recommend a little wait time when their API gives a 4xx or 5xx error. Just wait a minute when such error happens. That's what I implemented in the last update and will probably prevent hard blocks by their servers.

[SmarthomeAddicted]

But I think that would be the best way that we don't stress the Bosch servers with our map update requests every minute.

@SmarthomeAddicted - I think I have found your status-update code: https://github.com/SmarthomeAddicted/Indego-Integration/blob/e25557c91afed237f6df45ddfa0f6bd9944db10a/indego/__init__.py#L1212 I'm I right?

I think the map is an useful feature. My first thoughts after looking at the code are below. I hope it's helpful to find the right balance between functionality and server resources/calls.

1. You should only update the position when it's the very first time the integration is loaded or when map_update_available is True. That's under the assumption that map_update_available becomes True when the position changes (I have not tested this but I expect this behavior). This way you will only fire some additional calls to their servers when the mower is actually mowing and at the rate Bosch find acceptable (as they can control the rate on which map_update_available becomes True).
2. And I also recommend a little wait time when their API gives a 4xx or 5xx error. Just wait a minute when such error happens. That's what I implemented in the last update and will probably prevent hard blocks by their servers.

@sander1988 - yes you are right.

Regarding 1: I just put it in the _update_state because I thought that when this function is called it will give me everything which is documented (https://github.com/jm-73/pyIndego) for this function. So I thought it will not stress the server because they gave me e.g. 10 entries but in the integration just 5 was used. But when it is not so than I would have the same Opinion like and I need to test the behavior of map_update_available

Regarding 2: makes sense I will try to implement it. Thanks for the hint

[SmarthomeAddicted]

How do you get the ipdates for the map? Push or pull? And how often? Or is the position collected in the same way that the indego integration waits for data update from the Bosch API server?

I just added the following line of code to the async def _update_state(self):

` try:

await _update_position(self.indego.state.svg_xPos,self.indego.state.svg_yPos)

            svg = fromfile(f"www/mapWithoutIndego.svg")
            xpos = self.indego.state.svg_xPos
            ypos = self.indego.state.svg_yPos
            _LOGGER.info(f'Indego position (x,y): {xpos},{ypos}')
            circle = f'<circle cx="{xpos}" cy="{ypos}" r="15" fill="yellow" />'
            mower_circle = fromstring(circle)
            _LOGGER.info(f'Adding mower to map and save new svg...')
            svg.append(mower_circle)
            svg.save(f"www/mapWithIndego.svg")
        except Exception as e:
            _LOGGER.info("Update state got an exception: %s", e)`

and I also create a call service for downloading the map

async def _download_map(self, filename: str): _LOGGER.debug(f"Downloading map to {filename}") await self.indego.download_map(filename)

after map downloading is done ( only need to be done once) and the position changed it will create a new svg file (or overwrite the existing one) with the current position based on self.indego.state.svg_yPos and self.indego.state.svg_xPos

as I saw that this map feature is a wish by some users I tought about to add this in this integration here but I'm not sure if this will be integrated and if yes how it will have influence on the traffic on the Bosch server with how I programmed it. In addition to it I tried to get as much information as possible out of the serves with the pyIndego library.

Also I made some call services to mark the notification as read or delete it (for the last one or for all)

`async def async_delete_alert(call): """Handle the service call.""" alert_index = call.data.get(CONF_DELETE_ALERT, DEFAULT_NAME_COMMANDS) _LOGGER.debug("Indego.delete_alert service called, with command: %s", alert_index) await hass.data[DOMAIN]._update_alerts() await hass.data[DOMAIN].indego.delete_alert(alert_index) await hass.data[DOMAIN]._update_alerts()

async def async_delete_alert_all(call):
    """Handle the service call."""
    alert_index = call.data.get(CONF_DELETE_ALERT, DEFAULT_NAME_COMMANDS)
    _LOGGER.debug("Indego.delete_alert_all service called, with command: %s", "all")
    await hass.data[DOMAIN]._update_alerts()
    await hass.data[DOMAIN].indego.delete_all_alerts()
    await hass.data[DOMAIN]._update_alerts()   

async def async_read_alert(call):
    """Handle the service call."""
    alert_index = call.data.get(CONF_READ_ALERT, DEFAULT_NAME_COMMANDS)
    _LOGGER.debug("Indego.read_alert service called, with command: %s", alert_index)
    await hass.data[DOMAIN]._update_alerts()
    await hass.data[DOMAIN].indego.put_alert_read(alert_index)
    await hass.data[DOMAIN]._update_alerts()

async def async_read_alert_all(call):
    """Handle the service call."""
    alert_index = call.data.get(CONF_READ_ALERT, DEFAULT_NAME_COMMANDS)
    _LOGGER.debug("Indego.read_alert_all service called, with command: %s", "all")
    await hass.data[DOMAIN]._update_alerts()
    await hass.data[DOMAIN].indego.put_all_alerts_read()
    await hass.data[DOMAIN]._update_alerts()`

[jm-73]

Solved