API status endpoint changed format?

[gboeing]

Has the API status endpoint changed its format? If so is there documentation for it? It used to look like:

Connected as: 1807298401 Current time: 2022-06-03T21:31:20Z Rate limit: 2 2 slots available now. Currently running queries (pid, space limit, time limit, start time):

Now it looks like:

Connected as: 1807298401 Current time: 2022-06-03T21:31:20Z Announced endpoint: none Rate limit: 2 2 slots available now. Currently running queries (pid, space limit, time limit, start time):

The Announced endpoint line appears to be new and is breaking our plaintext parser we use to pause querying to obey the rate limit. This should be an easy fix on our end, but is there documentation for the status endpoint, or a way to guarantee consistent ongoing response format when we query it?

[gboeing]

Part of the challenge in parsing is that other public API instances (such as here, here, and here) are still using the old status response format, meaning that depending on the instance, we cannot consistently parse the same response text line to get the slot availability.

[gboeing]

@mmd-osm @drolbr let me know if you have any advice on parsing and response consistency. We want to ensure we are a "good neighbor" by obeying rate limiting properly.

[mmd-osm]

https://wiki.openstreetmap.org/wiki/Overpass_API/versions mentions: The possibility of setting a server name in the backend improves error messages on the frontend: when using DNS based load balancing, you are now told on which server your quota has been exceeded.

I'm not sure if the additional line Announced endpoint: z.overpass-api.de/api/ adds any value to osmnx. You've already implemented all logic to reliably connect to a single server and make sure that the /api/status endpoint response matches /api/interpreter.

Unless I'm missing something, the added line in /api/status won't resolve any of the issues with DNS round robin. If you don't put in extra effort on client side to make sure both /api/status and /api/interpreter always hit the same server, it's very hard to have meaningful load control (i.e. /api/interpreter might still fail, even though /api/status claims everything is fine, because both requests hit different servers).

Bottom line: I would simply ignore the new information.

[drolbr]

First of all, thank you for reporting back the problem. I'm sorry for the inconvenience. When I added this line, I was not aware that this may cause trouble downstream.

The "good neighbor": I am not aware that osmnx were causing any trouble to the servers. Sending back a HTTP 429 every few seconds is negligible for to the load. Rather, sending back a HTTP 429 multiple times per second raises the load. From the server's load perspective it would be perfectly fine to bluntly just wait 5 seconds and the retry no matter whether there is a chance.

If you care for the fact that multiple endpoints exist then I suggest to alternate between

• z.overpass-api.de and
• lz4.overpass-api.de The names are stable for the foreseeable future, i.e. some years. If you just please tell me when you implement a list with these entries then I will put the server names on my list of non-obvious relevant parts of the user interface that should not change without announcement. If I understand mmd correctly then you do already.

That there should be no extra lines in the /api/status endpoint is now on that list, too.

[mmd-osm]

Quickly commenting on the last section: OSMnx downloader is implemented in a way that it doesn't need actual server names like z.overpass-api.de or lz4.overpass-api.de. Instead, it uses the DNS response for overpass-api.de and does all the rest automatically, including server stickiness across requests.

Relevant code can be found here: https://github.com/gboeing/osmnx/blob/main/osmnx/downloader.py#L290-L301

[gboeing]

Thank you both for your responses. OSMnx will ignore the new Announced endpoint line accordingly, per @mmd-osm's advice.

@drolbr yes it would be great to keep the server names unchanged (though this doesn't directly affect OSMnx), as well as keeping the /api/status endpoint's output format unchanged for parsers that are expecting this format (which does directly affect OSMnx's rate limiting capabilities).

Thanks again. We're endlessly grateful for the work you do!