timwie / aio-overpass

Async client for the Overpass API
https://pypi.org/project/aio-overpass/
MIT License
4 stars 1 forks source link

Handle cutoff JSON responses #9

Closed timwie closed 11 months ago

timwie commented 11 months ago

There were two worklow runs here and here where the JSON response body is not complete:

...
2023-11-14T02:36:52.5363881Z {
2023-11-14T02:36:52.5363975Z   "type": "way",
2023-11-14T02:36:52.5364065Z   "id": 1154374586,
2023-11-14T02:36:52.5364155Z   "bounds": {
2023-11-14T02:36:52.5364248Z     "minlat": 40.3820610,
2023-11-14T02:36:52.5364439Z     "minlon": -3.7522717,
2023-11-14T02:36:52.5364539Z     "maxlat": 40.3822646,
2023-11-14T02:36:52.5364659Z     "maxlon": -3.7519612
2023-11-14T02:36:52.5364745Z   },
2023-11-14T02:36:52.5364831Z   "nodes": [
2023-11-14T02:36:52.5364922Z     10735908237,
2023-11-14T02:36:52.5365009Z     10735908269,
2023-11-14T02:36:52.5365095Z     10735908270,
2023-11-14T02:36:52.5365186Z     10735908271
2023-11-14T02:36:52.5365265Z   ],
2023-11-14T02:36:52.5365367Z   "ge
<end of response>

json.decoder.JSONDecodeError: Unterminated string starting at: line 1148050 column 3 (char 20992877)
...
2023-11-13T23:32:52.3483076Z {
2023-11-13T23:32:52.3483168Z   "type": "node",
2023-11-13T23:32:52.3483250Z   "id": 4058074348,
2023-11-13T23:32:52.3483340Z   "lat": 40.3816635,
<end of response>

json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 141863 column 21 (char 2133393)

This also caused us to log over 100000 lines of such a response, which is obviously not ideal either.