Overpass API partial response error

[Vectorial1024]

Have been trying to download a OSM map of Hong Kong + nearby area these few days, but kept getting an error which did not appear a few months ago.

The actual error is happening on Python, but I am reproducing this on overpass-turbo website https://overpass-turbo.eu/ for simplicity.

Query:

(
nwr(22.1038,113.52695,22.56655,114.4557);
- nwr(22.1038,113.52695,22.56655,114.4557)["seamark:type"];
);
out geom;

I am using Firefox; also open the network inspector of my browser

Expects:

Dialog box "the data you are trying to view is quite large, please reconsider" (i.e. the request is successful)

Actual:

Dialog box "Ajax Error: An error occurred during the execution of the overpass query! Error-Code: error (0)"

Network inspector says:

NS_ERROR_NET_PARTIAL_TRANSFER

0GB was returned.

Other Context

If I retry but with [timeout:30]; then it still fails, but there are like about 2MB of partial data returned, which is clearly not enough to describe Hong Kong.

[mmd-osm]

You're trying to download 857MB worth of XML data and display the results in your browser. This will simply not work, since it's too much data for your browser. In any case, you need to download the data to a file, and avoid displaying anything in your browser.

Also, I hope you're aware of OSM extracts, which provide even larger areas for download, e.g. from https://download.geofabrik.de/asia.html ... That's the preferred approach in your case.

[Vectorial1024]

I am already aware of OSM extracts, but considering the extreme size of the map of China (PRC) and the lack of sub-region extracts, using this to construct a map of Hong Kong is very difficult and inefficient. In this case, I prefer to download via Overpass API instead.

---

But still, if reproduction via browser is not really working, then perhaps I should directly provide a simplified version of my Python code.

I'm using the overpass Python library. https://pypi.org/project/overpass/

import overpass

query = """
(
nwr(22.1038,113.52695,22.56655,114.4557);
- nwr(22.1038,113.52695,22.56655,114.4557)["seamark:type"];
)
"""
api = overpass.API(debug=True)
response = api.get(query, responseformat="xml", verbosity="meta")
# save to file, etc

This eventually leads to different Python errors; one of the errors is:

  File "C:\ProgramData\miniconda3\Lib\site-packages\requests\models.py", line 818, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(82 bytes read, 8014 more expected)', IncompleteRead(82 bytes read, 8014 more expected))

This still indicates a potential problem on Overpass's side, unless somehow my Internet connection has some quirks that I am not aware of.

[SomeoneElseOSM]

(not related to Overpass but)

but considering the extreme size of the map of China (PRC) and the lack of sub-region extracts, using this to construct a map of Hong Kong is very difficult and inefficient.

China is about 1GB in OSM: http://download.geofabrik.de/asia/china.html - cutting Hong Kong out of that with a tool such as osmium is fairly straightforward; people do that sort of thing all the time (I certainly have example code that I can share). I'd suggest asking in the forum: https://community.openstreetmap.org/c/help-and-support/7/none .

[mmd-osm]

This still indicates a potential problem on Overpass's side, unless somehow my Internet connection has some quirks that I am not aware of.

This seems a bit unlikely. I was able to download 857MB using wget / curl without any issues. I'd suggest to enable compression then your download will only 140MB.

Also, I'm not getting any errors when running your script (which by the way could be optimized to avoid the "delta" operation).

Like @SomeoneElseOSM said, you can do the free server resources you're using here a great favor by using an extract instead.

[Vectorial1024]

Actually, the next step of my original Python script is to further trim the rectangular box by giving it to osmium with a tighter polygon, so I also know how to do that.

This is very weird... I should probably recheck my Internet somehow.

Seeing your reassurance I guess I can transition to using geofabrik extracts; supposedly my script would only run eg once per a few months, but given how strange my Internet is and the state of the free server, might as well change to geofabrik extracts.

Thanks for your answers!