search

wiktorn / Overpass-API

Overpass API docker image
MIT License
134 stars 48 forks source link

make area generation optional? #74

Closed nilsnolde closed 2 years ago

nilsnolde commented 3 years ago

I took a rather light-weighted OSM extract (a province in Germany) and fired up a container. All going smooth until it reaches the area generation (I think, judging by other issues). It seems to take a very long time compared to populating the db and it's stuck on this line of the entrypoint: https://github.com/wiktorn/Overpass-API/blob/4fe130a36a9df9b496e8c5cfceac21edf5342130/0.7.55.9/docker-entrypoint.sh#L83

with the output being along the lines of:

After 0h40m0s: in "recurse", part 0, on line 255. Stack: 0 of 0 171594 of 0 0 of 0
After 0h40m15s: in "recurse", part 0, on line 255. Stack: 0 of 0 173042 of 0 0 of 0
After 0h40m30s: in "make-area", part 0, on line 257. Stack: 0 of 0 175297 of 0
After 0h40m45s: in "make-area", part 0, on line 257. Stack: 0 of 0 176717 of 0
After 0h41m0s: in "make-area", part 0, on line 257. Stack: 0 of 0 178671 of 0

Before researching, I figured overpass made some nominatim call when using place names :sweat_smile: It's really super convenient for a lot of people, but really not everyone.

Would you accept a contribution to opt out of area computation via env var? I think the biggest hurdle is to disable the supervisord program dispatcher_areas, maybe we could use its autostart config option with a boolen env var OVERPASS_USE_AREAS which gets exported in the entrypoint script?

BTW, it seems to me the script updating the areas is never used (maybe on purpose)? https://github.com/wiktorn/Overpass-API/blob/master/0.7.55.9/bin/rules_loop.sh my bad, I deleted a part of the supervisord config already..

nilsnolde commented 3 years ago

I'm also not 100% why it's taking so long to generate areas for this small area: http://download.geofabrik.de/europe/germany/saarland.html.

I seem to remember geofabrik does include lots of irrelevant relations which are missing the corresponding ways, I'm not often working with relations. Could that be a problem for the area generator?

wiktorn commented 3 years ago

Hi,

Making area optional will be most welcome PR. What I think, would be good, is that it would be optional both at the startup as well as it should disable the background update of areas started through supervisord. I think, we could extend OVERPASS_RULES_LOAD values to allow 0 with the meaning, that areas will not be generated at all.

I have no idea why your small extract is taking so long to process. But as far as I know, missing ways in relations are only non-issue, in case of full planet dumps. In all partial extracts that I worked with there were some missing ways.

nilsnolde commented 3 years ago

What I think, would be good, is that it would be optional both at the startup as well as it should disable the background update of areas started through supervisord

yeah I think so too. we could also use your suggestion of OVERPASS_RULES_LOAD, just a couple of thoughts why IMHO a new "boolean" OVERPASS_USE_AREAS (true/false) would be beneficial:

I'm fine with whatever you're happier with though.

larsschwarz commented 2 years ago

+1

Also using an extract and DB setup finished hours ago, but still seeing

After 2h13m57s: in "recurse", part 0, on line 255. Stack: 0 of 0 636743 of 0 0 of 0
After 2h14m12s: in "make-area", part 0, on line 257. Stack: 0 of 0 638139 of 0
nilsnolde commented 2 years ago

oh damn, totally forgot about this.. actually you can try #75 @larsschwarz , it was pretty much done IIRC. I used that branch for a setup which seems to be working