Area search doesn't seem to work

[tomvo]

After a 24 hours parsing of the Europe dataset from Geofabrik I have the overpass api succesfully running inside the docker container. Simple overpass queries seem to work just fine but for some reason area limiting doesn't seem to work, no error, just no results. Below query works fine on the public overpass-turbo server but not on my own instance.

{{geocodeArea:Germany}}->.country;
rel(area.country)["boundary"="administrative"]["name"~"Main"][admin_level=10];
out geom;

area["ISO3166-1"="DE"][admin_level=2];
rel["boundary"="administrative"]["name"~"Mainz"][admin_level=6](area);
out geom;

When I remove the area restriction it works but it returns all matches inside Europe.

Any idea?

[wiktorn]

Are you sure that areas have been calculated? Area's calculation is not part of the initialization script. Please check contents of the file inside containter /db/db/rules_loop.log if any update has finished, like:

2018-12-19 06:52:08: update started
2018-12-19 07:08:29: update finished

Above entry is only for one country. For whole EU it might take considerably longer to compute.

[tomvo]

@wiktorn Ah, I guess it's still running then:

2018-12-19 06:39:46: update started

Nothing about it being finished. I'll wait for that then and try again. Thanks for your quick help here!

[wiktorn]

I guess that's solved.

[lukey78]

Is there any way to determine when area creation has finished and area queries become available? I've setup the docker image yesterday for the whole planet and area queries return no elements yet.

My rules_loop.log looks like this:

2020-06-12 06:20:55: update started 2020-06-12 07:31:02: update started 2020-06-12 07:31:03: update finished 2020-06-12 07:32:42: update started 2020-06-12 17:40:14: update started 2020-06-12 17:40:14: update finished 2020-06-12 17:40:17: update started 2020-06-12 17:47:28: update started

Thanks for a hint.

[wiktorn]

Entry in rules_loop.log like one that you see:

2020-06-12 17:40:14: update finished

Means that area calculation has ended and area queries should be working.

Though I worry that you see two starts and only one finish. What did you set for OVERPASS_RULES_LOAD?

Are you sure that you don't have two containers running from one database directory?

[lukey78]

Hm, I restarted the container 2 or 3 times today. I left OVERPASS_RULES_LOAD at the default, but changed it to 20 now on the latest start (the 17:47 time stamp).

First startup was yesterday in the evening. Today in the morning, after I saw the "Update finished" message, I tried a simple area query:

[out:json][timeout:5]; area[name="London"]; out body;

Then I waited until the evening. Nothing yet. Are there other signs to see if areas should be available?

p.s.: normal queries work well

[wiktorn]

If you use areas there should be header informing about the date of areas, as follows: curl -g 'http://localhost:12345/api/interpreter?data=[out:json];area[name="London"];out;'

{
  "version": 0.6,
  "generator": "Overpass API 0.7.56.3 eb200aeb",
  "osm3s": {
    "timestamp_osm_base": "2020-06-11T20:58:48Z",
    "timestamp_areas_base": "2020-06-11T20:58:48Z",
    "copyright": "The data included in this document is from www.openstreetmap.org. The data is made available under ODbL."
  },
  "elements": [

  ]
}

(This instance doesn't have UK data).

Anyway since c764547808a3089a9587c4d2b8d782c036a69881 areas are generated during initial container setup, so they should be working right away, after first exit of container.

[lukey78]

Ok, thanks. This is my output one day later:

{
  "version": 0.6,
  "generator": "Overpass API 0.7.56.3 eb200aeb",
  "osm3s": {
    "timestamp_osm_base": "2020-06-13T06:09:58Z",
    "copyright": "The data included in this document is from www.openstreetmap.org. The data is made available under ODbL."
  },
  "elements": [
  ]
}

I try again from scratch with this environment variables:

      - OVERPASS_META=yes
      - OVERPASS_MODE=clone
      - OVERPASS_DIFF_URL=https://planet.openstreetmap.org/replication/minute/
      - OVERPASS_RULES_LOAD=20
      - OVERPASS_UPDATE_SLEEP=300

I let you know in the evening how it went. Thanks for your help so far. This is a great image. I tried it myself last year (jhassler/overpass-api), but now I'm going to switch to this one. Great work!

[wiktorn]

Ok, so it looks like areas have failed to build. Do you have any logs from container creation, such as these:

Update finished with status code: 3
<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="Overpass API 0.7.56.3 eb200aeb">
<note>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</note>
<meta osm_base="2020-06-11T20:50:38Z" areas="2020-06-11T20:50:38Z"/>

After 0h0m15s: in "make-area", part 0, on line 29. Stack: 0 of 0 1109 of 0
After 0h0m30s: in "recurse", part 0, on line 255. Stack: 0 of 0 725 of 0 0 of 0
After 0h0m45s: in "recurse", part 0, on line 255. Stack: 0 of 0 3600 of 0 0 of 0
After 0h1m0s: in "make-area", part 0, on line 257. Stack: 0 of 0 6198 of 0
After 0h1m15s: in "make-area", part 0, on line 257. Stack: 0 of 0 7807 of 0
After 0h1m30s: in "recurse", part 0, on line 255. Stack: 0 of 0 10398 of 0 0 of 0
After 0h1m45s: in "make-area", part 0, on line 257. Stack: 0 of 0 13265 of 0

</osm>
Overpass ready, you can start your container with docker start

(This was for small extract). If you have whole world, your areas creation might have timed out, as by default, it's only allowed 86400 seconds to finish. See the file /app/etc/rules/areas.osm3s within container.

[lukey78]

I think when creating the container by cloning, the areas are not created initially. The process is started upon first start after cloning.

I restarted everything from scratch in the morning. Cloning was done in about an hour. Now I have one line in my rules_loop.log:

2020-06-13 07:37:58: update started

The current logs output stuff like this:

After 10h39m7s: in "make-area", part 0, on line 257. Stack: 0 of 0 1538393 of 0
After 10h39m22s: in "make-area", part 0, on line 257. Stack: 0 of 0 1540403 of 0
After 10h39m37s: in "recurse", part 0, on line 255. Stack: 0 of 0 1543817 of 0 0 of 0
After 10h39m52s: in "make-area", part 0, on line 257. Stack: 0 of 0 1546804 of 0
INFO: Using replication server at https://planet.openstreetmap.org/replication/minute/
DEBUG: Using given sequence ID 4064023
DEBUG: Starting download at ID 4064024 (max 100 MB)
DEBUG: Downloaded change 4064024. (102190 kB available in download buffer)
DEBUG: Downloaded change 4064025. (102013 kB available in download buffer)
DEBUG: Downloaded change 4064026. (101964 kB available in download buffer)
DEBUG: Downloaded change 4064027. (101803 kB available in download buffer)
DEBUG: Downloaded change 4064028. (101372 kB available in download buffer)
/app/bin/update_database --version=2020-06-13T18:17:01Z --compression-method=gz --map-compression-method=gz --flush-size=16 --meta
Reading XML file ... finished reading nodes. Flushing to database .....After 10h40m7s: in "make-area", part 0, on line 257. Stack: 0 of 0 1548868 of 0
After 10h40m22s: in "make-area", part 0, on line 257. Stack: 0 of 0 1550027 of 0
. done.
Reading XML file ... finished reading ways. Flushing to database ....After 10h40m37s: in "make-area", part 0, on line 257. Stack: 0 of 0 1551364 of 0
.After 10h40m52s: in "make-area", part 0, on line 257. Stack: 0 of 0 1552624 of 0
After 10h41m7s: in "make-area", part 0, on line 257. Stack: 0 of 0 1555364 of 0
After 10h41m23s: in "recurse", part 0, on line 255. Stack: 0 of 0 1557910 of 0 0 of 0
After 10h41m38s: in "make-area", part 0, on line 257. Stack: 0 of 0 1560289 of 0

I won't touch the container and see what happens tomorrow. Then I can say if it really times out or if it starts working.

[lukey78]

Ok, done.

After 24 hours and 1 minute the areas loop finished:

2020-06-13 07:37:58: update started
2020-06-14 07:39:09: update finished

... and the API returns data. Great!

Interestingly, the header still does not contain the "timestamp_areas_base".

{
  "version": 0.6,
  "generator": "Overpass API 0.7.56.3 eb200aeb",
  "osm3s": {
    "timestamp_osm_base": "2020-06-14T09:14:02Z",
    "copyright": "The data included in this document is from www.openstreetmap.org. The data is made available under ODbL."
  },
  "elements": [

{
  "type": "area",
  "id": 2436743047,
  "tags": {
    "name": "London",
    "natural": "wetland",
    "source": "NPE",
    "source:alt_name": "OS 1854",
    "wetland": "marsh"
  }
}
[...]

[lukey78]

Ok, it wasn't quite finished, I think it got killed like you said after 24 hours. Only a subset of the data was queryable.

I edited /app/etc/rules/areas.osm3s and adjusted the timeout, then I restarted the container and waited another 24 hours.

Now it displays the right header:

{
  "version": 0.6,
  "generator": "Overpass API 0.7.56.3 eb200aeb",
  "osm3s": {
    "timestamp_osm_base": "2020-06-16T20:06:01Z",
    "timestamp_areas_base": "2020-06-15T18:30:59Z",
    "copyright": "The data included in this document is from www.openstreetmap.org. The data is made available under ODbL."
  },
...

Actually I'm running on a very fast machine (10 Core 2.6 GHz, 256 GB RAM). Any way to speed this up or use multi processing?

[wiktorn]

I can only refer you to Overpass API tickets: drolbr/Overpass-API#540, drolbr/Overpass-API#349 and drolbr/Overpass-API#534.

Generally, maybe local fast SSD storage might help you more with that than cores/RAM (though probably almost whole database fits in your RAM).

Rules_delta might help updating areas faster than initially but this is not something that I've tested.