search

wiktorn / Overpass-API

Overpass API docker image
MIT License
134 stars 48 forks source link

How to increase update frequency? #68

Closed mgd722 closed 3 years ago

mgd722 commented 3 years ago

I ran the container as follows:

docker run \
  -e OVERPASS_MODE=clone \
  -e OVERPASS_META=attic \
  -e OVERPASS_DIFF_URL=https://planet.openstreetmap.org/replication/minute/ \
  -e OVERPASS_RULES_LOAD=20 \
  -e OVERPASS_UPDATE_SLEEP=15 \
  -v /big/docker/overpass_clone_db/:/db \
  -p 127.0.0.1:12346:80 \
  -i -t \
  --name overpass_world \
  --restart=always \
  wiktorn/overpass-api

It is running and updating just fine, but it seems to only update approximately every third minute and is always at least 2 mins behind the overpass-api.de server (and sometimes up to 5 mins behind). For example, responses to /api/augmented_diff_status would look something like this:

minute 0 1 2 3 4 5 6
me 4463225 4463225 4463225 4463227 4463227 4463227 4463230
overpass-api.de 4463227 4463228 4463229 4463230 4463231 4463232 4463233

I would think that with OVERPASS_UPDATE_SLEEP=15 it's checking ~4 times a minute for updates, but maybe I'm misunderstanding something else? I can't say I fully understand what OVERPASS_TIME actually does, so maybe that's involved? Ultimately I am trying to get it to stay up to the minute and essentially be a clone of overpass-api.de where I am not rate limited.

wiktorn commented 3 years ago

Your reasoning is good. OVERPASS_UPDATE_SLEEP=15 indeed means that you'll check 4 times a minute if there are any updates available.

I'd guess, that overpass-api.de might use different replication server. I'm not sure how value for /api/augmented_diff_status is calculated what makes it hard, to verify - if the state file in your replication source has also the lag.

I've run following commands for a few times:

$ date -u ; curl 'https://planet.openstreetmap.org/replication/minute/state.txt' https://overpass-api.de/api/augmented_diff_status
Tue Mar  9 18:28:29 UTC 2021
#Tue Mar 09 18:27:50 UTC 2021
sequenceNumber=4448355
timestamp=2021-03-09T18\:27\:49Z
4464690
$ date -u ; curl 'https://planet.openstreetmap.org/replication/minute/state.txt' https://overpass-api.de/api/augmented_diff_status
Tue Mar  9 18:28:36 UTC 2021
#Tue Mar 09 18:27:50 UTC 2021
sequenceNumber=4448355
timestamp=2021-03-09T18\:27\:49Z
4464691
$ date -u ; curl 'https://planet.openstreetmap.org/replication/minute/state.txt' https://overpass-api.de/api/augmented_diff_status
Tue Mar  9 18:28:56 UTC 2021
#Tue Mar 09 18:28:53 UTC 2021
sequenceNumber=4448356
timestamp=2021-03-09T18\:28\:52Z
4464691

And you can see in the middle call that /api/augmented_diff_status bumped although state file still was pointing to the same file. It might indicate, that overpass-api.de has some better source of diff files.

Though the state file itself, looks quite fresh (looking on the timestamp) as well as data provided there.

As for OVERPASS_TIME indeed, this is a bit misleading, but OVERPASS_TIME is used to set the time for rate limiting.

mgd722 commented 3 years ago

Thanks for taking the time to look into this! Since there's nothing in my config that was obviously wrong, I'm going to close the ticket-- I'll poke around a bit regarding the possibility of a different replication server and report back if I find anything of note.