Minutely updater failing and not restarting with container

[rastilin]

The minutely downloader seems to be crashing and it doesn't seem like it restarts when the docker instance is restarted.

I've noticed the minutely updater can be unstable in our previous (non-docker) instance so I know that it can be prone to crashing; I've also seen it restart from a previous point inside a docker instance so I'm confident it's configured correctly normally.

Are there any files I should grab off the container that might help in figuring this out? We'll keep it around for debugging purposes to triangulate this issue.

[wiktorn]

Files of interest:

• /db/changes.log

And logs available from docker logs.

What did you use to download updates, when you're updating overpass instance? This image uses pyosmium-get-changes.

You can check if you have /app/bin/update_overpass_loop.sh process running within your instance and what it is doing.

[rastilin]

Sorry for the delay in getting the logs to you.

The /db/changes.log seems to change over at ID 3772208. Before then it runs as normal, but after that it looks like it gets stuck applying the same change over and over again.

DEBUG: Using given sequence ID 3772203
DEBUG: Starting download at ID 3772204 (max 100 MB)
DEBUG: Downloaded change 3772204. (102400 kB available in download buffer)
DEBUG: Downloaded change 3772205. (102400 kB available in download buffer)
DEBUG: Downloaded change 3772206. (102400 kB available in download buffer)
Empty version, skipping file
<?xml version='1.0' encoding='UTF-8'?>
<osm version="0.6" generator="libosmium/2.15.2">
</osm>
There are still some updates remaining
INFO: Using replication server at http://download.openstreetmap.fr/replication/oceania/australia/minute/
DEBUG: Using given sequence ID 3772206
DEBUG: Starting download at ID 3772207 (max 100 MB)
DEBUG: Downloaded change 3772207. (102400 kB available in download buffer)
DEBUG: Downloaded change 3772208. (102400 kB available in download buffer)
/db/diffs/changes.osm exists. Trying to apply again.
XML parsing error at line 3, column 0: no element found
/db/diffs/changes.osm exists. Trying to apply again.
XML parsing error at line 3, column 0: no element found
/db/diffs/changes.osm exists. Trying to apply again.
XML parsing error at line 3, column 0: no element found

The rest of the log just repeats the same two lines until the end.

Doing a docker logs with the docker container name just repeats the same log, unless there's another file I'm not aware of.

We're just using what came with the image, so we would have pyosmium-get-changes as well.

A run of "docker container top" shows that "update_overpass_loop.sh" is running inside the container. Although if the changes.osm file is invalid it might not be doing anything.

Does this help?

[wiktorn]

Which version of the container do you use?

As I see., your logs are quite old, http://download.openstreetmap.fr/replication/oceania/australia/minute/003/772/ is no longer available to check if the problem was with the file itself.

Recent version (bcd0238eb4a9ab552469c8cb9159ec7236cabf5b) checks if the file contains anything to apply and if not skips it. So what your logs describe, should not happen with recent images, though it was the case earlier.

If you used pyosmium-get-changes and didn't check if there is anything to apply in what was downloaded, you might have encounter the same problem with your non-docker version.

Please try pulling recent version if this is the case.

I do not tag versions of the image separately from Overpass API version as of now.

[rastilin]

I think that's probably correct. I'm not sure which version that container was from, but the docker images are either Overpass version "0.7.55.7" or "0.7.55.6" if that helps.

We've pulled a new container to to run speeding against and it's been working perfectly for at least a month now.

[wiktorn]

Ok, closing the issue then.