search

zerebubuth / openstreetmap-changeset-replication

Simple Ruby script to replicate changes to changeset metadata to a stream.
7 stars 0 forks source link

Empty changeset diffs #2

Open pnorman opened 9 years ago

pnorman commented 9 years ago

http://planet.openstreetmap.org/replication/changesets/001/507/867.osm.gz is a zero-length file

This is the first one I've encountered. If it corresponded to a time with zero changeset differences, I'd have expected at least <osm />

Cross-ref ToeBee/ChangesetMD#17

ToeBee commented 9 years ago

Generating an invalid gzip file on the server is definitely a problem. Makes it impossible to tell if it was just a bad download or an empty diff file when trying to consume the diffs.

zerebubuth commented 9 years ago

This has me very confused - I can't see where in the code it's possible for it to create an invalid or zero-size file, so I've added some sanity checking in commit b6450a686df93430c6dad4b04b6d0a94ef8ab5f6 and hopefully that will provide a clue to what's going wrong.

ToeBee commented 9 years ago

Well since this has only ever happened once, as far as we can tell at least, maybe it was just glitch. Wasn't there a server problem with replication in general a little while ago? Maybe the process got killed mid-stream or something.

zerebubuth commented 9 years ago

It's unusual, sure. The reason I'd like to track it down is that I thought I'd written it in such a way as to make zero length files impossible.

According to my mental model, what should be happening is:

  1. The changeset XML is generated in memory from the database. Any failure here should result in an exception and no output.
  2. The XML and state files are written to /tmp. Any failure here should result in an exception leaving garbage in /tmp, but no visible changes on planet.osm.org.
  3. The XML and state files are moved into place, which should be an atomic operation. Any failure here might leave an unwanted file, but it shouldn't be empty or garbage.

Clearly I'm getting at least one assumption wrong... the question is: which one(s)?