osmosis export corrupted

[caleb87]

Not sure where the issue is here, but when running an osmosis export on the rails port database, sometimes it's successful and sometimes it gets corrupted. The full export is normally 8.6gb for North America, but sometimes when it exports it is only 6gb.

sudo osmosis --read-apidb-current host="localhost" database="openstreetmap" user="openstreetmap" password="openstreetmap" validateSchemaVersion="no" --write-pbf file="/var/www/html/export.osm.pbf"

It finishes without errors thrown, but it isn't a full export. If I run the command over again eventually it will finish correctly.

[tomhughes]

You realise this is not the osmosis bug tracker right?

[migurski]

Can you provide more detail, caleb87? Are you able to identify the differences between the too-small 6GB file and the expected 8.6GB one?

[caleb87]

I don't know much about the pbf files, but clearly they are compressed since when I tail the file, it just shows "3õGØÜÃÜl_t}*ùµ&¼Ðýìf·`Óý÷äìOÕ«[U×˳m?b÷g¾jos°ùÓ¥õïïÃ|m". I'm not sure what to do to differentiate them. I did another export, and the file size is 200mb different.

While I haven't confirmed, it looks like if the rails port is shut down it will export correctly. If it's running, it won't export correctly. There's no edits on the rails port happening, so it seems odd.

Would this be a rails bug or osmosis? Or is this expected?

[migurski]

Do you have the ability to localize any of the differences? For example: does it skip a particular geography, or a particular feature type, or IDs in some range?

[brettch]

You should be able to compare the contents of two PBF files using a command similar to this:

osmosis --read-pbf-fast <mygoodfile.pdf> --read-pbf-fast <mytruncatedfile.pbf> --derive-change --write-xml-change differences.osc

If you run that command (replacing the filenames as appropriate) you should end up with a differences.osc file describing the differences in XML format.