Remove redundant data, reduce osm size, boost import speed

[x1alpha76]

This is part request for comment, part to help others who have issue importing an osm file due to its large size.

I was having issue importing australia-latest.osm (http://download.geofabrik.de/australia-oceania/australia.html) which is approximately 7.7gig. After having a look at the osm2pgrouting interals i couldn't see it actually using the following attributes from the osm file: version, timestamp, changeset, uid & user.

So i ran the following sed command on the osm file to remove these attributes, resulting a ~3.6gig file. sed -r "s/version=\"[0-9]+\" timestamp=\"[^\"]+\" changeset=\"[0-9]+\" uid=\"[0-9]+\" user=\"[^\"]+\"//g" australia-latest.osm -i.org

I'm sure there is a better/faster way to do this, but it worked for me.

Osm2pgrouting then ran far quicker (~45minutes vs 3+ hours and not complete) for me as it didn't need as much memory and more importantly didn't need to use swap.

Does anyone see any issue with doing this? If not maybe it is something that could go into the readme?

Also, it might be helpful to others to have some estimate on how much memory is required to import a file of a particular size, it seems to be around 7x the file size. iirc it used about 28gig when i ran osm2pgrouting with the 3.6gig file.

[cvvergara]

@x1alpha76 I wonder if you could compare with the version that is on the "develop" branch

[cvvergara]

@x1alpha76 it certainly is a nice trick

[cvvergara]

@x1alpha76 https://github.com/pgRouting/osm2pgrouting/wiki/Documentation-for-osm2pgrouting-v2.2#tip