Use HOT Export Tool for download

[thomas-maschler]

Lets look into ways, on how to use the new HOT Export Tool. We shouldn't just provide logging roads map for the project but a complete road map for each country. Export should be as Shapefile for easy use. We will need to pick appropriate tags for the data layer

http://export.hotosm.org/

[thomas-maschler]

This works pretty sweet.

Created a preliminary preset files for roads in general and created a first export for testing.

Exports are limited to a certain extent, but we can do it in pieces and merge files afterwards. The API is not documented but I should be able to call the rerun exports function externally. I will do some further testing and set up a layer, containing all roads for the Congo Basin, which I will update daily.

[thomas-maschler]

Has some an idea how I can get add the last editing user and last editing date into a preset file?

[thomas-maschler]

@kriscarle

to start an export I have to make a call like this

curl "http://export.hotosm.org/api/rerun?job_uid=b91bf724-7d86-482d-a372-6203f23cf342&_=1444565259002" -H "Cookie: osm_export_tool2_sessionid=bsmf15mubbzoydma2zmzki1i8npymvds; csrftoken=05um5L7yxb8jn65RusWuhuaAfXYGgv9k"

Do you know you to best acquire a session ID from OSM Export Tool and a token from OSM using python?

[kriscarle]

Unless the OSM Export Tool is designed to support external calls to their backend, I don't think this will work. They would need to provide a permanent API key that essentially bypasses the need to do the user login to OSM. The way oauth works, the user explicitly grants access for OSM Export Tool to use their OSM account, so any attempt on our part to login the user from a different app won't work.

I think due to the size of the dataset, we'll need to maintain our own copy of the roads database and sync over nightly changes. We can use http://wiki.openstreetmap.org/wiki/Osmosis and the Africa extract from http://download.geofabrik.de/africa.html Then we can have a nightly script apply updates from http://download.geofabrik.de/africa-updates/ Then we use a script on the server to extract the data by country and turn it into shapefiles.

It will take a little bit of work to put this together. It is almost big enough of a project to warrant setting up something like africa-roads.org and having it be its own site? There are a few OSM extract sites out there, but none that manage daily extracts of entire African countries. HOT's tool does appear to be the best one out there, but the on-demand approach probably means it needs to limit the size of the processing jobs.

[bjohare]

Hi Kris, Thomas.. we (HOT) can provide you with a API Token which you can use to make calls to the HOT Export Tool backend. We can also increase the size of the allowed area to extract up to around 5,000,000 sq km (720MB extract in central Africa ish) see http://hot.geoweb.io/en/exports/7b303360-7ce0-4f01-ba4f-70b3e2e1fa9f/. We could probably go higher with this but I've noticed that Overpass API, which we use on the backend, sometimes aborts the reads for large queries so your mileage may vary. We need to look into this a bit more and are still getting to know the limitations of the new system. I've started to add API documentation to the export tool wiki at https://github.com/hotosm/osm-export-tool2/wiki/HOT-Export-API. If you are doing nightly extracts it would be good to set them as 'published=false' so that they will be purged after 48 hours and not hang around on the system.

Maybe the best way to request a key for the moment is to add it to the issue tracker for the tool and mark it with the label 'API TOKEN REQUEST'.

[thomas-maschler]

This sounds great.

Thanks Brian

[thomas-maschler]

moved to hotosm/osm-export-tool2#96

[thomas-maschler]

Just to follow up. Got a token from Brian and are able to rerun exports using the API. Nice! Will schedule automatic downloads and integrate data in our various data GFW platforms. I will also provide a stable download link for a merged shapefile will all roads for the six countries.