search

Open-Historical-Map-Labs / openhistoricalmap-docker

A dockerized configuration to create the OpenHistoricalMap backend
9 stars 3 forks source link

Import latest Planet into local database #28

Open danrademacher opened 4 years ago

danrademacher commented 4 years ago

I think this will supersede further work on #24:

With the recent outage, we should prioritize putting this database at the "main" AWS site, http://openhistoricalmap-1936150974.us-east-1.elb.amazonaws.com/, and then setting up #27 , so we can repoint the vector tile server at this database.

With http://planet.openhistoricalmap.org/planet/ back online at last, I downloaded a 2020-04-13 Planet, and we can use this latest planet to populate local database on our staging site.

that is here: https://www.dropbox.com/s/7615vc6jjzmzan9/ohm_planet_2020-04-13.osm.bz2?dl=0

Also here for the moment http://planet.openhistoricalmap.org/planet/

Obviously, this means we don't need to try to recreate the data from old planet + diffs.

cc @jeffreyameyer

geohacker commented 4 years ago

@danrademacher great that we have recent planet dump to work with. Applying diffs incrementally in a reliable way is a bit tricky.

The staging setup needs more work to replicate our current pipeline. Thinking that importing the latest planet shouldn't be blocked by that. Hope to resume that work shortly. As a shortcut to move forward with importing latest planet dump, I would suggest this workflow:

  1. Use osm-seed. Either setup osm-seed locally or on the cloud
  2. osm-seed comes with a task/script that can import an OSM planet file into the API database
  3. We can then take a dump of this database and import directly to the main site.

I'm currently standing up osm-seed locally to try this out. I will also check with Ruben on our team if he can do this quicker than me and give us a postgresql dump.

geohacker commented 4 years ago

My attempts to restore this locally failed obviously because of disk space limitations. The unarchived planet is 10G+. Today I will work with Ruben on standing up an osm-seed instance in the cloud.