search

whosonfirst / go-whosonfirst-dist

Go package for working with Who's On First distributions
BSD 3-Clause "New" or "Revised" License
1 stars 1 forks source link

Feature request: frequent smaller DBs containing just date-range limited changes #6

Open simonw opened 6 years ago

simonw commented 6 years ago

The thing I'd really love to see is a weekly or monthly database that contains just the records that were updated in the past 30 days. This would be extremely useful for keeping my own copy (in this case it's a custom schema in MySQL which I populate from the Who's On First SQLite DB) up-to-date without having to pull the full 13GB file.

For example, https://dist.whosonfirst.org/sqlite/whosonfirst-data-changed-in-july-2018.db.bz2 would be a SQLite database with exactly the same format as the giant 13TB one but that only includes the geojson records that were updated at least once during the month of July 2018.

This could be weekly, monthly or something else, I don't really have a strong opinion.

simonw commented 5 years ago

Any chance we could get this done? Anything I can do to help?

thisisaaronland commented 5 years ago

There is a lot of work related to building distributions and doing indexing using ECS happening in a another project that we are hoping to build on (here) soon.

Once it has then it becomes much easier to think about variations on the build process. I don't have a firm ETA yet.

In the meantime if you want to keep the ball rolling then either spec-ing out the interfaces (command line, packages, etc.) and/or starting a branch of wof-dist-build would be great.

nvkelso commented 5 years ago

+1 for this.

stepps00 commented 5 years ago

Also +1-ing