search

whosonfirst / whosonfirst-sources

Where things come from in Who's On First.
Other
21 stars 13 forks source link

Integration of own data into the WhosOnFirst-Data #195

Open dawidl11 opened 2 years ago

dawidl11 commented 2 years ago

Hi there,

I have a question regarding the integration of own data into the WhosOnFirst-Data development process.

What do I have? Administrative boundaries in Germany accessible under a link (zipped shapefile with names and geometries). The source data can also be adapted to another format and stored locally.

What do I need? "whosonfirst-data-admin-latest.db.bz2" file with administrative boundaries for the whole world, but for Germany with my own administrative boundaries.

Question: Have you perhaps already developed code or a solution for such use cases (integrating your own data to the "whosonfirst-data-admin-latest.db.bz2" file)?

Many thanks in advance for the answers! Dawid

stepps00 commented 2 years ago

Hey @dawidl11 - the Geocode Earth team hosts Who's On First data distributions here: https://geocode.earth/data/whosonfirst/ Administrative boundaries for the entire world are hosted here: https://geocode.earth/data/whosonfirst/combined/

It sounds like you'd like to download the combined distribution, and swap the German data for a local dataset of your own.. is that correct?

dawidl11 commented 2 years ago

Thank you @stepps00 for the hints.

Yes exactly, the goal is to get a combined "whosonfirst-data-admin-latest.db.bz2". That is, to replace the German data with a local dataset and leave the data for the rest of the countries as it is original in the WhosOnFirst data. Have you maybe already developed a solution/code for such use cases?

stepps00 commented 2 years ago

Who's On First has not developed anything like that.. but pinging @missinglink who may have some tips for you.

missinglink commented 2 years ago

Hi again, once you've converted your data to the same format as WOF you can use our bundling tools to package them up as a SQLite database. . https://github.com/pelias/wof

For a single file it would look like this:

cat 101914243.geojson | wof sqlite import --rm wof.db

and if you have a bunch of features laid out in the same format as one of the whosonfirst-admin repos you can replace the cat command with wof fs export or wof git export to import them all.

missinglink commented 2 years ago

The global file is a concatenation of the country files, so while it's not the fastest running solution you can use wof sqlite export to dump all the features from our hosted global database, pipe to grep -v whosonfirst-data-admin-de to remove any from Germany and pipe to wof sqlite import to create a new database.

This new database will be all of WOF sans Germany, you then import your new data as above (avoiding the --rm flag) and you're all set.

note: I've written this on my phone at 11pm so don't expect my commands to work verbatim 😂

missinglink commented 2 years ago

Out of curiosity can you share a link to the Germany data?

dawidl11 commented 2 years ago

Hello @missinglink again 😊, thank you a lot for your answers. Each of the information is very valuable for me. We will try it out. As for the data, we need the latest administrative units from BKG: https://gdz.bkg.bund.de/index.php/default/digitale-geodaten/verwaltungsgebiete.html . Currently the data in WOF file are bit outdated.