search

openaddresses / openaddresses-ops

Issues-only repo for discussion of operational considerations for OA
6 stars 5 forks source link

Accepting data mirrors? #29

Open markwoodbury opened 3 years ago

markwoodbury commented 3 years ago

Hi team,

Following #28 and the mentions both on this issue tracker and your site, I was reaching out to provide you with an offer of mirroring for your files. I currently do the same with the OpenStreetMap Planet file (mirrored here and tracked here).

As a proof of concept, I've setup this mirror which has a copy of the last published version of the global collection from your site.

If you would find this useful, I would appreciate your confirmation of the top files that would be beneficial to be mirrored, along with confirmation of the associated license to ensure I have the data configured correctly.

Cheers, Mark

ingalls commented 3 years ago

@markwoodbury We would gladly accept data mirrors for the project. We don't yet have a way of surfacing them via the site, but I would be happy to explore a UI addition to do so.

You've got it! The most important file is the global zip. Each source is individually licensed, so unfortunately we can't have a single license for the whole file.

markwoodbury commented 3 years ago

Thanks for the confirmation. I've updated the script to automate pulling the new files once they're updated on your site and then update the mirror page with the new link. Would like to test it for around a week or so before you send any real traffic to make sure it's working ok. I'll confirm back once it's done at least 2 updates automatically.

Regarding the license - perhaps it would be best for me to link the mirror page to a page that explains the licensing from your site? Does such a page exist? I see references to the license files within the downloads themselves on the results page, but is there a more detailed page to use instead perhaps?

iandees commented 3 years ago

It's probably best to link to https://openaddresses.io/attribution/ for now.

markwoodbury commented 3 years ago

Thanks for the link - I've updated the script outputs, which are in progress of running. Will update later in the week once everything looks good.

markwoodbury commented 3 years ago

I've updated the script to handle the cache issues, made some minor tweaks (e.g. to present human readable size and more detailed updated time). Should be good to go from my side, so would welcome you to test and confirm.