search

openaddresses / machine

Scripts for running OpenAddresses on a complete data set and publishing the results.
http://results.openaddresses.io/
ISC License
97 stars 36 forks source link

Slippy map rendering of individual sources as part of pull request #466

Open iandees opened 7 years ago

iandees commented 7 years ago

A recurring theme in https://github.com/openaddresses/machine/issues/440 and similar, it'd be nice to have slippy maps (a la the dot map on openddresses.io) alongside the raster-rendered previews that openaddresses-bot currently posts to PRs.

I'm not sure of the best way to implement this: maybe in addition to rendering the PNG, we could render a set of vector tiles to S3 (or straight from a DB) that have the points for a single source at a single commit and a template slippy map that loads said vector tiles given a template URL? That'd be a lot of stuff in S3, though.

migurski commented 7 years ago

I think this would be possible as part of an effort to store every point in a DB, rather than just metadata pointing to static files. Experimental but useful?

NelsonMinar commented 7 years ago

I like the idea of serving slippy maps / database access. But is it possible to build this as a fully standalone project downstream from machine? Ie, a separate codebase that consumes the CSV output files and produces database / slippy tiles / PNG images?

I see a couple of advantages. It'll keep the machine code simpler, allowing someone to contribute to one project or the other but not both. Also it's a good test that our CSV output is actually useful for real applications. Ie, that the important metadata is being published.

OTOH it may mean twice the work with managing AWS ops.

migurski commented 7 years ago

Only trick with separating is that it’d make it harder to loop this in to the pull request process. Maybe if we want this, it could be an additional commit hook on the same repo?

migurski commented 7 years ago

Started some light work on this in https://github.com/openaddresses/dotmaps.

migurski commented 7 years ago

After talking with Nelson over the weekend, it seems like the work I’ve started in the dotmaps repo may actually belong here, in the form of MBTiles uploads to S3 similar to the current preview PNGs and no live database. Some notes from Nelson on our conversation in this blog post.

migurski commented 7 years ago

Okay, we've now got MBTiles data quietly publishing alongside PNG previews for future pull requests. While that’s happening, I’ll need to figure out the right place to do UI work on this. It’ll probably need to be linked from job pages like this one.

migurski commented 7 years ago

We now have visible links to slippy map previews, visible on jobs pages like this one. The server they link to is running code from https://github.com/openaddresses/dotmaps so it’s not all the way integrated yet.