search

mapbox / osm-edit-report-deprecated

Report editing activity on OpenStreetMap
https://www.mapbox.com/osm-edit-report/
ISC License
5 stars 7 forks source link

Stickshift? #9

Closed lxbarth closed 8 years ago

lxbarth commented 9 years ago

@Rub21 @tmcw - longer term, how could we use Stickshift to power OSM edit reports? https://github.com/mapbox/stickshift

cc @aaronlidman @geohacker @planemad

tmcw commented 9 years ago

Huh, this could be very interesting. Really what we need is something of a hybrid @miccolis @ericfischer monster that could build something like:

Beyond this, I have some UI tweaks to make to make more advanced charts possible but for the most part this is a matter of getting the data in place.

yhahn commented 9 years ago

Following. cc @rclark

I'd love to see an OSM => S3 changesets mirror (would love to use this for other stacks rather than hitting OSM directly), and whenever an object is pushed to this s3 bucket it could trigger s3 notifications or lambda for redshift processing.

ianshward commented 9 years ago

@yhahn @Rub21 @lxbarth

lxbarth commented 9 years ago

@ianshward - at a minimum we're talking about mirroring the replication feeds in these directories:

http://planet.osm.org/replication/

How many changeset files are published each day?

  • One per minute, hour, day for data updates
  • One per minute for changesets

Is there any notification / feed / etc. for when a new changeset file is published currently?

There's no notification when replication files are updated, you'd poll them.

whenever an object is pushed to this s3 bucket it could trigger s3 notifications or lambda for redshift processing.

This is why the s3 notifications are interesting.

@lxbarth - would it be useful to be able to do one-off / adhoc queries of the last 24 to 48 hours of data, on a daily basis?

No, at least not within the scope of a replication mirror.

@ianshward - could we run this as a public service for anyone looking for an on-AWS mirror? This is interesting because I'd like to be able to share any code we produce supporting the data team as openly as possible. If we build on proprietary services sharing our code becomes less useful.

rclark commented 9 years ago

I have code already written to poll http://planet.osm.org/replication/ for changes in dynamosm. When it finds new minutely files there it creates SNS notifications that a file is ready to be downloaded.

If this is broadly useful I can pull that code out of dynamosm into a standalone service. Systems could listen to this SNS in order to grab files from planet.osm.org when they're ready, or this service could fuel a full-blown mirror.

rclark commented 9 years ago

Our OSM mirror is alive and mirroring changesets onto S3. I got it running to see if it works, but we don't need to keep it running if no one is ready to consume it yet.

If we build on proprietary services sharing our code becomes less useful.

The changesets on S3 are currently publicly accessible for download, but the notifications are simply pushed to SNS topics that require adequate permission to subscribe to.

ianshward commented 9 years ago

@rclark nice!

@lxbarth @rclark @Rub21 if adhoc querying is not interesting, then I wonder whether it'd make sense to run a service which does something like:

I'm not sure if it's interesting or even practical that the unloaded data be all the data, but, maybe the parts that we wrap in such a service are already open source or could be written as such, and there'd be a facility to unload data related to only a specified list of users.

I don't know enough about the data in .osc to know if possible, but could you also create any interesting vector tiles out of it?

ianshward commented 9 years ago

Ahh, I left out, that once you have days, months or years of these JSON files in S3, you could then suck them into Redshift w/ something like metrics-warehouse, and if you wanted, do adhoc querying from StickShift that way. This assumes you're happy with whatever attributes you've included in the JSON files.

rclark commented 9 years ago

could you also create any interesting vector tiles out of it?

@ianshward sounds like you're coming close to describing what dynamosm is intended to do -- transform this stream of .osc into a stream of geojson, basically.

You may or may not have to go that far to extract metrics from osc files, it all depends on what exactly the metrics are that you want to collect.

lxbarth commented 8 years ago

Stale.