search

Streets-Data-Collaborative / OpenStreetCam-GeoParsing-Tool

Create a tool that, given a city, give can pull each OpenStreetCam (OSM) Track file No. associated with that city.
Apache License 2.0
7 stars 2 forks source link

Housekeeping #8

Open patwater opened 6 years ago

patwater commented 6 years ago

Not urgent though at some point would be helpful to update the readme explaining at a high level what getAllTracts.py, getIntersection.py, etc. do. Also best practices would not be to have the code in the top level directory but rather something like queryScipts or just scripts.

Also BONUS POINTS: @charlie-moffett have you checked out the Shared Streets spec? Would be curious to your thoughts on how that might sync up with this pipeline.

charlie-moffett commented 6 years ago

@patwater I was actually just reading about SharedStreets. I could see an intermediary function added to this pipeline that converts the OSM sequence_ids to SharedStreets identifiers, or even just adding to getTracksheet() such that both OSM and SharedStreets IDs for each street segment in the given city are stored in the 'SQUID Uploader (Responses)' gSheet. Is one necessarily more valuable than the other at this time or for the foreseeable future?

I've also updated the readme with info on the scripts and moved the scripts out of the top level directory.

patwater commented 6 years ago

Awesome! I would defer to @dmarulli about that technical implementation decision though storing both sounds prudent. Perhaps we can also get @kpwebb to weigh in at some point about leveraging shared streets with SQUID :)

dmarulli commented 6 years ago

@charlie-moffett - interesting thoughts. Some technical details:

OSC's sequence_ids and the SSRIDs (SharedStreets Reference IDs) actually track different things. OSC's sequence_ids also called Track file No.s reference an entire "trip" whereas the SSRIDs are at the level of a street segment--see here for additional details on the SharedStreets side of things. So each sequence_id can be associated with many SSRIDs.

That said, farther along the pipeline SSRIDs could be very useful. Right now our SQUID maps display point data extracted from OSM using sequence_ids, but these ground-truthed street quality measurements could be aggregated from the GPS coordinate-level to the street segment-level and tagged using the existing SharedStreets Reference ID standard. Coordinating to a standard has a lot of benefits, so this is definitely a foreseeable logical endpoint of this work.

charlie-moffett commented 6 years ago

@patwater I'm adding my presentation slides below:

project showcase.pptx

patwater commented 5 years ago

@charlie-moffett hope you've been well! Was huddling with @vr00n and he was looking for a getting started guide for running these tools. That'd be helpful for the broader open data community.

We were also thinking it'd be cool to add more OSC utility tools. Specifically thinking of a way to easily get stats on availability of OSC data.

We need a quick way to tell us how much OSC data a city, county, or state has and how old it is. Ex: What % of LA's streets have OSC imagery and dates for each street.

charlie-moffett commented 5 years ago

Happy to put something together on the readme! Travelling at the moment but can get to it in a week or so. Does that timing work?

On Thu, Oct 4, 2018, 4:44 AM Patrick Atwater notifications@github.com wrote:

@charlie-moffett https://github.com/charlie-moffett hope you've been well! Was huddling with @vr00n https://github.com/vr00n and he was looking for a getting started guide for running these tools. That'd be helpful for the broader open data community.

We were also thinking it'd be cool to add more OSC utility tools. Specifically thinking of a way to easily get stats on availability of OSC data.

We need a quick way to tell us how much OSC data a city, county, or state has and how old it is. Ex: What % of LA's streets have OSC imagery and dates for each street.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Streets-Data-Collaborative/OpenStreetCam-GeoParsing-Tool/issues/8#issuecomment-426773825, or mute the thread https://github.com/notifications/unsubscribe-auth/AYdys4YINCKc7VnTdViPpUVgtp9mxFoOks5uhROogaJpZM4RexIO .

patwater commented 5 years ago

Yessir! That'd be great and please ping @vr00n @dmarulli when ready for review. Thanks much

charlie-moffett commented 5 years ago

@dmarulli could you please email me a copy of our convo from the ARGO Labs slack space? I remember you explaining at least in part how you deployed the scripts in this repo for a demo and would like to be able to put together a getting started guide. I've since been deactivated from that slack space - thanks in advance!

dmarulli commented 5 years ago

Hmm, not seeing our convo @charlie-moffett. Perhaps it was removed by Slack?

That said, after reviewing the files, it looks like this one has all the necessary functions. (The other files seem redundant.) The docstrings and comments for each function in this file look pretty descriptive as well.

So as far as instructions for using these scripts in the way they were intended goes, here's the high-level description:

After cloning the repo, one would:

  1. Set up the appropriate Google assets.
  2. Replace the placeholder values for the credentials for these assets.
  3. Run the getTracksheet(city, state, country) function for a particular city.

After the function finishes executing--after the web-scraping wraps up--the Google Sheet associated with the credentials in the script should be updated with the OSC tracks from the intended city.

patwater commented 5 years ago

@charlie-moffett that get you what you need for the readme?

charlie-moffett commented 5 years ago

@patwater yes it does, thanks @dmarulli !

patwater commented 5 years ago

Awesome! Please let us know when the ReadMe update is ready :) Hope all is well with you btw! LMK if you're ever in LA 1t1d