Get Realtime People Mover data

[brendanbabb]

The muni is working on getting realtime data, but PeopleMover has had GPS data for 5+ years through the data through Bustracker. http://bustracker.muni.org/InfoPoint/

Can we scrape this for all the buses in less than 30 seconds? I estimate 32 buses running at any given time, We could grab their current info, or grab the info from each stop via the text interface: http://bustracker.muni.org/InfoPoint/noscript.aspx?route_id=2

Then we could make good proof of concept apps that the Muni could adapt like text your bus stop number to see if the bus is late? And feed into a bunch of existing apps like One Bus Away.

[thermokarst]

This would be awesome, count me in.

[NigelKibodeaux]

That sounds cool, @brendanbabb. Do bus stops have numbers on them? If so, I think it would be pretty easy to scrape the data from the site and send the user back the scheduled departure time and estimated departure time of that stop.

[brendanbabb]

That is a great idea. Bus stops do have numbers on them I am pretty sure. Will take a walk later to see how easy it is to see them. But if they don't have an easy one to read, we could write an easy GIS query that you could text the cross streets and do a query for the closest stops and their stop number.

Putting the GTFS data in to Github right after this.

[thermokarst]

Also, if stop id's are listed at the stops, do they correlate with the URI stop_id query param? (e.g. http://bustracker.muni.org/InfoPoint/departures.aspx?stopid=1509)

[brendanbabb]

Good question

[brendanbabb]

Here is the link for the GTFS data. It hasn't changed in a while and should change until next year. Should I upload it somewhere. http://gtfs.muni.org/People_Mover.gtfs.zip

It has lat and long data for all the bus stops. Should be doable to have an SMS based interface that you send cross streets too, it looks it up and translates to address, looks up lat and long and does a query for stops within 1/4 mile of there and if more than one stop, returns texts with multiple stop addresses and stop number, user replies with stopnumber, program looks up current buses and arrival times for that stop. If only one stop in area then user gets bus and arrival times for that stop.

[brendanbabb]

Haven't done field trip yet, but did determine that Bustracker ID is different than GTFS bus stop id.

L St. and 9th is http://bustracker.muni.org/InfoPoint/departures.aspx?stopid=2177 and 1172 in GTFS.

https://www.google.com/maps/@61.2131297,-149.903825,18z

[brendanbabb]

The Bus Stop ID is on the schedule frame and is from the GTFS file and People Mover literature and not in the Bustracker URL.

These photos are from 6th and K St ESE stop 1361 in Google and People Mover and 2307 in Bustracker http://bustracker.muni.org/InfoPoint/departures.aspx?stopid=2307

Another challenge is that the bus route number isn't listed just the route names.

[brendanbabb]

Additional idea, use IFTTT to say send me a text if my certain bus after 4pm is 10 minutes late. So when I get that I will head down from work to catch the bus. Or text me if it is more than 5 minutes behind etc..

[ralreegorganon]

I would be remiss to not point out that I wrote some real janky code a few years ago to pull all this info as a proof of concept at https://github.com/ralreegorganon/wheels-on-the-bus, and it still works just fine. I wouldn't suggest using it as a starting point for anything from the code side, but it gives you everything you need to pull data out of that site.

[thermokarst]

Thanks @ralreegorganon!

[brendanbabb]

Cool @ralreegorganon. Thanks a bunch.

[brendanbabb]

So one Minimum Viable Product I was thinking was text the bus stop number to a Twilio number, it looks up the Bustracker ID based on the bus stop number (1079 bus stops), and returns the text output from the URL as a text to you. This would be a long text example which would probably be 3 texts. http://bustracker.muni.org/InfoPoint/departures.aspx?stopid=2264

One step further, the program filters to only return the next time (first row) and sends as one text. Leverage @ralreegorganon code and some Ruby Twilio code from Code for America Balance. Hardest part might be mapping all the Google Bus Stop numbers to Bustracker stop numbers, and paying for the texts.

[brendanbabb]

@ralreegorganon is it easy to get a list of the BusTracker stop ids and the lat longs for all stops? The problem I am trying to solve is to have a table that maps People Mover stop ID (GTFS ID) to the Bustracker URL ID so given a GTFS ID I can just grab the right http://bustracker.muni.org/InfoPoint/departures.aspx?stopid=2264 and then parse that HTML and send back a text with the current arrival times. I am thinking I can use a GIS Query in Carto DB to find the closest stops for the lat longs of the GTFS stops with the Bustracker Lat Longs. Open to other ideas or just doing it for Route 3 say. I have Windows and am having trouble running the ruby code locally or on Heroku. Thanks for any help or tips.

[thermokarst]

@brendanbabb Why not just relate them by matching the intersection names between the two datasets?

[brendanbabb]

Matt - Realizing that is an easier way to do it and I did think of that before. That is the best solution and I think it would match really well. I think I secretly wanted to learn GIS distance queries, but short on time so can do that later. Also just got the code to run locally. The Wheels code is good, just Windows challenges on gems that is a problem.

[ralreegorganon]

@brendanbabb The code you'd need is already in the existing code, as it pulls that data as part of the overall dataset. If you wanted to just extract stops, something like the following would work (I just ripped out the extra code and put the stops in a set instead. You can modify the output as needed (e.g. output csv instead):

#!/usr/bin/env ruby

require 'nokogiri'
require 'open-uri'
require 'json'
require 'set'

doc = Nokogiri::HTML(open('http://bustracker.muni.org/InfoPoint/noscript.aspx'))
stops = Set.new 
routes = doc.css('.routeNameListEntry').map {|l| {:id => l['routeid'], :name => l.content}}
routes.each do |r|
  doc = Nokogiri::XML(open("http://bustracker.muni.org/InfoPoint/map/GetRouteXml.ashx?routeNumber=#{r[:id]}"))
  doc.css('stop').each do |s|
    stops.add({:id => s['html'], :name => s['label'], :lat => s['lat'], :lng => s['lng']})
  end
end
puts JSON.generate(stops.to_a.sort{|x,y| x[:id] <=> y[:id]})

[thermokarst]

@ralreegorganon: thanks, but the InfoPoint stop ID's are not the same as the People Mover System IDs that are physically printed at each bus stop (also the IDs used in Google Maps integration). @brendanbabb summarized in this comment: https://github.com/codeforanchorage/project-ideas/issues/4#issuecomment-62053943.

[ralreegorganon]

@thermokarst I realize that and already read the comment as well. I was responding to the following from @brendanbabb:

@ralreegorganon is it easy to get a list of the BusTracker stop ids and the lat longs for all stops?

[thermokarst]

@ralreegorganon: thanks for clarifying what you were responding to!

[brendanbabb]

Thanks @ralreegorganon, I was able to get CSV out. The stop names aren't completely consistent and I did matching by the Lat and Long and only have 17 out of 1000+ that don't match. Thanks everyone for the help.

[brendanbabb]

Also I will try to set up an actual git hub project with data and starter code later today.

[brendanbabb]

Got the go ahead from CfA for free Twilio credits if we wanted to try doing this for 1-2 months or costs less than $100 a month which would be about 6000 requests and responses per month. We can use my Twilio account for testing to begin with because it gets us better log information.

Andrew also referred me to this project: which is similar to the general library I would like to create for interfacing existing CartoDB and Google Maps API apps with SMS capabilities for people with texting but not smart phones. http://codeforsanfrancisco.org/blog/LocalFreeWeb/