search

CUTR-at-USF / bullrunner-gtfs-realtime-generator

Desktop application that retrieves AVL data from the USF Bull Runner's AVL system and produces Trip Updates and Vehicle Positions files in GTFS-realtime format.
Other
2 stars 3 forks source link

Update Syncromatics API, add per-vehicle timestamp, remove TripUpdates #24

Closed minhhpham closed 6 years ago

minhhpham commented 6 years ago
  1. Change data API
  2. Create API key file not included in source code
  3. Update GTFS file with new bullrunner webpage link and new route (Lot 18 Express)
  4. Update README
CLAassistant commented 6 years ago

CLA assistant check
All committers have signed the CLA.

minhhpham commented 6 years ago

I revised all the points above and pushed a new commit.

barbeau commented 6 years ago

Also, I noticed that the new Bull Runner GTFS has the new Lot 18 Express route in routes.txt but it's not represented in trips.txt or stop_times.txt or shapes.txt. If we add Lot 18 Express to the data we need to add the necessary data in those files too, or remove it from the GTFS data at this point.

barbeau commented 6 years ago

If you run the program without having the key.txt file in the directory, you currently get this:

Exception in thread "main" java.io.FileNotFoundException: ..\key.txt (The system cannot find the file specified)
    at java.io.FileInputStream.open0(Native Method)
    at java.io.FileInputStream.open(FileInputStream.java:195)
    at java.io.FileInputStream.<init>(FileInputStream.java:138)
    at java.io.FileInputStream.<init>(FileInputStream.java:93)
    at java.io.FileReader.<init>(FileReader.java:58)
    at edu.usf.cutr.gtfs_realtime.bullrunner.GtfsRealtimeTripUpdatesProducerDemoMain.run(GtfsRealtimeTripUpdatesProducerDemoMain.java:127)
    at edu.usf.cutr.gtfs_realtime.bullrunner.GtfsRealtimeTripUpdatesProducerDemoMain.main(GtfsRealtimeTripUpdatesProducerDemoMain.java:68)

Could you please update with a more user-friendly message saying that you need to create a file that contains an API key (same as README)? I'd suggest just catching the java.io.FileNotFoundException from line 127 in GtfsRealtimeTripUpdatesProducerDemoMain.java and then exiting the program.

barbeau commented 6 years ago

Actually, I get the same exception java.io.FileNotFoundException even if I do have the key.txt file in the root directory when trying to run the program from within IntelliJ. The instructions to run the program should work from whatever directory the JAR file is run from - right now I believe the directions work if you switch to the /target directory because the key.txt file is one directory up in the hierarchy - but we shouldn't assume this. Could you fix this and just read it from the current directory?

barbeau commented 6 years ago

I'm getting a similar error when trying to read Bull Runner GTFS data files:

java.io.FileNotFoundException: ..\bullrunner-gtfs\trips.txt (The system cannot find the path specified)
    at java.io.FileInputStream.open0(Native Method)
    at java.io.FileInputStream.open(FileInputStream.java:195)
    at java.io.FileInputStream.<init>(FileInputStream.java:138)
    at java.io.FileInputStream.<init>(FileInputStream.java:93)
    at java.io.FileReader.<init>(FileReader.java:58)
    at edu.usf.cutr.gtfs_realtime.bullrunner.BullRunnerConfigExtract.generateTripMap(BullRunnerConfigExtract.java:100)
    at edu.usf.cutr.gtfs_realtime.bullrunner.GtfsRealtimeProviderImpl.start(GtfsRealtimeProviderImpl.java:197)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.onebusaway.guice.jsr250.JSR250Module$ObjectAndMethod.execute(JSR250Module.java:139)
    at org.onebusaway.guice.jsr250.JSR250Module$LifecycleServiceImpl.start(JSR250Module.java:197)
    at edu.usf.cutr.gtfs_realtime.bullrunner.GtfsRealtimeTripUpdatesProducerDemoMain.run(GtfsRealtimeTripUpdatesProducerDemoMain.java:163)
    at edu.usf.cutr.gtfs_realtime.bullrunner.GtfsRealtimeTripUpdatesProducerDemoMain.main(GtfsRealtimeTripUpdatesProducerDemoMain.java:68)

The solution for this and the key file should just be to remove ../ before the folder/file name.

barbeau commented 6 years ago

We should also cap bearing and speed at 1 or 2 decimal places - I'm seeing exponential notation for bearing in testing, which we can't include:

entity {
  id: "1"
  vehicle {
    trip {
      route_id: "A"
    }
    position {
      latitude: 28.065973
      longitude: -82.41317
      bearing: 1.0374368E-9
      speed: 15.0
    }
    timestamp: 1536091036
    vehicle {
      id: "491"
      label: "1534"
    }
    occupancy_status: MANY_SEATS_AVAILABLE
  }
}
minhhpham commented 6 years ago

I added new commit fixing all the above comments.

barbeau commented 6 years ago

For GTFS routes.txt, let's change route_long_name of MSC Express to:

barbeau commented 6 years ago

@minhhpham Could you also run the new Bull Runner GTFS though FeedValidator and ScheduleViewer as listed here to make sure it doesn't flag any problems and that the shapes look good on the map, if you haven't already?

http://gtfs.org/testing/

barbeau commented 6 years ago

@minhhpham Thanks for making the changes. I've flagged a few more things above.

Also, when I try to run the application right now from IntelliJ using these command-line options:

--tripUpdatesUrl=http://localhost:8088/trip-updates --vehiclePositionsUrl=http://localhost:8088/vehicle-positions

I'm getting this exception:

2018-09-10 12:08:49,009 INFO  [Server.java:284] : jetty-9.0.5.v20130815
2018-09-10 12:08:49,178 INFO  [AbstractConnector.java:265] : Started ServerConnector@1198b989{HTTP/1.1}{0.0.0.0:8088}
2018-09-10 12:08:50,237 INFO  [GtfsRealtimeProviderImpl.java:170] : starting GTFS-realtime service
2018-09-10 12:08:50,241 INFO  [GtfsRealtimeProviderImpl.java:386] : refreshing vehicles
2018-09-10 12:08:51,334 WARN  [GtfsRealtimeProviderImpl.java:390] : Error in vehicle refresh task
org.json.JSONException: JSONArray[1] not found.
    at org.json.JSONArray.get(JSONArray.java:234)
    at org.json.JSONArray.getJSONObject(JSONArray.java:326)
    at edu.usf.cutr.gtfs_realtime.bullrunner.GtfsRealtimeProviderImpl.downloadVehicleDetails_C(GtfsRealtimeProviderImpl.java:359)
    at edu.usf.cutr.gtfs_realtime.bullrunner.GtfsRealtimeProviderImpl.refreshTripVehicle(GtfsRealtimeProviderImpl.java:210)
    at edu.usf.cutr.gtfs_realtime.bullrunner.GtfsRealtimeProviderImpl.access$200(GtfsRealtimeProviderImpl.java:63)
    at edu.usf.cutr.gtfs_realtime.bullrunner.GtfsRealtimeProviderImpl$VehiclesRefreshTask.run(GtfsRealtimeProviderImpl.java:387)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
barbeau commented 6 years ago

@minhhpham One other item I spotted in the GTFS for the MSC Express in stop_times.txt - the trips in seem to last 6 minutes long. In other words, the vehicle would complete the loop in 6 minutes (see below). This doesn't seem realistic to me. Note that frequencies.txt headway_secs is the amount of time between vehicles visiting a specific stop - if a rider is standing at a stop, it's the typical time elapsed from when one vehicle departs and the next arrives. If there is only one vehicle on the route then the time from trip beginning to trip end in stop_times.txt is equal to headway_secs. If there are X vehicles running a route, then headway_secs = time_to_complete_trip / X.

16,08:30:00,08:30:00,414,1
16,08:30:36,08:30:36,801,2
16,08:30:52,08:30:52,803,3
16,08:31:51,08:31:51,807,4
16,08:32:38,08:32:38,813,5
16,08:32:57,08:32:57,815,6
16,08:33:23,08:33:23,821,7
16,08:34:19,08:34:19,827,8
16,08:35:06,08:35:06,829,9
16,08:35:36,08:35:36,411,10
16,08:36:10,08:36:10,401,11
17,15:30:00,15:30:00,414,1
17,15:30:36,15:30:36,801,2
17,15:30:52,15:30:52,803,3
17,15:31:51,15:31:51,807,4
17,15:32:38,15:32:38,813,5
17,15:32:57,15:32:57,815,6
17,15:33:23,15:33:23,821,7
17,15:34:19,15:34:19,827,8
17,15:35:06,15:35:06,829,9
17,15:35:36,15:35:36,411,10
17,15:36:10,15:36:10,401,11
18,08:30:00,08:30:00,414,1
18,08:30:36,08:30:36,801,2
18,08:30:52,08:30:52,803,3
18,08:31:51,08:31:51,807,4
18,08:32:38,08:32:38,813,5
18,08:32:57,08:32:57,815,6
18,08:33:23,08:33:23,821,7
18,08:34:19,08:34:19,827,8
18,08:35:06,08:35:06,829,9
18,08:35:36,08:35:36,411,10
18,08:36:10,08:36:10,401,11
19,15:30:00,15:30:00,414,1
19,15:30:36,15:30:36,801,2
19,15:30:52,15:30:52,803,3
19,15:31:51,15:31:51,807,4
19,15:32:38,15:32:38,813,5
19,15:32:57,15:32:57,815,6
19,15:33:23,15:33:23,821,7
19,15:34:19,15:34:19,827,8
19,15:35:06,15:35:06,829,9
19,15:35:36,15:35:36,411,10
19,15:36:10,15:36:10,401,11
minhhpham commented 6 years ago

@minhhpham Thanks for making the changes. I've flagged a few more things above.

Also, when I try to run the application right now from IntelliJ using these command-line options:

--tripUpdatesUrl=http://localhost:8088/trip-updates --vehiclePositionsUrl=http://localhost:8088/vehicle-positions

I'm getting this exception:

2018-09-10 12:08:49,009 INFO  [Server.java:284] : jetty-9.0.5.v20130815
2018-09-10 12:08:49,178 INFO  [AbstractConnector.java:265] : Started ServerConnector@1198b989{HTTP/1.1}{0.0.0.0:8088}
2018-09-10 12:08:50,237 INFO  [GtfsRealtimeProviderImpl.java:170] : starting GTFS-realtime service
2018-09-10 12:08:50,241 INFO  [GtfsRealtimeProviderImpl.java:386] : refreshing vehicles
2018-09-10 12:08:51,334 WARN  [GtfsRealtimeProviderImpl.java:390] : Error in vehicle refresh task
org.json.JSONException: JSONArray[1] not found.
  at org.json.JSONArray.get(JSONArray.java:234)
  at org.json.JSONArray.getJSONObject(JSONArray.java:326)
  at edu.usf.cutr.gtfs_realtime.bullrunner.GtfsRealtimeProviderImpl.downloadVehicleDetails_C(GtfsRealtimeProviderImpl.java:359)
  at edu.usf.cutr.gtfs_realtime.bullrunner.GtfsRealtimeProviderImpl.refreshTripVehicle(GtfsRealtimeProviderImpl.java:210)
  at edu.usf.cutr.gtfs_realtime.bullrunner.GtfsRealtimeProviderImpl.access$200(GtfsRealtimeProviderImpl.java:63)
  at edu.usf.cutr.gtfs_realtime.bullrunner.GtfsRealtimeProviderImpl$VehiclesRefreshTask.run(GtfsRealtimeProviderImpl.java:387)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)

not sure why this happened. It ran fine on my IntelliJ.

barbeau commented 6 years ago

not sure why this happened. It ran fine on my IntelliJ.

I'm guessing it was related to the response that was returned from the Syncromatics API at that time (just before noon, IIRC). The line of the exception points to:

String pattern_name = response_pattern.getJSONObject(1).getJSONObject("pattern").getString("name");

I'm guessing there isn't always a 2nd JSON object, or not always a "pattern" included. I'll see if I can grab the exact request/response if I see it again.

minhhpham commented 6 years ago

not sure why this happened. It ran fine on my IntelliJ.

I'm guessing it was related to the response that was returned from the Syncromatics API at that time (just before noon, IIRC). The line of the exception points to:

String pattern_name = response_pattern.getJSONObject(1).getJSONObject("pattern").getString("name");

I'm guessing there isn't always a 2nd JSON object, or not always a "pattern" included. I'll see if I can grab the exact request/response if I see it again.

I know why. I was trying to grab the first pattern name in the response array because pattern name is repeated, but I forgot that index starts at 0 and not 1. I changed this to response_pattern.getJSONObject(0)

minhhpham commented 6 years ago

@minhhpham One other item I spotted in the GTFS for the MSC Express in stop_times.txt - the trips in seem to last 6 minutes long. In other words, the vehicle would complete the loop in 6 minutes (see below). This doesn't seem realistic to me. Note that frequencies.txt headway_secs is the amount of time between vehicles visiting a specific stop - if a rider is standing at a stop, it's the typical time elapsed from when one vehicle departs and the next arrives. If there is only one vehicle on the route then the time from trip beginning to trip end in stop_times.txt is equal to headway_secs. If there are X vehicles running a route, then headway_secs = time_to_complete_trip / X.

16,08:30:00,08:30:00,414,1
16,08:30:36,08:30:36,801,2
16,08:30:52,08:30:52,803,3
16,08:31:51,08:31:51,807,4
16,08:32:38,08:32:38,813,5
16,08:32:57,08:32:57,815,6
16,08:33:23,08:33:23,821,7
16,08:34:19,08:34:19,827,8
16,08:35:06,08:35:06,829,9
16,08:35:36,08:35:36,411,10
16,08:36:10,08:36:10,401,11
17,15:30:00,15:30:00,414,1
17,15:30:36,15:30:36,801,2
17,15:30:52,15:30:52,803,3
17,15:31:51,15:31:51,807,4
17,15:32:38,15:32:38,813,5
17,15:32:57,15:32:57,815,6
17,15:33:23,15:33:23,821,7
17,15:34:19,15:34:19,827,8
17,15:35:06,15:35:06,829,9
17,15:35:36,15:35:36,411,10
17,15:36:10,15:36:10,401,11
18,08:30:00,08:30:00,414,1
18,08:30:36,08:30:36,801,2
18,08:30:52,08:30:52,803,3
18,08:31:51,08:31:51,807,4
18,08:32:38,08:32:38,813,5
18,08:32:57,08:32:57,815,6
18,08:33:23,08:33:23,821,7
18,08:34:19,08:34:19,827,8
18,08:35:06,08:35:06,829,9
18,08:35:36,08:35:36,411,10
18,08:36:10,08:36:10,401,11
19,15:30:00,15:30:00,414,1
19,15:30:36,15:30:36,801,2
19,15:30:52,15:30:52,803,3
19,15:31:51,15:31:51,807,4
19,15:32:38,15:32:38,813,5
19,15:32:57,15:32:57,815,6
19,15:33:23,15:33:23,821,7
19,15:34:19,15:34:19,827,8
19,15:35:06,15:35:06,829,9
19,15:35:36,15:35:36,411,10
19,15:36:10,15:36:10,401,11

Thanks for the explanation. I updated frequencies.txt and stop_times_txt. Previously, I used the stop_times from route C to estimate MSC_Express. I used Google Maps to revise the stop times and got it to finish in 10 minutes.

barbeau commented 6 years ago

I know why. I was trying to grab the first pattern name in the response array because pattern name is repeated, but I forgot that index starts at 0 and not 1. I changed this to response_pattern.getJSONObject(0)

Ok, thanks! Just remember that at night when the service isn't running there may not be any entries in the feed. We should gracefully handle that case and not throw an exception but instead skip the relevant section of code.

minhhpham commented 6 years ago

I just ran the GTFS FeedValidator and got 3 warnings:

barbeau commented 6 years ago

I think the first two are fine.

Does USF PATS post any holiday hours now? If not then 3rd warning is ok too. If they do, we should add these exceptions to calendar_dates.txt.

barbeau commented 6 years ago

@minhhpham If you haven't already, could also also add trip_id to the VehiclePositions TripDescriptor as well as schedule_relationship=UNSCHEDULED?

minhhpham commented 6 years ago

Feed now looks like:

header {
  gtfs_realtime_version: "1.0"
  incrementality: FULL_DATASET
  timestamp: 1537202527
  [transit_realtime.oba_feed_header] {
    incremental_index: 3
    incremental_heartbeat_interval: 60
  }
}
entity {
  id: "1"
  vehicle {
    trip {
      trip_id: "1"
      schedule_relationship: UNSCHEDULED
      route_id: "A"
    }
    position {
      latitude: 28.06578
      longitude: -82.41214
      bearing: 270.0
      speed: 18.0
    }
    timestamp: 1537202522
    vehicle {
      id: "1534"
      label: "1534"
    }
    occupancy_status: EMPTY
  }
}
barbeau commented 6 years ago

@minhhpham Looks good to me!

@scrudden Please see above - does this look good to you for a new frequency-based GTFS-rt VehiclePositions feed for the USF Bull Runner?

barbeau commented 6 years ago

@minhhpham Also, before merging, let me run it locally myself and take one last look at the code.

scrudden commented 6 years ago

@barbeau. Yes the speed, bearing and the vehicle time-stamp are good.

These will be read and used by TheTransitClock. The resulting vehicle position gtfs-rt feed from TheTransitClock will have the "starttime" listed in the trip descriptor, which is not present in this feed.

barbeau commented 6 years ago

@minhhpham I'm going to close this PR and move the discussion over to https://github.com/CUTR-at-USF/bullrunner-gtfs-realtime-generator/pull/26. I want to change some things prior to merging, but right now I can't push to your repo (in the future we can work off a branch on this repo, which is what I'm doing in https://github.com/CUTR-at-USF/bullrunner-gtfs-realtime-generator/pull/26).