Optimize fetching feeds - compression and HTTP header "if modified"

[barbeau]

Issue by barbeau Wednesday Apr 12, 2017 at 15:58 GMT Originally opened as https://github.com/CUTR-at-USF/gtfs-realtime-validator/issues/124

---

Summary:

We should be sure to look at the following, which will optimize the fetching of feeds (both GTFS and GTFS-realtime):

• If-Modified-Since header
  • https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.25
• Last-Modified header - allows us to see if the data has changed since our last request -
  • http://stackoverflow.com/a/15544499/937715
  • https://www.mnot.net/cache_docs/
  • https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Last-Modified
• accept-content: gzip - allows us to download compressed content
  • http://www.rgagnon.com/javadetails/java-HttpUrlConnection-with-GZIP-encoding.html

I believe we'd want to first look at If-Modified-Since header, then Last-Modified header.

[barbeau]

Comment by skinkie Wednesday Apr 12, 2017 at 16:10 GMT

---

http://stackoverflow.com/questions/7095897/im-trying-to-use-javas-httpurlconnection-to-do-a-conditional-get-but-i-neve

[barbeau]

Comment by barbeau Tuesday May 23, 2017 at 14:55 GMT

---

TriMet's feeds are a good one to test this with:

• GTFS - http://developer.trimet.org/schedule/gtfs.zip
• GTFS-realtime - http://developer.trimet.org/ws/V1/TripUpdate?appID=0000000000000000000000000

Their GTFS downloads very slow for some reason.

@Suryakandukoori This would be another issue you could take a shot at (see above thread). GTFS data is fetched in GtfsFeed, and GTFS-rt data in BackgroundTask. I believe you'd need to add a new field "lastRetrieved" to the Hibernate model classes in the model package. Let me and @mohangandhiGH know if you have any questions.

[barbeau]

Comment by barbeau Tuesday May 23, 2017 at 14:57 GMT

---

Also, we'd want to test this against a few servers, including ones that don't support the HTTP headers (if we can find some) to make sure it works under both conditions. More GTFS-rt feeds (and corresponding GTFS data) can be found at http://transitfeeds.com/search?q=gtfsrt.