Add option to reduce bandwidth usage

[barbeau]

Summary:

From a user:

I love the app but have recently discovered a problem. I am currently throttled on my data plan down to 128kb/s. It's painful but liveable.

However, One Bus is not performing well. The situations below were never noticeable when I was running with my full LTE speed.

• Getting an initial load of stops near me can take quite a while, perhaps 30 seconds or more.

• Once stops are loaded, the map will rubber band back to the location where it started. This is quite annoying since normally I have already scrolled away from that spot (last known location is cached from yesterday so I have hit the location button to get to my current location).

• Loading routes, once a stop is selected, takes even longer. This morning I estimate it took 60-120 seconds to show me anything for a stop (76372, King County Metro). Staring at a spinner for that long is not an engaging proposition.

Thanks for the app. I hope you can find some ways to lighten the payloads for us folk that occasionally land in bandwidth jail.

Stop reported by user - http://pugetsound.onebusaway.org/where/standard/stop.action?id=1_76372

API request for arrival times for this stop - http://api.pugetsound.onebusaway.org/api/where/arrivals-and-departures-for-stop/1_76372.json?minutesAfter=65&key=TEST

Steps to reproduce:

(Reported by user) Set bandwidth to 128kb/s and scroll around map, tap on stop

Expected behavior:

Perform reasonably well

Observed behavior:

(Reported by user) High lag

Device and Android version:

App Version: 2.1.9
Model: Nexus 6P
OS Version: 8.0.0 / 26
Google Play Services App: 11.3.02 (440-161239932)
Google Play Services Library: 9452000
Region/API: Puget Sound (selected automatically)
Loc: network XXXX, 4 second(s) ago

[barbeau]

One option to help reduce response size is to eliminate the references section in the response by adding includeReferences=false to the request - see "References" section here: http://developer.onebusaway.org/modules/onebusaway-application-modules/current/api/where/index.html

Depending on the stop, this could potentially cut a lot of text, as IIRC OBA includes elements that are two degrees of separation from the currently viewed element. But, this could potentially break the app, if it's pulling detailed entity information from the references section. Risk of breaking something probably varies per request, as I don't recall off the top of my head what features are using the extra references section info. So, this would need to be tested for any API calls to which we add it. If the only effects are cosmetic, users would likely still find this mode valuable when their plans are throttled.

[barbeau]

Quick Test

API request to get arrival times for reported stop - http://api.pugetsound.onebusaway.org/api/where/arrivals-and-departures-for-stop/1_76372.json?minutesAfter=65&key=TEST

The above request yields a JSON response of 76329 characters.

Here's the includeReferences=false version: http://api.pugetsound.onebusaway.org/api/where/arrivals-and-departures-for-stop/1_76372.json?minutesAfter=65&key=TEST&includeReferences=false

This request yields a JSON response of 46336 characters - so it's definitely a huge improvement in size.

[barbeau]

Requested by another user, but this time in context of panning the map:

The one thing I wanted to note is that I feel this app uses an inordinate amount of data. I've used 70 MB of data in the last couple weeks and while that's not that much to pay for (I'm on a pay per GB plan), it seems high for what this app shows me. I suspect it also means sometimes the slowness of the app is due to my connection. I ride mostly just a few buses to get between Seattle and Bellevue, so sometimes when I'm scrolling around looking for connection timing and alternative options, I think the app is downloading all information for all stops on the map, even though 90% of them I will never click on or be interested in.

For another data saving feature, we could only fetch stops (i.e., call stops-for-location API) when the user is zoomed close into the map. And, we'd display a message ("Data saver on - zoom in to see stops") when zoomed out. This would prevent a lot of fetching when the user is scrolling across the city, and would also improve performance on slower connections.

[changhaitravis]

or also implement GZIPInputStream and use it if it's detected in the http response header. I have a compression proxy endpoint set up: http://us-central1-dizzy-165804.cloudfunctions.net/OneBusAway/

In the following file: /src/main/java/org/onebusaway/android/io/ObaDefaultConnection.java

All that's needed is to check

if(mConnection.getContentEncoding() == 'gzip')
{
return new new InputStreamReader(
    new GZIPInputStream(mConnection.getInputStream()), 8 * 1024)
)
}

Do this for both GET and POST Also make sure the request header ("Accept-Encoding", "gzip") is present (they seem to be, but it could be coming from my nodejs express compression middleware on my cloud function).

I'm gonna make a build with these changes this weekend.

[barbeau]

@changhaitravis Agreed, gzip support is important. However, the Android HttpURLConnection: https://developer.android.com/reference/java/net/HttpURLConnection

...should use this by default if it's supported by the server. The above docs say:

By default, this implementation of HttpURLConnection requests that servers use gzip compression and it automatically decompresses the data for callers of URLConnection.getInputStream(). The Content-Encoding and Content-Length response headers are cleared in this case. Gzip compression can be disabled by setting the acceptable encodings in the request header:

urlConnection.setRequestProperty("Accept-Encoding", "identity");

Setting the Accept-Encoding request header explicitly disables automatic decompression and leaves the response headers intact; callers must handle decompression as needed, according to the Content-Encoding header of the response.

I haven't tested this myself with OBA servers so I'd be interested in seeing any results from your tests.

[changhaitravis]

To be honest, the original intent was just for me to create a compression proxy endpoint and hook it up to OBA to it a custom API, as a workaround to it not being activated on the OBA server.

However after I got my proxy compression endpoint confirmed working, on the browser with key=TEST and confirmed up to 90% savings, I then tried it on the the android app and it works, but bandwidth usage has not improved (I have zlib compression set to 9).

I'll report back later with my findings, after I produce a build with GZIPInputStream used instead of Buffered.

[changhaitravis]

Okay my bad, so the issue was my proxy was hosted on google's serverless "functions" which will only return "content-encoded:gzip" if the user-agent was a supported browser or had the keyword "gzip" somewhere in it.

I made an nginx proxy using proxy_pass now, with gzip enabled. Much faster with this, but I don't have as much free egress bandwidth (1GB vs. 5GB with serverless) (otherwise I would've gone with this from the get-go)

Here's the nginx proxy (temporary): http://onebusaway.railpage.org/

I have the compression level set to 9, but I'm seeing massively reduced data utilization. It might be good to bring this up with whoever's sysadmin/webadmin of the official puget sound OBA server.

[barbeau]

So the good news is that the sys admins are rebuilding the OBA backend for Puget Sound and will include gzip support in the next iteration. The bad news is that there isn't an immediate time frame, it's more of a "we're working on it."

[changhaitravis]

This is good to hear.

Here's my findings after about a month of using my compression proxy.

I hit a similar data cap to the op last month and was throttled to the same 128 kbps (~16kBps). When I use my proxy when throttled, it's nowhere near 60-120 seconds. Closer to 10-15 seconds for initial loading arrival at a transit center stop. Map panning was actually pretty good, iirc.

Total Bandwidth Consumption for me for the past month (according to Android 4.4 Data Usage) coming from 1 bus away is ~5MB (~400KB background). But I usually request my commute's stops straight from favorites, I refresh quite a bit. I also commute between Bellevue and Seattle like the 2nd guy.

The way T-mobile throttles still leaves you with LTE response times, so very small bursts of data come through almost instantly.

You could try to optimize the json payload, but I'd say compression alone solves this bandwidth problem, since it barely sips data now. Sorry I don't have baseline bandwidth usage numbers to give you, since I was using a compression app before and only got rid of it after I set up my proxy for OBA.

[barbeau]

Thanks for sharing! The background data is interesting, as the only background data we use is after a trip plan we'll keep checking for realtime changes to your trip itinerary after for a while and for reminders before the scheduled arrival time. If you're not using either of these features, though, OBA app code shouldn't be triggering any network call, so this would be related to Google Play Services and analytics or maps background activity.