Closed barbeau closed 9 years ago
We should also include an opt-out setting for GA.
As per my comment at https://github.com/OneBusAway/onebusaway-iphone/issues/263#issuecomment-35180750 I disagree that allowing an opt-out would be a good idea at this point.
Here's what OBA iOS is tracking, and what we should use as a template for tracking in OBA Android.
Apparently Google Analytics is now integrated in Google Play Services with the latest update, so a separate SDK for analytics is no longer required.
I'm new to GA - does anyone know if we can report custom statistics via GA?
I'm thinking of how to report the number of favourite stops that users have, or the number of reminders people use etc.
@jrharshath I believe one option to track those type of statistics might be custom metrics.
I've started working on this and hope to have a PR for initial feedback this coming weekend.
@bbodenmiller Thanks for the heads up! I was actually planning to start work on this late this week or early next week as part of a project I'm working on. I assume you saw that I created an Android view in GA? Please keep me posted on how this is going, and I'd be glad to help speed up development.
Also, things we're interested in (beyond what OBA iOS is tracking), include tracking stopIDs information for requests (ideally that could be categorized by region), and rough locations and/or distance to stop for which info is being accessed (for example, to help answer the question if users are accessing OBA while they are standing at the stop, or from anther location such as home, work, etc.). Location info is tricky, since we need to protect privacy as well. So, we'll need to obfuscate location to some level - I plan to read more about existing techniques for this. Simple schemes include reducing accuracy of lat/longs through simple randomization of an offset or simply recording distance to stop or some general categorization of proximity to stop (e.g., near, medium, far). If we record raw distance to stop, we should obfuscate location before this too, since in some cases you could reverse engineer the probable source location. Likely something for discussion on the OBA dev list too, once I put together a plan for this.
Don't think I saw your GA view, could you point me to it?
https://github.com/bbodenmiller/onebusaway-android/commit/f4feed5fcc2a40f4a18ed74905832d862a4a09e8 has the progress I've made so far.
Any idea how we can get around Play Services with GA only working with SDK 9+ or do you think it is okay for us to up the minimum SDK to 9?
The type of information you are looking to log beyond what iOS has right now certainly sounds interesting but might require some creative solutions. That might be a good place to start looking in to beyond what I've started to get this on par with iOS tracking.
We're down to about 4% usage on Android 2.2. I could be convinced to make 10+ the minimum.
@bbodenmiller If you go to your GA account, Android view should be listed under "OneBusAway->OneBusAway Smartphone->OneBusAway Android." I think I've got the filtering correctly set up now so iOS data should only be visible under "OneBusAway iOS", and Android is visible under "OneBusAway Android". There doesn't seem to be a way to limit an initial spike in OBA Android numbers that occurs in history before you can set up the filter to exclude all iOS devices, so we'll just have to ignore the Day 0 numbers when running reports.
tl;dr:
IMO, since we're looking at implementing a few Play Services-based features, it will be simpler to start with Android 2.3 as a baseline and drop 2.2 (API 8) support.
Full explanation:
If we use newest version of the GA SDK (v4), that means using Google Play Services 4.3 or higher, which is only compatible with Android 2.3 (API Level 9) or higher. Note that according to the docs that this doesn't require Google Play Services to be installed on the device for GA to work at runtime:
Note: The SDK can be used and will work on devices that do not have Google Play Services. In this case the SDK will will automatically fall back to local dispatching.
Only option I see to keep support for Android 2.2 is to go with a legacy version of the GA SDK.
@paulcwatts just beat me to it, but since I have it typed up - from the Google Play Developer console, it looks like we still have ~8,400 OBA users (4.03%) on Android 2.2. Based on the Android dashboard, globally it looks like only 1% of Android devices accessing Google Play are still on Android 2.2.
We could do Maps v2 and Location API v2 while still keeping support for Android 2.2 via Google Play Services for Froyo (which just freezes the version at v3.2.65), but we'll need to drop Android 2.2 eventually if we want to keep pace with the latest Google Play Services version, since updates for Play Services are no longer being released for Android 2.2. And, I've seen misc. issues (e.g., problems with location updates suddenly stopping) that are (supposedly) fixed in later Play Services versions (e.g., >= 4.3.x). If we do an initial release for Android 2.2 with Google Play Services for Froyo, we'll likely be stuck with some problems out of the gate that can't be fixed.
I'm all for SDK 9 being the minimum just wanted to check with you guys. I also saw the note about GA working without Google Play Services being installed and wondered if that might allow us to still support SDK 8. Unfortunately Android Studio just complains when I put compile 'com.google.android.gms:play-services:4.4.52'
in build.gradle
and don't up the minimum SDK.
As a side note, what exactly does target SDK mean? Is there a reason behind keeping it at 17?
Target SDK is the SDK we're building against, and what APIs we think we can use. So it allows us to "target" newer versions of Android while maintaining compatibility with older SDKs.
We could move it to the current SDK, if we want to target APIs that are unavailable in <= 17.
I believe @paulcwatts is confusing targetSdkVersion
with compileSdkVersion
- compileSdkVersion
defines which SDK you're building against, and you can only access APIs/methods in your source code that are <= compileSdkVersion
. As @paulcwatts states, I'd only suggest bumping this if we need APIs that are only available in > compileSdkVersion
, to avoid accidentally accessing an API that is not backwards compatible to our minSdkVersion
(buildToolsVersion
, on the other hand, should be kept up to date, to avoid bugs in the tool chain).
targetSdkVersion
is one of the more confusing attributes - it should be the latest version of Android that you've actually tested your application against. Its a hint to the platform whether certain backwards compatibility features should be enabled for your app - you can see a list of these compatibility features on the Build.VERSION_CODES page, and a full description of targetSdkVersion
on the uses-sdk page.
A good example is when the physical menu key was removed from some Android devices. If you had targetSdkVersion
<= 13, on API 14 and higher Android would add a soft menu key to the app, to ensure users could access your legacy menu options. If you bump targetSdkVersion
to 14 or higher, Android assumes you've updated your app to make legacy menu options available some other way (e.g., via ActionBar), and removes the soft menu key. If you didn't actually update your app to make your menu functions available via another method, then users on devices without menu keys couldn't access the menu functions (we actually got bit on this one on another app).
So, if we bump targetSdkVersion
to > 17, we should definitely do testing on devices with API > 17 to ensure everything behaves as expected (and closely look at the Build.VERSION_CODES page above to see if any changes affect us).
Good explanation @barbeau, I've been out of the game for too long :). For what it's worth, I use a KitKat (SDK=19) device all day, way more than I would use a SDK=17 device.
Thanks for the explanation. I'm not saying we need to up the targetSdkVersion
yet but I did notice on the uses-sdk page it says:
To maintain your application along with each Android release, you should increase the value of this attribute to match the latest API level, then thoroughly test your application on the corresponding platform version.
@paulcwatts Good to know, I'm on KitKat now too as my personal device (Galaxy S3, formerly stock, now Cyanogen). So, if we bump targetSdkVersion
we should be able to get some good testing in.
@bbodenmiller Key word in that sentence is thoroughly :).
@cagryInside is currently working on this. As mentioned earlier, we’d like to capture a metric for how far the user is in real-time from their bus stop when they check the arrival time. For example, is the user waiting at the bus stop when they check the time, or from a more distant location (e.g., work, coffee shop)? Does this differ depending on the stop?
To do this in a privacy-sensitive manner, we’re proposing that we capture distance ranges for each time the user checks the real-time info, without capturing the exact lat/long.
Here’s our current range definitions that would be captured for each arrival time check:
Stop ID is captured for each arrival time check as well. So, we’d be able to determine the breakdown of near/middle/far access for each stop in the system, but without capturing precise locations for the access. We’d also make the assumption that locations with an extremely large accuracy uncertainty value (e.g., > 200m) would automatically get tossed into the “FAR” category (or thrown out completely?), no matter the exact location.
Any comments/suggestions? Any different opinions for what the range definitions should be, and the number of ranges? The current choice is somewhat arbitrary, so improvements are welcome.
Above question also posted to oba-dev and users lists: https://groups.google.com/forum/#!topic/onebusaway-users/bWe1nWm_gwI https://groups.google.com/forum/#!topic/onebusaway-developers/bWe1nWm_gwI
Are you planning on collecting this information using Google Analytics? Before you go through all the effort of wiring this up using GA I'd be sure to check that GA allows you to crunch the numbers or export the way you'd like to.
If you are going through the effort I think creating another bucket for everything outside the "FAR" category would be good also. This might show other unusual usage patterns or allow you to detect issues if a large number of people start appearing in a "VERY FAR" category.
Are you planning on looking at the users reported location accuracy and taking that in to account somehow? How about elevation vs. stop elevation (could indicate working in a high rise)?
I think your buckets are either too small or you need more of them. For example my house is about 2 blocks from the bus stop walking and about 1.5 as the crow flies but I would appear in the "FAR" bucket. The nearest RapidRide stop is .5 miles or a 10 minute walk and it appears I wouldn't be counted when viewing from home which I do very often. Perhaps you might do 5 buckets:
(Assuming walking speed of 2.5 mph from WolframAlpha). On Wed, Jan 7, 2015 at 2:28 PM Sean Barbeau notifications@github.com wrote:
Above question also posted to oba-dev and users lists: https://groups.google.com/forum/#!topic/onebusaway-users/bWe1nWm_gwI https://groups.google.com/forum/#!topic/onebusaway-developers/bWe1nWm_gwI
— Reply to this email directly or view it on GitHub https://github.com/OneBusAway/onebusaway-android/issues/105#issuecomment-69104104 .
Are you planning on collecting this information using Google Analytics? Before you go through all the effort of wiring this up using GA I'd be sure to check that GA allows you to crunch the numbers or export the way you'd like to.
@bbodenmiller Yes, @cagryInside has been working with a PoC for this with Google Analytics, and it seems to work as expected. The buckets are essentially treated as arbitrary labels and can be sliced/diced as desired.
Thanks for the examples. I agree that we'd probably benefit from a few more buckets, and I like trying to frame it within walk times - perhaps we'd benefit from a "bike" bucket as well, so we could differentiate a reasonable bike distance for an immediate trip from users looking at stops from across the city or across regions. We'll look into this further.
Are you planning on looking at the users reported location accuracy and taking that in to account somehow?
Yes, we could either throw out locations with a high accuracy uncertainty value, or automatically place them in the "FAR" bins. Picking the threshold here is a bit tricky.
How about elevation vs. stop elevation (could indicate working in a high rise)?
Thanks for bringing this up. Altitude values derived purely from GPS aren't precise enough to determine level within a building. But, over the last year or so phones have been shipping with embedded barometers, and other sensor fusion, that are making altitude more accurate. I'll have to think more about how this might be integrated into analytics - our system above is two dimensional. Today's altitude data probably wouldn't be very useful, but I could see technology progressing to a point in the next year or so where the altitude precision and accuracy would be much better, and usable. The other challenge here is that you'd also need an altitude value for the nearest bus stop or base map layer at a minimum, and potentially building data, to compare against and determine if someone is in a high-rise, and which "floor" someone is on. This would need to be pulled from another dataset.
One other challenge I should mention related to capturing and using altitude - on Android, the only location accuracy value we have is horizontal accuracy. We do not have a comparable vertical accuracy value. So, given the current Location APIs on Android, determining whether or not an altitude value from a device is usable (i.e., from a newer-gen device with a barometer and sensor fusion) can be difficult - you'd really have to code around specific device make/models that you know support advanced altitude positioning.
IIRC, I believe iOS does have a vertical accuracy value.
We have implemented the fallowing segmentation hierarchy for capturing a metric for how far the user is in real-time from their bus stop when they check the arrival time.
In GA interface, we could segment these data.
Here is the results from GA:
Segment by search distance:
Search Distance Far:
Region: Tampa
What did you guys decide on the distances? Do you know if there is a limit on number of reported events? An event for every stop a user views could be a lot of events.
On iOS we report the users region to GA slightly differently. Since iOS and Android will use the same GA account it would be nice if they reported it the same way so the data can be aggregated together. Once the regions been reported once I believe you can just use that in your data view.
We also implemented exact same region reporting mechanism with IOS. This is an extra event for reporting distance.
Here is our latest study on distance:
Average walking speed from Wolfram Alpha Average biking speed from Wikipedia
@cbrakewood or @kariwatkins - at the OBA Annual Meeting you mentioned a reference for past research on how far (on avg) transit riders were willing to walk to use public transportation. Could you share those, so we can tweak the above Google Analytics distance thresholds based on past research?
I still think it would be beneficial to have a smaller bucket so you can tell how many people check the schedule while they are at the stop.
I agree with @bbodenmiller - we will want an "AT STOP" minimum bucket that has something like 30m distance. My fault, @cagryInside, for not catching that earlier for the above table :).
I also commented to @cagryInside offline that we might want to change the labels to make it easier to visualize the relative distance category order in GA. Here's what it currently looks like:
Right now it’s hard to visually identify relative order of fields.
Maybe “1 – AT STOP", "2- CLOSE WALK”, “3 – MEDIUM WALK”, etc.? Any other ideas?
Do you know if there is a limit on number of reported events? An event for every stop a user views could be a lot of events.
From the Google Analytics Android SDK quota documentation:
For each tracker instance on a device, each app instance starts with 60 hits that are replenished at a rate of 1 hit every 2 seconds. Applies to All hits except for ecommerce (item or transaction).
I think that should be sustainable, and I doubt that the distance-to-stop metric would push us over the edge, considering that each UI change gets logged - @cagryInside could you please look at this when you're testing to see approx. how many hits we're averaging per app session?
There is also a "property" limit:
These limits apply to the Web Property / Property / Tracking ID. 10 million hits per month per property. If you go over this limit, the Google Analytics team might contact you and ask you upgrade to Premium or implement client sampling to reduce the amount of data being sent to Google Analytics.
Sounds like that's more of a soft limit, though, and we'll know when Google tells us :).
According to Google the limitations are:
Universal Analytics Enabled
This applies to analytics.js, Android iOS SDK, and the Measurement Protocol.
200,000 hits per user per day
500 hits per session not including ecommerce (item and transaction hit types).
Average # of hit per session: 20 (production data)
On android we are reporting same with IOS + stop distance
Average # of hit per session: 14 (test data)
What's the reason for including the region as part of the event category? If you go down that route you could theoretically end up with an infinite number of categories which I don't think is desired. I think something along the lines of an event category of stop_metrics
with event action being the regioncode+stopid and then the event label being User Distance: 0 - At Stop
would be best. If you put all regions under one event category you can still filter things down to an individual region by using a custom secondary dimension as demonstrated below:
What I haven't yet figured out is how you can get an aggregate of all event labels across all the event actions - is this something the team has figured out how to do? Say for example you want to know what percentage of OBA users viewing the stop data at the stop vs very far away from the stop (event label), is there a way to get this information without looking at each stop (event action) individually?
For walk distance, see TCQSM – pg 4-18 - http://onlinepubs.trb.org/onlinepubs/tcrp/tcrp_rpt_165ch-04.pdf
Also this paper: https://www.jtlu.org/index.php/jtlu/article/view/308/338
There’s probably more if you need me to do a search.
The rule of thumb I’ve heard is ¼ mile walk to bus stops, ½ mile walk to rapid transit, 2 mile bike to rapid transit.
It’s probably good to think through if you want circles to represent things (short walk, long walk, bike, etc) or if you want equally spaced circles (1000 ft, 2000 ft, etc). I think the bike distance is important, but that means that your circles will overlap substantially on most bus lines. It seems like you want a circle that represents “station area” (maybe 500 feet), then a short walk distance (1000 feet?) and then 2000, 3000, 4000, 5000 & 10,000 feet perhaps.
I agree with @bbodenmiller about putting the region name into category.
Here is my simple analysis:
Only question I have is what is the region code and why is it useful? If region code is the region name, since we already implement the region names as secondary dimension, we won't need it anymore.
Currently the stop_id's looks like the fallowing in the GA report:
For Atlanta:
Stop Id: MARTA_137024
Stop Id: CCT_72
...
For Tampa:
Stop Id: Hillsborough Area Regional Transit_3105
Stop Id: Hillsborough Area Regional Transit_4547
...
I'm good with just stop id. I misspoke & meant agency id, not region id. My understanding is that in the Puget Sound King County Metro stop id's in OBA are: 1_stop-id-on-sign (e.g. 1_600) and other agencies are: agencyid_stop-id-on-sign. I was under the impression that other regions were setup the same way so essentially I just meant stop id as OBA tracks it rather than stop id as it is shown on the physical stop.
Generally, OBA identifies stops internally as:
<agency_id>_<stop_id>
This is how it appears in Puget Sound.
If the GTFS data from an agency doesn't specify agency_id
, then the format is:
<agency_name>_<stop_id>
This is how in appears in Tampa and Atlanta.
In OBA you can also remap the agency_ids if desired (e.g., if two agencies use the same agency_id in their GTFS data).
stop_id
is the unique stop identifier in the GTFS data for that agency (note this is per agency, not per region). stop_code
in GTFS data is the ID that is shown on the physical stop sign. In some GTFS datasets these are the same values, but in others they are not.
So, I believe we should be good with the general OBA internal identification of the stops that you see in the REST API requests, which is either <agency_id>_<stop_id>
or <agency_name>_<stop_id>
. Either combination should be unique within a region. And, as long as we can associate each stop with its region via secondary dimension, then that combination should be globally unique.
The rule of thumb I’ve heard is ¼ mile walk to bus stops, ½ mile walk to rapid transit, 2 mile bike to rapid transit.
It’s probably good to think through if you want circles to represent things (short walk, long walk, bike, etc) or if you want equally spaced circles (1000 ft, 2000 ft, etc). I think the bike distance is important, but that means that your circles will overlap substantially on most bus lines. It seems like you want a circle that represents “station area” (maybe 500 feet), then a short walk distance (1000 feet?) and then 2000, 3000, 4000, 5000 & 10,000 feet perhaps.
@kariwatkins Yeah, good points.
So, how about this, which is a compromise between even estimates (in meters) and some of the above rules-of-thumb:
// ~330 ft
// ~650 ft
// ~ .25 mi buffer
// ~ .5 mi buffer
// seems like we should have a ~1 mi buffer, to keep things even
// ~ 2 mi buffer
Any thoughts on whether using a constant gap between categories is better than an exponential gap? Above each category doubles the one before it.
We should also keep in mind that these are "as the crow flies" straight line distances that we're measuring, too, not actual travel path distance based on road/sidewalks. It would be interesting if we could capture actual walk distance using something like OpenTripPlanner for each view, but I don't think we're there yet - maybe in the next round of analytics :). This is one reason for the exponential gaps above as well - I feel that we'll be able to capture our most detailed information closest to the stop - the further you get from the stop the less likely that straight-line distance = actual travel distance. We also have to consider in both bus stop position error (e.g., a HART bus stop near CUTR is around 50m from its recorded lat/long) and real-time GPS error - we won't know either of these when looking at these numbers.
Thank you @barbeau for pointing out <agency_id>_<stop_id>
.
Currently the report looks like this with unique stop_id
:
Since the bucket names seem very subjective why don't we just label them with the distances and let each report make its own assumptions about what the distances mean? Label name's I'm proposing are formatted as code
below with other information not being part of label and just included here for information purposes. I believe the label name should include both the start and end boundaries of the bucket.
0 - 0-50m
<45 second walk1 - 51-100m
45 second-1.5 minute walk2 - 101-200m
1.5-3 minute walk3 - 201-400m
3-6 minute walk4 - 401-800m
6-12 minute walk / 2.5-5 minute bike5 - 801-1600m
12-24 minute walk / 5-9.5 minute bike6 - 1601-3200m
24-48 minute walk / 9.5-19 minute bike7 - >3200m
>48 minute walk / >19 minute bike(Assuming walking speed of 2.5 mph from WolframAlpha & biking speed of 166m/minute from Wikipedia).
Additionally I believe the code I saw threw out values if the accuracy was below a certain threshold. I think that is a good idea but what do others think?
Changing the label names to the distance ranges makes sense to me. We could probably drop the numeric prefix then, as well. It would be simpler to add/remove categories in the future if this isn't included in the label. Does anyone know if there is a way to assign ordinal values to labels so they are sortable in GA, without including the ordinal value in the label itself?
There is one other field that we don't use right now. We can assign values to each label and sort them by values.
@cagryInside Ok, cool, that might work. Is this changeable within the GA console for each label, or is it set permanently by the reporting client?
@barbeau it is set permanently by the reporting client.
We could add zeros to the label names to make them permanently sortable even as new buckets are added:
0000-0050m
0050-0100m
0100-0200m
0200-0400m
0400-0800m
0800-1600m
1600-3200m
3200-∞m
These look good. I like the most recent ones Ben proposed with the zeros. BTW – from a more official walking & biking speed source (FHWA), it appears that walking speeds are around 1.2 m/s and biking speeds are around 5.6 m/s on trails. I didn’t see info about on street biking though. My personal experience is that biking on street is about 3 times walking speed. In sum, your buckets are good and times may be a bit off but close enough:
The AASHTO Green Book indicates that "average walking speeds range from 0.8 to 1.8 m/s." The MUTCD assumes an average walking speed of 1.220 m/s (4 ft/s). However, research on pedestrian walking speeds has demonstrated that more than 60 percent of pedestrians walk more slowly and that 15 percent of pedestrians walk at less than 1.065 m/s (3.5 ft/s) (Kell and Fullerton, 1982).The AASHTO Green Book recommends a walking rate of 1.0 m/s (39 in/s) for older pedestrians (AASHTO, 1995).
The AASHTO Guide for the Development of Bicycle Facilities (p. 36) specifies 30 km/h (20 mi/h) as the minimum design speed that should be used on shared use paths (figure 47).) It goes on to provide modifiers for grades and strong prevailing winds. Of the bicyclist participants in this research project, only four, or 1 percent, exceeded the suggested 30-km/h (20-mi/h) design speed. The 85th percentile speed for bicyclists was 22 km/h (14 mi/h). This suggests that the 30-km/h (20-mi/h) design speed may be conservative; however, further research is needed.
Ok, sounds good. Seems like it would be wise to add one more 0 in the prefix, though, just in case we need to define a bucket above 9,999m (e.g., if horizontal accuracy measurement fails to filter out outliers). So:
00000-00050m
00050-00100m
00100-00200m
00200-00400m
00400-00800m
00800-01600m
01600-03200m
03200-∞m
Does that sound reasonable?
I think, adding fixed labels to distances could be useful.
User Distance: 00000-00050m
User Distance: 00050-00100m
User Distance: 00100-00200m
User Distance: 00200-00400m
User Distance: 00400-00800m
User Distance: 00800-01600m
User Distance: 01600-03200m
User Distance: 03200-∞m
If we want to see only the metrics for how far the user is in the label. We could use this fixed label in the search field.
Yeah, that makes sense too. Is the event label sortable alphanumerically?
If so, maybe swapping the two would be better, to still allow sorting by
initial distance:
00000-00050m - User Distance
On Jan 22, 2015 12:11 PM, "Cagri Cetin" notifications@github.com wrote:
I think, adding fixed labels to distances could be useful.
User Distance: 00000-00050m User Distance: 00050-00100m User Distance: 00100-00200m User Distance: 00200-00400m User Distance: 00400-00800m User Distance: 00800-01600m User Distance: 01600-03200m User Distance: 03200-∞m
If we want to see only the metrics for how far the user is in the label. We could use this fixed label in the search field.
[image: image] https://cloud.githubusercontent.com/assets/2777974/5860385/039e345e-a22f-11e4-9d1f-cf0dd92529a7.png
[image: image] https://cloud.githubusercontent.com/assets/2777974/5860490/98aae75e-a22f-11e4-8f2e-53b11859f553.png
— Reply to this email directly or view it on GitHub https://github.com/OneBusAway/onebusaway-android/issues/105#issuecomment-71057132 .
Is the event label sortable alphanumerically?
Yes, labels are alphanumerically sortable.
e.g.:
Ok, then having the swapped label order makes sense to me.
On a different note - we probably shouldn't report exact custom server URLs for privacy reasons, because they could be IP addresses. Maybe we just have a CUSTOM_URL generic name. On Jan 22, 2015 1:29 PM, "Cagri Cetin" notifications@github.com wrote:
Is the event label sortable alphanumerically?
Yes, labels are alphanumerically sortable. e.g.: [image: image] https://cloud.githubusercontent.com/assets/2777974/5861799/b21179fa-a23a-11e4-98ec-6c1a32c13a36.png
— Reply to this email directly or view it on GitHub https://github.com/OneBusAway/onebusaway-android/issues/105#issuecomment-71071731 .
As discussed in https://github.com/OneBusAway/onebusaway-iphone/issues/263#issuecomment-35123537, we'd like to add Google Analytics to both the OBA iOS and Android apps to have better information about how they are being used. As recommended in GA Best Practices, we plan to use the same property for both apps:
Tracking data is needed for research projects such as StopInfo.