Fields to track usage?

[greggish]

Chris Hwang of First 5 Alameda suggested that we might want to add fields for 'usage metadata' to the spec. I think I understand her to be suggesting two fields: specifically, a field for number of times a record has been listed in search results, and another field for the number of times a record has been accessed.

Thoughts?

[monfresh]

Is it possible that Chris is confusing the spec with an app that consumes and displays the data, like SMC-Connect?

Usage metrics like these are usually tied to some application that will be consuming and displaying the data, such as a web search client like SMC-Connect. If these fields were added to the spec, for them to be useful, each application that consumed the data would need to track their own metrics, then someone would need to contact each application owner to get the metrics from them (assuming they would want to share that info), then all the metrics would need to be aggregated. This process would have to be repeated to keep these statistics up to date.

For example, let's say DC published its data in the OpenReferral format, and now they want to allow developers to build useful applications based on this data. Providing an API would be a great way to make it easy for those applications to be built. Ohana API is one solution since it supports the OpenReferral format. Now, once the API is available, developers will start building applications using this data, and each of these applications will be displaying records in search results, and visitors of these applications will be accessing records. To measure how often an app is accessed, which records have been accessed, etc., an application can use one of many analytics services, such as Google Analytics.

Now, if DC wanted to know how many times "Bread for the City" was accessed overall across any application, they'd have to ask all of these application owners to share their metrics data with them and then they'd have to put it all together. You'd probably also need to track metrics outside of applications, like 211 calls.

Then let's say there was a field in OpenReferral to represent these metrics, DC would publish a new version of their OpenReferral data, but this type of data would not be imported into Ohana API, for example. It seems like it would only be useful internally in DC to keep track of metrics, which they might share with various organizations so they can know how many times their services have been looked up. This kind of metric gathering and sharing does not need a field in the OpenReferral spec to be possible.

[chrisatfirst5]

Ah, thanks for clearing the fog (smart people!)... Perhaps usage data does not reside as part of the spec, but suggested as a matter of consistent measurement across local projects. I'm admittedly hung up on local tactics.

[michaelwang1]

I would still argue that usage statistics should warrant at least a bit more consideration at the Ohana API level. The use case you are describing holds true internally if there is only a single application being developed within a certain geography. However, if there are multiple front end applications accessing a the common Ohana API, it would be useful for the local govermnet to have accumulated usage statistics accross all applications; especially if the local government is funding some of these organizations. While I recognize this may be impractical because there would likely be high variability in how APIs report back usage statistics, I thought I'd just throw this use case out there. -Mike

On Mon, Jul 21, 2014 at 1:29 PM, chrisatfirst5 notifications@github.com wrote:

Ah, thanks for clearing the fog (smart people!)... Perhaps usage data does not reside as part of the spec, but suggested as a matter of consistent measurement across local projects. I'm admittedly hung up on local tactics.

— Reply to this email directly or view it on GitHub https://github.com/codeforamerica/OpenReferral/issues/55#issuecomment-49660811 .

[michaelwang1]

Sorry...for some reason I blanked out on the middle two paragraphs of Moncef's reply. Basically, my comment is that if openreferral could make it easier for DC to ask how many times "Bread for the City" was accessed across all platforms, it might be worthwhile to add the usage statistics to the specs. That said, I completely understand the likely impracticality of multiple UIs trying to syncronize usage data. It would probably require a separate table of log data to be anything useful since DC would want to know how many times it was accessed in a specific time period. -Mike

On Mon, Jul 21, 2014 at 12:30 PM, Moncef Belyamani <notifications@github.com

wrote:

Is it possible that Chris is confusing the spec with an app that consumes and displays the data, like SMC-Connect?

Usage metrics like these are usually tied to some application that will be consuming and displaying the data, such as a web search client like SMC-Connect. If these fields were added to the spec, for them to be useful, each application that consumed the data would need to track their own metrics, then someone would need to contact each application owner to get the metrics from them (assuming they would want to share that info), then all the metrics would need to be aggregated. This process would have to be repeated to keep these statistics up to date.

For example, let's say DC published its data in the OpenReferral format, and now they want to allow developers to build useful applications based on this data. Providing an API would be a great way to make it easy for those applications to be built. Ohana API is one solution since it supports the OpenReferral format. Now, once the API is available, developers will start building applications using this data, and each of these applications will be displaying records in search results, and visitors of these applications will be accessing records. To measure how often an app is accessed, which records have been accessed, etc., an application can use one of many analytics services, such as Google Analytics.

Now, if DC wanted to know how many times "Bread for the City" was accessed overall across any application, they'd have to ask all of these application owners to share their metrics data with them and then they'd have to put it all together. You'd probably also need to track metrics outside of applications, like 211 calls.

Then let's say there was a field in OpenReferral to represent these metrics, DC would publish a new version of their OpenReferral data, but this type of data would not be imported into Ohana API, for example. It seems like it would only be useful internally in DC to keep track of metrics, which they might share with various organizations so they can know how many times their services have been looked up. This kind of metric gathering and sharing does not need a field in the OpenReferral spec to be possible.

— Reply to this email directly or view it on GitHub https://github.com/codeforamerica/OpenReferral/issues/55#issuecomment-49653597 .

[monfresh]

@michaelwang1 What you are describing is a service that is already provided by various companies out there. @spara can correct me if I'm wrong, but usage statistics are outside of the scope of the spec. Ohana API and the OpenReferral spec are two different things. If a local community decided to only use Ohana API, and all front-end apps were consuming Ohana API only, then if you wanted to track usage at the API level, you have the options below:

1. Pay for a service like Mixpanel. There's a free option as well for low traffic. If you're hosting the API on Heroku, there are many analytics services to choose from.
2. Use open-source solutions like Grafana.
3. Implement your own logging mechanism.

This is not something that we think should be baked into the API because each community should be able to choose which analytics service they want to use, just like we don't automatically host the app on any particular server.

[michaelwang1]

Sweet. I think this is fair. I think, however, there is a distinction between website usage statistics vs "actual" usage statistics and Ohana can be a means of standardizing the reporting of "actual" usage statistics. To define a bit:

1. Web site usage statistics: Typical logging/access data that is captured by the first two options you describe above.
2. "Actual" (real-life) usage statistics: Two primary examples off the top of my head. 1. How many times was a person referred to this resource? 2. How many times did a person successfully connect to a resource? Again, I think it is reasonable to defer to the communities, but just wondering if we have an opportunity to standardize and make things easier for the communities (a la medicare and its reporting standards). -Mike

On Tue, Jul 22, 2014 at 9:06 AM, Moncef Belyamani notifications@github.com wrote:

@michaelwang1 https://github.com/michaelwang1 What you are describing is a service that is already provided by various companies out there. @spara https://github.com/spara can correct me if I'm wrong, but usage statistics are outside of the scope of the spec. Ohana API and the OpenReferral spec are two different things. If a local community decided to only use Ohana API, and all front-end apps were consuming Ohana API only, then if you wanted to track usage at the API level, you have the options below:

1.

Pay for a service like Mixpanel https://mixpanel.com/. There's a free https://mixpanel.com/pricing/ option as well for low traffic. If you're hosting the API on Heroku, there are many analytics services https://addons.heroku.com/#analytics to choose from. 2.

Use open-source solutions like Grafana http://grafana.org/. 3.

Implement your own logging mechanism.

This is not something that we think should be baked into the API because each community should be able to choose which analytics service they want to use, just like we don't automatically host the app on any particular server.

— Reply to this email directly or view it on GitHub https://github.com/codeforamerica/OpenReferral/issues/55#issuecomment-49760476 .

[monfresh]

Weird, I don't see @michaelwang1's comments anymore (UPDATE: looks like he deleted his GitHub account). Here is his latest comment:

Sweet. I think this is fair. I think, however, there is a distinction 
between website usage statistics vs "actual" usage statistics and Ohana can 
be a means of standardizing the reporting of "actual" usage statistics. To 
define a bit: 
1. Web site usage statistics: Typical logging/access data that is captured 
by the first two options you describe above. 
2. "Actual" (real-life) usage statistics: Two primary examples off the top 
of my head. 1. How many times was a person referred to this resource? 2. 
How many times did a person successfully connect to a resource? 
Again, I think it is reasonable to defer to the communities, but just 
wondering if we have an opportunity to standardize and make things easier 
for the communities (a la medicare and its reporting standards). 
-Mike 

"Actual" usage statistics are a different story. I was only talking about app usage statistics, which is what @greggish's original comment referred to. But I think I'm starting to understand what you're looking for here. It's not the initial collection of the statistics that you want OpenReferral to handle (it can't), but rather how to report this information after you've collected the stats. Is that right? If so, I will defer to @spara to comment on that.

[michaelwang1]

Hrm. Not sure what happened with my reply either. Yes, I think it would be interesting to actually report on both website traffic usage statistics (not for Ohana to handle) as well as real-world resource utilization statistics. Ohana would be an interesting place for front-ends to submit "actual" usage statistics to so they can be compiled easily. Also, having them built into the spec would help standardize reported measures even when front ends are built even if they don't initially report back the data. -Michael

On Tue, Jul 22, 2014 at 10:17 AM, Moncef Belyamani <notifications@github.com

wrote:

Weird, I don't see @michaelwang1 https://github.com/michaelwang1's comments anymore. Here is his latest comment:

Sweet. I think this is fair. I think, however, there is a distinction between website usage statistics vs "actual" usage statistics and Ohana can be a means of standardizing the reporting of "actual" usage statistics. To define a bit:

1. Web site usage statistics: Typical logging/access data that is captured by the first two options you describe above.
2. "Actual" (real-life) usage statistics: Two primary examples off the top of my head. 1. How many times was a person referred to this resource? 2. How many times did a person successfully connect to a resource? Again, I think it is reasonable to defer to the communities, but just wondering if we have an opportunity to standardize and make things easier for the communities (a la medicare and its reporting standards). -Mike

"Actual" usage statistics are a different story. I was only talking about app usage statistics, which is what @greggish https://github.com/greggish's original comment referred to. But I think I'm starting to understand what you're looking for here. It's not the initial collection of the statistics that you want OpenReferral to handle (it can't), but rather how to report this information after you've collected the stats. Is that right? If so, I will defer to @spara https://github.com/spara to comment on that.

— Reply to this email directly or view it on GitHub https://github.com/codeforamerica/OpenReferral/issues/55#issuecomment-49770226 .

[spara]

Ot of scope, this is an implementation issue not a spec issue