App usage analytics

[akoscz]

The other day I was wondering if we have an idea of the size of our user base. Which lead me to start thinking about how we could add some analytics to the app to track the number of users that use Metastone along with some other useful data, such as how many people use the deck builder, simulation and sandbox mode.

I did a bit of reading about Analytics libraries/services for Java apps but I didn't find any free ones other than Google Analytics. So I whipped up a quick proof of concept using the Google Analytics Measurement Protocol web api to see how difficult it would be for us to build it into the Metastone app. Turns out that it's not all that difficult.

Here's a quick and dirty GoogleAnalytics.java impl that uses the Lombok annotation library to generate a bunch of boiler plate code to create a builder used to create the analytics requests.

This is what an example usage would look like for tracking the application startup event:

public class MetaStone extends Application {
    // TODO: create a real trackingId with GoogleAnalytics
    private static String trackingId = "UA-12345677-12";
    // TODO: save and reuse clientId
    private static UUID clientId = UUID.randomUUID();

    @Override
    public void start(Stage primaryStage) throws Exception {
        GoogleAnalytics ga = GoogleAnalytics.builder(trackingId, clientId,
            GoogleAnalytics.HitType.event, BuildConfig.NAME)
                .category("application")
                .action("startup")
                .build();
        ga.post();
                ....
        }

So back to the question of Analytics. Wanted to get a conversation going on this topic to see if this something that we would find useful, or is it just me :) If so, what are some other things that we could track other than application startup?

[mayuso]

I never even realize this existed, not sure if the guys will be interested in implementing this into MetaStone, but I am definitely interested in learning more about it, and implementing it to other projects.

I am also curious about how many people actually play/use MetaStone.

[webadict]

App Analytics would be cool, and I'd definitely use it as a learning opportunity. As for the number of people that use MetaStone... Maybe 200-300 people? It's a guess based on traffic here. However, I'm sure the lack of a public release might've decreased the number of users recently. I'd really like one so that I can get some people that make custom cards involved again.

[akoscz]

I've got a preliminary Analytics implementation up and running. Take a look at this commit: https://github.com/akoscz/metastone/commit/3311587ec75f9b14a1ebd7dd1a5a11adea893d1d

I further built out the simple GoogleAnalytics library I mentioned above and turned it into a separate project. https://github.com/akoscz/GoogleAnalytics I didn't want to pull it into the Metastone code base since it heavily relies on Lombok for code generation which would add unnecessary complexity especially when building from an IDEA.

I have a temporary GoogleAnalytics tracker ID set up for testing. We would need to create an official Metastone GoogleAnalytics account if we decide to go with this analytics approach. Feel free to fork and build my analytics branch. I can also send out some invites for the GoogleAnalytics dashboards if you want to check them out.

[demilich1]

Thanks for the research regarding this topic. This is certainly an interesting topic, especially once we got a larger user base. However, for now, I don't think we should implement this. There are several reasons for this:

• increased code/build complexity
• we got many improvements and bugs open, most of which I consider more urgent and essential for the project
• legal issues: I am pretty sure we would need to add a disclaimer and an opt-out for the tracking
• with the current user base, I don't think the data would be particularly useful

[akoscz]

Yeah absolutely we would need a disclaimer. It's actually part of the ULA of Google Analytics which says that you must allow the user to opt-out of participating in gathering tracking data.

The motivation for looking into this topic initially was pure curiosity to answer the question of How many unique users we actually have using Metastone? I don't think we know the answer to that question. Correct me if I'm wrong. You mention This is certainly an interesting topic, especially once we got a larger user base. but if you cannot quantify the size of your use base, how can you know when your user base grows or shrinks.

I wholeheartedly agree with your points 2 and 3. There's no rush to get this into the codebase, hence the ** DO NOT MERGE ** in the title of the PR. Obviously allowing the user to opt-out is completely lacking which will need to get addressed.
As for your 4th point, I do not fully agree. Depending on what questions need to be answered, the gathering of data to answer those questions is always useful regardless of your user base. There's obviously a lot more that can be learned by tracking various events in the app other than startup and shutdown. So we should identify some (more) questions that we would find useful to have answers to if and when we decide to include an analytics implementation into the app.

Now about point number 1. Hmm... I have to disagree. Increased code complexity, minimal. Adding analytics tracking events is as simple as calling the builder methods on the GoogleAnalytics.Tracker object, then build() and send(). Can't get any simpler than that. As for build complexity, there is none. Just drop in a GoogleAnalytics jar file into the lib dir and your done. Heck we can even add a gradle dependency to have the jar downloaded automatically so we do not have to bundle it with our codebase. Now if you're speaking to the build complexity of the GoogleAnalytics library I wrote, that's a different story. It's not rocket science, but it does take a little bit of reading to learn about how the Lombok annotation library code generator works.

So to bring this full circle, the PR above is a Proof of Concept and should not be merged into our upcoming release. Obviously it's not ready and we have yet to identify questions that the analytics implementation could help us answer. I would like to turn this topic in that direction. That is to ask _What questions would we like to get answers to if we were to implement an analytics solution?_

The first question I propose is:
How many unique users we actually have using Metastone?

[webadict]

I'm actually kinda siding with @akosc here. It would be great if we knew what users were using in the program, so that we could focus on either what we can improve or what we're missing.

Things I would like to answer:

• How many users access Play Mode?
• How many users access Sandbox Mode?
  • Simulation Mode?
• How many users access the Deck Builder?
  • How many users build a deck?
  • An arbitrary deck?
• How many users play the Rogue class?
• How many users play the card Yogg-Saron?
• What are the most played cards in each mode?
• What are the most played classes in each mode?

I think it'd be cool to have this information, especially if we add features, so we can see how often they're used (How often are arbitrary decks made now? What if we add a custom card creator, anyone? Eh?)

[kairos4242]

I'm not even sure the userbase is large enough to justify this at the moment. Since custom card creation is by far the most useful application of Metastone, and with all the instability around making custom cards lately, I suspect many may be looking elsewhere. Once that is stable, and @webadict starts marketing on reddit and elsewhere again, then maybe analytics will be of more use.

[akoscz]

@kairos4242 Custom card building instability has been resolved. You can make custom cards to your hearts content by editing the json files. As for how many users we might have lost due to bugs in an internal release, we may never know because we don't have any app usage data :) Hence why this thread exists.

As for marketing on reddit and elsewhere that's great and all. But... without having analytics capability in the code BEFORE advertising there's no way to tell how many users you have attracted to the platform. Not to mention the ability to measure how many you have retained over time.

[kairos4242]

@akoscz Oh, okay, I was unaware all the custom card issues had been resolved.

The more I think about this, the more I'm starting to switch sides. This seems fairly simple to implement, and the data gained from doing so before advertising could actually allow us to see where advertising is the most effective easily. And a disclaimer isn't a big deal at all.

My one remaining qualm: Could we allow users to opt out of tracking? How easy is that to do?

[webadict]

That's a necessary part of implementing this change. If users do not wish to be tracked, that is perfectly acceptable. I have no problems with being upfront about what we track, and letting them change their tracking decision at any time. On Jun 23, 2016 10:35 AM, "kairos4242" notifications@github.com wrote:

@akoscz https://github.com/akoscz Oh, okay, I was unaware all the custom card issues had been resolved.

The more I think about this, the more I'm starting to switch sides. This seems fairly simple to implement, and the data gained from doing so before advertising could actually allow us to see where advertising is the most effective easily. And a disclaimer isn't a big deal at all.

My one remaining qualm: Could we allow users to opt out of tracking? How easy is that to do?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/demilich1/metastone/issues/174#issuecomment-228090062, or mute the thread https://github.com/notifications/unsubscribe/AFsY_oqVcFJA1WFepuzxsOQYqi2Ivtgaks5qOqfdgaJpZM4I2Itx .

[akoscz]

The opt-out would be fairly easy to implement. We would need to put a disclaimer in the app with a toggle switch to turn off tracking if the user wishes to exercise that option. I propose we add the text "Disclaimer" to the main menu screen's bottom right corner. Bottom left is the app version, middle is the donate button and bottom right could be the text "Disclaimer". Clicking on this text would bring up a modal dialog with some explanation text about what we track and why we track it followed by a toggle switch to disable analytics tracking. By default this toggle switch is in the ON state. This toggling this switch to OFF would then write a value to the metastone.properties file, something like analytics.tracking=false. On app startup we would always check for this property and behave accordingly. If the property is false then we do not gather analytics data, if it's true or the property does not exist then we do gather analytics data.

[webadict]

That's really small, and black on dark gray is not the best. Like, I know everyone hates being tracked, but I feel if we let them know what's being tracked, most users won't care. On Jun 23, 2016 11:01 AM, "AkosCz" notifications@github.com wrote:

The opt-out would be fairly easy to implement. We would need to put a disclaimer in the app with a toggle switch to turn off tracking if the user wishes to exercise that option. I propose we add the text "Disclaimer" to the main menu screen's bottom right corner. Bottom left is the app version, middle is the donate button and bottom right could be the text "Disclaimer". Clicking on this text would bring up a modal dialog with some explanation text about what we track and why we track it followed by a toggle switch to disable analytics tracking. By default this toggle switch is in the ON state. This toggling this switch to OFF would then write a value to the metastone.properties file, something like analytics.tracking=false. On app startup we would always check for this property and behave accordingly. If the property is false then we do not gather analytics data, if it's true or the property does not exist then we do gather analytics data. [image: screen shot 2016-06-23 at 8 47 16 am] https://cloud.githubusercontent.com/assets/1360047/16310284/dbff19ba-3920-11e6-8a11-542615a451a4.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/demilich1/metastone/issues/174#issuecomment-228097802, or mute the thread https://github.com/notifications/unsubscribe/AFsY_vs3Zkp0jj65_TQDbhc4Muaf9YExks5qOq2xgaJpZM4I2Itx .

[demilich1]

Also 'Disclaimer' not really suggests that it contains an option to turn off tracking. It doesn't even look like a clickable element. I am no expert regarding the legal stuff, but I don't think you can hide the option to disable tracking in the darkest corner of your UI hoping that no user finds it.

Regarding the technical implementation using the metastone.properties file: I like it.

[akoscz]

Yep, I hear ya. A more prominent visual component is necessary. Are you suggesting that we call it something else other than "Disclaimer" ? I wiped up these disclaimer screen in the this commit https://github.com/demilich1/metastone/pull/181/commits/0669bc252d91e3d7c487ecd97c1e79c4f8952de9 I'm open to renaming things to make it clear to the user what it's all about.

Also I added a analytics_disclaimer.txt to the resource folder. This file will hold the text that we want to use for our analytics disclaimer verbiage.

[webadict]

That actually looks pretty good. It's noticeable, explains what's happening, and has the opt-out.

[akoscz]

I moved all the analytics code into a MetastoneAnalytics class in the shared module so that both app and game modules will be able to register analytics events. See:https://github.com/demilich1/metastone/pull/181/commits/ea5597a4d582a736b9538038a63f04f8a1592ee2 Also created the mechanism by which we can enable and disable the logging of analytics events. See: https://github.com/demilich1/metastone/pull/181/commits/814d5b65fc8e39718d8fc984ed3212fbb263f7f1

[akoscz]

Ok, so I've added analytics tracking of the main screens in the app with commit: https://github.com/demilich1/metastone/pull/181/commits/29ac36bf7112d1dc183344b2faf934b898ba08e5 Take a look to see if this is the right place to track navigation in the app. I put the tracking in the Mediator classes right after SHOW_VIEW is broadcast. I also added tracking of when users opt out of analytics tracking so that we have an idea of how many people are choosing to opt out. https://github.com/demilich1/metastone/pull/181/commits/39a350e3adf3ebb7c069ae70c7371a7b266f9e8a

[akoscz]

Here are the various event types, categories and actions we have implemented so far.

• Register the application startup / shutdown event.
  • type : event
  • category : application
  • action : startup, shutdown
• Register navigation to a given screen.
  • type : screenview
  • category : navigation
  • action : show
  • screenName : parameter The screen name that was shown.
• Register a dialog being shown or dismissed by the user.
  • type : screenview
  • category : dialog
  • action : show, dismiss
  • screenName : parameter The title of the dialog that was dismissed
  • label : parameter The type of dialog that was dismissed. [CONFIRM, INFO, WARNING, ERROR]
  • value : parameter The result the dialog was dismissed with. [OK = 0, CANCEL = 1]
• Register when analytics tracking opt-out setting has been toggled.
  • type : event
  • category : tracking
  • action : parameter [opt-out, opt-in]

Here's the full set of Google Analytics Measurement Protocol parameters https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters I don't have all of them implemented in my GoogleAnalytics library yet. If there's a parameter that you guys thinks we should track and it's missing from my implementation, let me know and I can add it.

[akoscz]

Sample App Overview dashboard with screenview data from my testing the other day.