Admin interface analytics

[mxstbr]

For us maintainers it's really hard to judge which features of the admin interface people are using/not using.

It'd be nice to add opt-out analytics where we can collect (anonymized, obviously) information about what people are doing with the Keystone admin interface so we can adapt to what users are actually doing!

Yes I realize this isn't a bug, but this wouldn't make sense to be on Product Pains since it's mostly maintainer-facing

[rturk]

Can we also use this thread for missing features?

[mxstbr]

No, this issue is specifically about introducing something like Google Analytics to the admin interface. Please submit individual issues to Github or feature requests to Product Pains!

[vkammerer]

Could this issue relate to the overall architecture of the project, and more specifically to the potential split between a "core" codebase and "plugins"? What I mean by that is that if we had a "core" package, and then "plugin" packages, you would know which additional features are being used just by looking at the number of downloads / issues created / PR submitted for each individual plugin. I feel that adding another feature to the project (this time, analytics) is not a good solution to help shipping v4... Instead, it should be the opposite, the codebase should be getting lighter. It doesn't really make sense that there are so many issues related to Cloudinary and so on, when a lot of users will never use that service and are just waiting for a stable release of basic v4 (like me :) ) The example of babel and their strategy for going from v5 to v6 seems a good example, don't you think? They isolated a core and externalized a lot of plugins, and in the process had to design a clear API for each plugin to integrate with core. I would love to see that with Keystone as well.

[mxstbr]

They isolated a core and externalized a lot of plugins, and in the process had to design a clear API for each plugin to integrate with core.

That's what we're working towards and is one of the big reasons v4 is such a long update! (sorry about that…) We are now using react, which will very easily allow us to split out and integrate new fields (like Cloudinary), which in combination with #3052 will make our life maintaining the core a lot easier.

We will definitely not make that happen for v4, since we do want to get that released asap, but it's very very high up on our roadmap!

I feel that adding another feature to the project (this time, analytics) is not a good solution to help shipping v4...

This is not a feature per-se, as no user will ever see this. This is solely for us maintainers, and it's very important and I do want to get this in before the v4 release.

We maintainers have no clue what the hell people are doing with Keystone, and while Product Pains was a good first step to get more clarity it's not enough. Adding analytics to the interface (which will hopefully will not be too difficult, shouldn't really delay the release anyway) will help us massively to focus on making those features that people are actually using better!

[vkammerer]

Thanks @mxstbr for linking to https://github.com/keystonejs/keystone/pull/3052 , it's good to see such progress is underway! Don't be sorry about v4 being delayed, it's already great this project exists :) (and since I am using it for a small site, I am building it from the github master, so I get to use v4 in practice already).

[mxstbr]

so I get to use v4 in practice already

Please let us know what problems you run into and submit issues! That might be the most important thing for us, so we know we broke with the rework and can fix it :+1:

[r1b]

Big thumbs down from my end. Remember when Strongloop tracked user version distribution by abusing npm? Granted, this planned tracking in Keystone is a very different (and less shady) scenario but please read those user's reactions to being tracked by OSS.

Do other popular npm modules engage in this kind of opt-out tracking? I expect this kind of thing from SaaS but never in OSS.

If you want usability feedback there are less underhanded and frankly more effective ways of getting this data, including compensated IRL usability trials & voluntary usability surveys.

This kind of tracking blurs the line between commercial product & OSS in a way that I am not comfortable with.

[mxstbr]

including compensated IRL usability trials & voluntary usability surveys.

Sure, with all the cash we get from Keystone we'll do that. Oh wait…

---

I personally don't think this is a big deal, and I'm a big privacy advocate.

It's all open source anyway, it'll be easily disableable (just a single option away), it's not like we can do anything shady. We're not abusing anything here, we really just want to know which buttons people click and which they don't. We currently have no clue whatsoever.

[r1b]

I respectfully disagree. I urge the KeystoneJS team to seek out other free/low-cost alternatives to usability testing that are non-intrusive.

[mxstbr]

non-intrusive

How are analytics intrusive?

[webteckie]

It should be disabled by default if implemented.

[mxstbr]

It should be disabled by default if implemented.

Why? Just throwing out that statement doesn't really help :grin:

[jstockwin]

Most people start keystone using the generator anyway right? So we could just add an additional question into the generator options that the user is presented with?

[mxstbr]

Absolutely @jstockwin, great idea! I'd still make it opt-out though, so maybe something like this?

We anonymously collect analytics of the admin interface usage. No personal data is ever
transferred, only loading times, button clicks, etc.
(This helps us to make Keystone better with every release!)

Do you want to disable this? (y/N)

[JedWatson]

That's how Yeoman does it, seems like that's a pretty good precedent.

We should have a look at how they've implemented it behind the scenes too, I think.

Slightly different paradigm though; since it's a CLI tool they ask on first use then can easily remember the setting, whereas we'd need to put the opt-in/out in the project config which makes it easier to miss for developers who don't use the generator.

[webteckie]

@mxstbr no explanation...I'm simply in the you need to opt in camp. Why would you ever want to transmit info out of someone's personal system by default?

[mxstbr]

Just like every other website out there?

[morenoh149]

but is this really all that beneficial? aren't we already tracking feature demand through product pains?

[JedWatson]

@morenoh149 those are features we don't have yet, not how our existing features are used.

To provide my perspective on why this is valuable, we currently have several APIs that integrate file uploading to S3 and Cloudinary integrated with TinyMCE. I'd love to rip them out and make room for better ones based on the new Storage stuff that's in the works. The problem is I don't know whether there are 5 or 5000 people using those features.

Knowing what's being used, and how, is only going to become more important in terms of understanding which features are safe to deprecate, which to invest in, and whether we're actually optimising Keystone for its real audience. We only interact with a small fraction of users here and in other channels, in terms of total installs. Most of our usage is totally opaque.

These aren't questions that usability trials can answer for us.

re: Strongloop's blip - that's hostile because it was sneaky, you couldn't opt out of it, was a technical abuse of peer dependencies, exposed a vulnerability, and caused install delays / errors. I didn't see a lot of complaints that would be relevant given the way @mxstbr is designing this for Keystone.

In terms of whether there's a precedent for OSS to track usage, I think that's mixed. I mentioned Yeoman already. Node itself obviously doesn't/can't; npm tracks heaps of things. If you're serving something, nobody questions whether you can do analytics; if you're charging people, the same. We're disadvantaged because we're building software, not services, and giving it away under the most liberal license possible for free. Kind of ironic :)

Something I'm really interested in is whether django, wordpress, and other apps similar to Keystone do any of this (unfortunately a search for "Wordpress analytics" is pretty useless). Although I'm sure wordpress have a tonne of usage data just from wordpress.com which is completely controlled by Automattic.

[webteckie]

@JedWatson the argument is not whether it should be done or not. I as well think it is indeed valuable. The argument is whether it should be enabled by default. If keystone somehow asks users and they agree to it then fine. But if one can somehow setup keystone bypassing this opt-in query then I think it should not be enabled by default.

[mxstbr]

Why? Have you seen a website that let's you opt-out of analytics? Yeah, me neither.

You want to opt-out of analytics you install an ad blocker. Oh wait, that still applies! But apart from that we also allow you to globally disable it.

Not sure what all this discussion about this is needed for to be honest, this isn't some crazy secret fuck-the-users thing.

[wmertens]

BTW, if these analytics mean that users can be individually tracked, that means Keystone has to comply with the EU laws and post a "cookie warning".

Note that if you use cookies but not for tracking users, it is not needed, and if you track users but don't use cookies to do that, you still need to warn.

[r1b]

@JedWatson

Atom is another project that does this. Adding metrics to atom is/was a contentious issue.

If you look at the dependents of the yeoman insights package, you can see other projects that use Google Analytics to track users. Big players I recognized include Apache Cordova, GitHub command line tools and Jenkins command line tools. So there is precedent.

I don't believe that tracking users via Google Analytics is the only way to get the information you want. That being said, if you must do this, an opt-in approach is a satisfactory compromise. Obviously, you don't have to listen to me :smile:. I think this is the best way to communicate your intent-to-track to the user & allow them to explicitly provide consent.

@mxstbr You may not think this is a big deal but many other people in this community do (see linked atom thread). KeystoneJS risks alienating users if it adopts a cavalier attitude towards user privacy.

[mxstbr]

This issue is specifically for admin interface analytics. That's a website. Just like any other website out there it has analytics. (what's the last website you visited that did not have analytics?)

How is that a "cavalier attitude towards user privacy"? Every privacy minded person will have an ad blocker installed anyway. This is not a command line tool, this is not a standalone app. It's a website.

[mxstbr]

I do agree that analytics can be a problem, I do agree that command line tool/app analytics should be opt-in, but this is a website. I'm really not sure what all the fuss is about?

[JedWatson]

@r1b good links, thanks!

@wmertens interesting point.

@mxstbr I think the concern here is that it's not our website, it's somebody else's and we're installing our tracking in it. We have tracking in keystonejs.com and that's not an issue.

I'm fine with making this opt-in, adding it to the generator, and having the default answer there be "yes". If you're using Keystone already and upgrade without noticing that we've added analytics, and you're working on a sensitive project, that could be a real fuckup.

[r1b]

I shot an email over to rms. He recommended the Piwik project to ensure that the analytics collected can only be used by the KeystoneJS team.

[tnottu]

Since this is an issue about the admin interface, couldn't the analytics choice be presented in the admin somehow?

Maybe when a user with necessary privileges logins to the admin for the first time, a modal dialog appears, confirming the user's choice about the analytics. The choice is then saved to the database or a config file. It doesn't have to be a modal dialog because those are annoying, the UI solution could be post-install config screen or anything else. The point is: ask the question where it actually matters. It would also be a good place to explain what the analytics are and are not.

This way, existing installs will also get to make a choice when they update to a newer version.

[Noviny]

Closing this as a concluded/old discussion