Add telemetry for generic usage metrics collections in Graylog

[ousmaneo]

Add the following features to core to support telemetry:

• Saving user preference for telemetry data metrics
• Add tools to configure and send metric
• Instrument metric collection for events. (ex. pageview)

[kroepke]

I'll keep updating this comment. Please feel free to break out issues where it makes sense.

For each cluster, we want to create a group whose key is the actual cluster ID the telemetry generates. We probably only need to load these group properties once when the application loads unless there are ever metrics that change dramatically between page loads and might affect cohorts.

Additional properties we'd like to track:

• [x] The initial cluster creation date. It is available from the cluster_config collection (e.g., db.cluster_config.find({"type" : "org.graylog2.plugin.cluster.ClusterId"}).pretty()) either directly as last_updated or encoded in the ObjectID of the entry.
• [x] 30-day rolling average of cluster traffic, available through the cluster traffic metrics
• [x] Graylog cluster information (number of nodes, potentially hardware information, version)
• [x] Search cluster information (number of nodes, other topology information, product name, version) For the node detail information initially lets take the following json path elements:
  • $._nodes.total
  • $.nodes[*].version
  • $.nodes[*].roles
  • $.nodes[*].os
  • $.nodes[*].jvm.mem.heap_max_in_bytes
• [x] number of total users in the cluster (excluding service accounts)

[digdilem-work]

How can we disable telemetry tracking globally, please?

Our company policy requires we do not expose even "non private" information. (Trusting a third party's definition of 'private' is hard for companies that deal with sensitive information)

As it's automatically enabled per-user, yet shares global server information (above) it seems unduly difficult to disable this - possibly deliberately so. Apparently needing to ask each user to manually untick something that didn't exist in 5.0

[hydrapolic]

How can we disable telemetry tracking globally, please?

Our company policy requires we do not expose even "non private" information. (Trusting a third party's definition of 'private' is hard for companies that deal with sensitive information)

As it's automatically enabled per-user, yet shares global server information (above) it seems unduly difficult to disable this - possibly deliberately so. Apparently needing to ask each user to manually untick something that didn't exist in 5.0

I'd also like to disable it globally or have the possibility to disable it per user as an administrator.

[htolic]

Until Graylog ships new changes that will allow changing telemetry to off globally something along the lines will suffice. Run it against Mongo every time you add new users to Graylog:

db.getCollection("users").aggregate([
    { "$set": { "user_id": { "$toString": "$_id" }, "telemetry_enabled": false, "telemetry_permission_asked": true } },
    {
        "$unset": ["_id", "timezone", "auth_service_id", "roles", "auth_service_uid", "account_status",
            "password", "session_timeout_ms", "full_name", "permissions", "external_user", "email",
            "username", "preferences", "startpage", "last_name", "first_name", "service_account"]
    },
    { "$out": "telemetry_user_settings" }
])

[melvin-suter]

This needs to be addressed fast!