[SLO] Add snapshot telemetry for SLO configurations

[jasonrhodes]

Similar to the work planned for Observability Alerting Rules, we want to enhance the existing SLO snapshot telemetry by adding more info about the configurations used within SLOs.

Examples of what we want to capture

• Fields used to filter and group by
• Other settings withint he SLO?

Stretch goal: move the existing SLO snapshot telemetry out of the main xpack-phone-home indices (where it isn't being mapped) and into its own index, so we can analyze the data in our telemetry ES cluster and not ONLY in the Big Query / Looker Studio tools. We are doing this for the alerting rules telemetry so hopefully we can use those lessons to make this simpler. We can also punt this to a separate ticket.

[elasticmachine]

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)

[kdelemme]

@jasonrhodes Do we want to report the actual group by fields or just the usage of the group by feature, e.g.

• Number of SLO
  • without groupby: count
  • with 1 groupby: count
  • with 2 grouby: count
  • with 3 groupby: count
  • with more than 3 groupby: count

Also, at the moment, the reported snapshot looks like:

    total: 0,
    by_status: {
      enabled: 0,
      disabled: 0,
    },
    by_sli_type: {},
    by_rolling_duration: {},
    by_calendar_aligned_duration: {},
    by_budgeting_method: {
      occurrences: 0,
      timeslices: 0,
    },

Do we want to update it so we can have a more breakdown view e.g. (probably a question for the data team)

{
  total: count,
  kql: {
    total: count,
     occurrences: {
       total: count,
       rolling: {
         total: count,
         7d: count,
         30d: count,
         90d: count
       }
       calendar: { },
     },
     timeslices: { },
  }
}

[jasonrhodes]

I like that idea for capturing group by counts. Would it be reasonable to do some kind of "top hits" like query against all configured SLOs in a cluster to see which fields they are grouping by? As it's stored in saved objects I don't know if this is as simple as it would be if it were in ES directly but it would be nice to have that info to look through at times.

I don't have a strong opinion on the other question, but whatever you and the team and the BI team think is best, so long as we can get what we need out of it, is good to me.