[RCA] Initial telemetry collection setup for investigations (register usage collector, snapshot telemetry)

[jasonrhodes]

We want to collect snapshot telemetry specifically for our investigations UI, to track how users are using this new app once it's enabled.

Acceptance criteria

• Unmapped snapshot telemetry is being sent for investigations UI (set up similar to SLO)
• Initial metrics required:
  • Basic count of investigations per status (On the server side)
  • Number of investigations per origin (started from scratch/an alert)
  • Number of observations added per investigation
• Stretch metrics if possible (drop anything that proves to be too difficult to calculate for now):
  • Number of notes per investigation
  • Number of discrete users per investigation (creator, added any notes, added any observations)
  • Some sort of stats about whether this investigation integrated with LLM, not sure exactly what makes sense here but something should be possible...

Follow up tasks

• [ ] Investigate storing these telemetry in a separate, mapped index so we can query them in ES, and not just in BI/Big Query
  • Look into this with @shahzad31 on how they are doing this in Synthetics
• [ ] Follow up with more specific investigation telemetry metrics, as needed

[elasticmachine]

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)

[maryam-saeidi]

What we can cover now:

• Basic count of investigations per status (On the server side)
• Number of investigations per origin (started from scratch/an alert)

[jasonrhodes]

Thank you, @maryam-saeidi -- this is a great start. Could we easily get some info about:

• Number of observations added per investigation
• Number of notes per investigation
• Number of discrete users per investigation (creator, added any notes, added any observations)
• Some sort of stats about whether this investigation integrated with LLM, not sure exactly what makes sense here but something should be possible...

[maryam-saeidi]

Kevin proposed the two above-mentioned items, and he will create some tickets with instructions about the implementation based on what we have for SLO.

@kdelemme Would you please also create tickets for the items that Jason shared?

[jasonrhodes]

I wonder how much we can do in a single ticket scope? I'll update this description with the ideas suggested here in comments and let's see if we can do that in a single set of work. If it blows up we can carve things out and push them to a separate ticket, but getting this started is the most important thing.

[jasonrhodes]

OK I've updated the description to add a scoped set of AC and I've unassigned Kevin, as this is now ready. I'm moving it to Ready for the current cycle.

[kdelemme]

Number of observations added per investigation

I guess I can gather the avg/p90/min/max number of items/notes across all investigation.