SPIKE: Evaluate, document, and choose the service DIBBs will use to demo Open Telemetry log visualization and dashboarding

[bamader]

To kick off our work making telemetry data visualizable for demo purposes, we'll need to figure out a way to visualize queried or persisted logs. The scope of this ticket is to do some spike research to choose an appropriate visualization and/or dashboarding system we can use for persisting and analyzing log outputs. The tool we choose should, at minimum, be:

• open source
• containerized (or able to be containerized)
• integrable with and support OT
• flexible with the DB solutions and storage tools we use might use for persisting logs
• lightweight, free, and most importantly, low effort to get a demo running

Current options being considered include Graphana and Metabase

[robertandremitchell]

@DanPaseltiner and @bamader here is my write-up of grafana and metabase. Based on the criterion here: https://docs.google.com/document/d/14rhbDRFZupFWPqKAO4PgzojfnJ20IEB5CLpGuDsQmCE/edit#heading=h.3d2m2ye5jnak, I think we would be better served by Grafana. However, Grafana seems to have a tightly-knit relationship with Prometheus, so I wonder if we still want to have it if we do end up with a different service after this spike: https://app.zenhub.com/workspaces/skylight-dibbs-viper-6480ba23b5a07644e0e46d23/issues/gh/cdcgov/phdi/1474.

Grafana does seem to have support for opentelemetry (link to repo in the write-up), so I don't think we need to be locked into Prometheus if we don't find it passes muster. Just noting.

[robertandremitchell]

also, looking at OpenObserve, which wasn't mentioned above but seems interesting and was in the dashboard doc.

[robertandremitchell]

Grafana+Prometheus will be explored as part of our MVP.