Next evolution of AAW's ECK instance

[chuckbelisle]

@Jose-Matsuda let's bring up to speed @wg102 with our ECK solution.

• [x] Share docs and confirm she can login
• [x] Explain what was done prior to now and go over the goals for the end-state
  • Current state: used for sifting through the many logs on the cluster that we get. Helpful for debugging / finding a certain phrase from the logs
  • Pat has created very nice dashboards for us to use that show who's been accessing kubeflow, actions on notebooks etc.
• [x] Go over what are the next steps for the solution (Need, wants, etc) Possible goals / see if possible to add
  • A dashboard / query that shows who started or submitted a notebook
  • Distributed tracing frontend / backend (integration with OpenTelemetry)
  • Make it a first stop for debugging the cluster (instead of relying on k9s + l for logs)
  • Better documentation (Pat might have just need to find), simpler pre-made queries?
• [x] Talk about the current stability and/or existing issues
  • [x] CNS ticket here regarding dev.
  • Prods been accessible for me / no problems recently. If it shows up we need to make an issue for it to track it

[wg102]

The documentation for elastic can be found here: https://github.com/StatCan/aaw-argocd-manifests/blob/6f0ed9da36e9a96ec091f11455632fc9143d63fc/monitoring-system/elastic/README.md

[mathis-marcotte]

Kibana can be used to make dashboards https://www.elastic.co/guide/en/kibana/current/dashboard.html

Not sure what the difference would be between what those could be compared to our grafana dashboards that already exist(if grafana and elasticsearch share the same logs, if they get different data then that's the difference), but still something to note as being possible.

[Jose-Matsuda]

Notes from talk WIP, these can be implemented as saved queries or even dashboards.

• Aggregate and connect logs for kubeflow applications and possibly have displayed in a dashboard
• Check logs for anything that says "bad certificate"
• Create an easy way to observe controller logs (dashboard?) and connect them.
• Istio container logs
• Secrets notification (say SSL certs Pat has done a PoC for dev here maybe)
  • Say specifically for oauth2-proxy or kubeflow under the Advanced Analytics Workspaces (Dev)
• Free elastic doesnt have notifs (maybe prometheus would be better)
• RUM (real user monitoring)

Perhaps also look into bettering our alerting with grafana?

• https://github.com/StatCan/aaw/issues/1621

[wg102]

Initial discussion is done, now we are creating a few issues to start with.