Improved monitoring/telemetry for Jupyter Notebooks

StatCan / aaw-contrib-jupyter-notebooks

Jupyter Notebooks to be used with Advanced Analytics Workspace platform

Other

10 stars 13 forks source link

Improved monitoring/telemetry for Jupyter Notebooks #36

Open ca-scribner opened 3 years ago

ca-scribner commented 3 years ago

To help detecting/fixing bugs, let's improve the information we collect from Jupyter Notebooks. Some ideas:

Jupyter has logs that will record detailed failure info, etc. Can we always capture these and store then centrally for later use if users find issues?
Are there ways to collect telemetry like how many python/r/... kernels have been run? when/what packages do people install? ...

More brainstorming would be helpful here.

blairdrummond commented 3 years ago

Screenshot from 2021-01-30 16-06-38

The answer is "Very Yes". We already have prometheus metrics available and we just need to pull them.

blairdrummond commented 3 years ago

We might be able to work with this:

https://github.com/kubeflow/kubeflow/issues/5216

I think we'd be able to scrape for pods with the label notebook (like in our minio credential injector), and then scrape metrics using the blackbox exporter that @Ito-Matsuda has been playing around with.

It looks like it's possible to right a cross-namespace ServiceMonitor using that notebook label as a selector.

https://stackoverflow.com/a/64451100