Document DoS method on your bank account :)

I maintain a JupyterHub deployment on GKE for our academy classes and finally found the time to look into our billing spikes:

As you can see on some days logging gets 'excited', 20% of our bill 'excited' :)

Peeking into GCP's Log Explorer, I found all the stdout (and probably stderr) of the notebooks were unexpectedly being logged:

Now I probably did something stupid, though I have searched the documentation for 'notebook log' and really only found 'debugging' which is already disabled for me.

Others may also be following my steps though and being hit unnecessarily with a logging bill.

Proposed change

Highlight that "cloud bills you for everything, including logs". :)

Also that users can directly affect your bank balance in a negative way with no more than a while True: print("hello world\n") loop.

Alternative options

If it does not exist, include an option to crank up and down the logging level for particular log sources, in particular 'notebook' not just a global level.

Suggesting documentation, or making it easer to locate, as this probably already exists and I just missed it.

Maybe change the defaults too, if it turns out it is turns out this is enabled by default.

Who would use this feature?

Everyone who does not have infinite money.

(Optional): Suggest a solution

For GCP, I have installed a log sink to /dev/null logs I am not interested in.

resource.labels.container_name="notebook"
logName=~"projects/kx-academy-learn-jupyterhub/logs/std.*"

Which as a gcloud command becomes:

gcloud --project kx-academy-learn-jupyterhub logging sinks update _Default --add-exclusion name=notebook,filter="resource.labels.container_name=\"notebook\" AND logName=~\"projects/kx-academy-learn-jupyterhub/logs/std.*\""

jupyterhub / zero-to-jupyterhub-k8s