I'm currently stress-testing our application at-scale using a Kubernetes cluster running in Azure, and profiling memory usage using bytehound.
Getting the profiling data out of the cluster. or accessing it at all to begin with, is proving to be quite the challenge.
I can use kubectl cp to copy the .dat file out of an individual worker pod while it's running, but that leaves me with an incomplete profile because it inevitably fails as it hits an error at the tail end of the file which is still being written. I can't just do it when the worker pod is complete because kubectl cp doesn't work for pods that have exited, and also since this is a stress-test a common failure mode of the application is the pod getting OOM-killed.
I'm configuring the workers to write their profiling data to a persistent volume backed by Azure Files, but that only gets me halfway. I could probably just log into the Azure dashboard and download the contents of the volume manually, but I would really like to just be able to use bytehound server to inspect the data in-situ.
In theory, that's easy to do as I just have to set up a deployment to run bytehound server against the volume after the stress-test job completes, and expose that with a Kubernetes service so I can access it over the Internet.
However, since bytehound server doesn't require any authentication and by its very nature exposes extensive details about my application (potentially including the whole binary), that leaves it wide open to any attacker who's running automated scans of Azure's public IP ranges looking for anything juicy.
As a workaround, I'm using Caddy to reverse-proxy the bytehound API and inject an HTTP Basic Auth challenge, but that's kind of finicky and a whole extra dependency I'd rather not need. I recognize that this is also doable with the NGINX Ingress controller for Kubernetes, but that's even more annoying to set up if it's not already installed in your cluster.
Proposal
I think a command-line option that adds an HTTP Basic Auth challenge to bytehound server's responses would be a great value-add for any kind of remote profiling.
Obviously that only goes so far without TLS, but that part's surprisingly easy with cloud providers these days.
A fully fledged login page would make for a more unified experience, but is honestly overkill for my needs.
Background
I'm currently stress-testing our application at-scale using a Kubernetes cluster running in Azure, and profiling memory usage using
bytehound.Getting the profiling data out of the cluster. or accessing it at all to begin with, is proving to be quite the challenge.
I can use
kubectl cpto copy the.datfile out of an individual worker pod while it's running, but that leaves me with an incomplete profile because it inevitably fails as it hits an error at the tail end of the file which is still being written. I can't just do it when the worker pod is complete becausekubectl cpdoesn't work for pods that have exited, and also since this is a stress-test a common failure mode of the application is the pod getting OOM-killed.I'm configuring the workers to write their profiling data to a persistent volume backed by Azure Files, but that only gets me halfway. I could probably just log into the Azure dashboard and download the contents of the volume manually, but I would really like to just be able to use
bytehound serverto inspect the data in-situ.In theory, that's easy to do as I just have to set up a deployment to run
bytehound serveragainst the volume after the stress-test job completes, and expose that with a Kubernetes service so I can access it over the Internet.However, since
bytehound serverdoesn't require any authentication and by its very nature exposes extensive details about my application (potentially including the whole binary), that leaves it wide open to any attacker who's running automated scans of Azure's public IP ranges looking for anything juicy.As a workaround, I'm using Caddy to reverse-proxy the bytehound API and inject an HTTP Basic Auth challenge, but that's kind of finicky and a whole extra dependency I'd rather not need. I recognize that this is also doable with the NGINX Ingress controller for Kubernetes, but that's even more annoying to set up if it's not already installed in your cluster.
Proposal
I think a command-line option that adds an HTTP Basic Auth challenge to
bytehound server's responses would be a great value-add for any kind of remote profiling.Obviously that only goes so far without TLS, but that part's surprisingly easy with cloud providers these days.
A fully fledged login page would make for a more unified experience, but is honestly overkill for my needs.