[Epic] Application annotations for Cryostat Agent integrations

[andrewazores]

Describe the feature

There should be some defined annotation prefix (like io.cryostat/) and keys like inject-agent, inject-tls-cert. Users should be able to add these annotations to their applications (on the Deployment or Pod) within any Namespace associated with a Cryostat instance. When the Operator observes that an application has such an annotation, an integration feature should be enabled:

• Cryostat Agent injection. The Operator mounts a volume to the annotated application, containing the Cryostat Agent JAR.
  • if the Operator can detect the base image or framework that the application is using, and if we determine some safe method to do so (maybe with another optional annotation), the Operator will also modify the application's Deployment to automatically instrument the application with the Agent JAR, ie. by setting the -javaagent JVM flag as well as setting environment variables to point the Agent at the correct Cryostat instance
• TLS cert injection. This would be used primarily for https://github.com/cryostatio/cryostat-agent/issues/138 . If the user has requested Cryostat Agent injection to their application, they can use this second annotation to also request a TLS cert. This requires cert-manager integration. The Operator uses cert-manager to generate a TLS cert and places that into a ConfigMap or Secret and mounts that to the application. The Operator can also automatically add configuration to the application so that the injected Agent instance uses this new cert for its webserver.
  • the same TLS cert ConfigMap/Secret should also be mounted to the associated Cryostat server instance and included in its TLS truststore
• the associated Cryostat instance's TLS certificate should also be mounted to the application so that the Agent's HTTP client can trust the certificate

This way we can automate end-to-end TLS in both directions between Cryostats and their Agent instances, with the user doing as little work as possible to enable this security feature.

Tasks:

• [x] #936
• [x] #941
• [ ] #952

[andrewazores]

Some more thoughts about the Agent injection/integration.

1. We want our Operator to be able to supply the Agent JAR to the application - the user should be able but not required to rebuild their application image containing our Agent.
2. Configuring the Agent should be as simple and automatic as possible - ideally, the user just flips one switch to opt in, which the Operator sees and takes care of the rest.

---

For point 1, packaging the Agent JAR version that corresponds to the Cryostat version which the Operator will deploy seems relatively simple. As discussed previously, the Operator can put this into some resource like a ConfigMap or Secret and mount it to the application Deployment.

If the Operator is already modifying the Deployment then it is trivial to add environment variables for configuring the Agent, to point it at the correct Cryostat instance, use the Pod IP for the callback URL, etc.

---

But how to actually attach the Agent to the application JVM? We have talked about trying to do some detection of the application's framework or base image and manipulating entry points, environment variables, etc. to add the -javaagent switch. This seems error-prone and conflict-prone.

We have also experimented with running an Attach Agent as a sidecar process and connecting to the main container, but this might be tricky to put together in practice and probably runs into a lot of security constraints.

What if we can do the equivalent of a kubectl exec on containers from within the Operator controller?

https://github.com/operator-framework/operator-sdk/issues/4302

https://github.com/zalando/postgres-operator/blob/master/pkg%2Fcluster%2Fexec.go#L18-L44

https://github.com/operator-framework/operator-sdk/issues/1561

If the Agent JAR is already available in the application container (via mounted ConfigMap), then there is an easy solution when combined with agent dynamic attach:

https://github.com/cryostatio/cryostat-agent/pull/234

Just exec the Agent process in PID autodetect mode.

[andrewazores]

If we can do an equivalent to kubectl cp to copy the Agent JAR from the Operator into the application container, instead of mounting a ConfigMap, then this can even be done without causing an application restart. Fully dynamic and online.

[andrewazores]

https://stackoverflow.com/questions/51686986/how-to-copy-file-to-container-with-kubernetes-client-go