search

cryostatio / cryostat-legacy

OUTDATED - See the new repository below! -- Secure JDK Flight Recorder management for containerized JVMs
https://github.com/cryostatio/cryostat
Other
224 stars 31 forks source link
containerjfr containers docker flightrecorder hacktoberfest hacktoberfest2021 java jdk jfr jmx jvm kubernetes mission-control missioncontrol monitoring observability oci openshift podman profiling

readme

CI build Quay Repository Google Group : Cryostat Development

A container-native JVM application which acts as a bridge to other containerized JVMs and exposes a secure API for producing, analyzing, and retrieving JDK Flight Recorder data from your cloud workloads.

This repository is no longer maintained. See cryostat for the source repository for Cryostat 3.0+ and active development.

SEE ALSO

CONTRIBUTING

We welcome and appreciate any contributions from our community. Please visit our guide on how you can take part in improving Cryostat.

See contribution guide →

REQUIREMENTS

Build Requirements:

Run Requirements:

BUILD

Setup Dependencies

Build project locally

Build and run project locally in development hot-reload mode

Build and push to local podman image registry

TEST

Unit tests

Integration tests and analysis tools

Skipping tests

Running integration tests without rebuild

RUN

Run on Kubernetes/Openshift

Run on local podman*

Note: If you get a 'No plugin found' error, it is because maven has not downloaded all the necessary plugins. To resolve this error, manually run ./mvnw help:evaluate to prompt maven to download the missing help plugin.

Run on local podman with Grafana, jfr-datasource and demo application*

To run on local podman, cgroups v2 should be enabled. This allows resource configuration for any rootless containers running on podman. To ensure podman works with cgroups v2, follow these instructions.

Note: If your podman runtime is set to runc v1.0.0-rc91 or later it is not necessary to change it to crun as recommended in the instructions, since this version of runc supports cgroups v2. The article refers to an older version of runc.

CONFIGURATION

Cryostat can be configured via the following environment variables:

Configuration for cryostat

Configuration for Automated Analysis Reports

Configuration for JMX Connections and Cache

Configuration for Logging

Configuration for Event Templates

Configuration for Archiving

Configuration for database

MONITORING APPLICATIONS

In order for cryostat to be able to monitor JVM application targets the targets must have RJMX enabled or have the Cryostat Agent installed and configured. cryostat has several strategies for automatic discovery of potential targets.

The first target discovery mechanism uses the OpenShift/Kubernetes API to list service endpoints and expose all discovered services as potential targets. This is runtime dynamic, allowing cryostat to discover new services which come online after cryostat, or to detect when known services disappear later. This requires the cryostat pod to have authorization to list services within its own namespace. By default this will look for Endpoints objects with ports named jfr-jmx or numbered 9091. This behaviour can be overridden using the environment variables CRYOSTAT_DISCOVERY_K8S_PORT_NAMES and CRYOSTAT_DISCOVERY_K8S_PORT_NUMBERS respectively. Both of these accept comma-separated lists as values. Any observed Endpoints object with a name in the given list or a number in the given list will be taken as a connectable target application. To set the names list to the empty list use -. To set the numbers list to the empty list use 0.

The second discovery mechanism is JDP (Java Discovery Protocol). This relies on target JVMs being configured with the JVM flags to enable JDP and requires the targets to be reachable and in the same subnet as cryostat. JDP can be enabled by passing the flag "-Dcom.sun.management.jmxremote.autodiscovery=true" when starting target JVMs; for more configuration options, see this document . Once the targets are properly configured, cryostat will automatically discover their JMX Service URLs, which includes the RJMX port number for that specific target.

The third discovery mechanism is the Podman API. If the Podman API socket is available at its default filesystem location then Cryostat will query the libpod/containers endpoint to determine what target applications may be available. Containers must have the Podman label io.cryostat.connectUrl applied, and the value should be the remote JMX or Cryostat Agent HTTP connection URL that Cryostat can use to communicate with the target.

To enable RJMX on port 9091, the following JVM flag should be passed at target startup:

    '-Dcom.sun.management.jmxremote.port=9091'

The port number 9091 is arbitrary and may be configured to suit individual deployments, so long as the port property above matches the desired port number and the deployment network configuration allows connections on the configured port.

Additionally, the following flags are recommended to enable JMX authentication and connection encryption:

-Dcom.sun.management.jmxremote.authenticate=true # enable JMX authentication
-Dcom.sun.management.jmxremote.password.file=/app/resources/jmxremote.password # define users for JMX auth
-Dcom.sun.management.jmxremote.access.file=/app/resources/jmxremote.access # set permissions for JMX users
-Dcom.sun.management.jmxremote.ssl=true # enable JMX SSL
-Dcom.sun.management.jmxremote.registry.ssl=true # enable JMX registry SSL
-Djavax.net.ssl.keyStore=/app/resources/keystore # set your SSL keystore
-Djavax.net.ssl.keyStorePassword=somePassword # set your SSL keystore password

JMX Connectors

Cryostat supports end-user target applications using other JMX connectors than RMI (for example, WildFly remote+http) using "client library" configuration. The path pointed to by the environment variable CRYOSTAT_CLIENTLIB_PATH is appended to Cryostat's classpath. This path should be a directory within a volume mounted to the Cryostat container and containing library JARs (ex. jboss-client.jar) in a flat structure.

In the particular case of WildFly remote+http, you might do something like the following to add this capability:

$ podman cp wildfly:/opt/jboss/wildfly/bin/client/jboss-client.jar clientlib/

EVENT TEMPLATES

JDK Flight Recorder has event templates, which are preset definition of a set of events, and for each a set of options and option values. A given JVM is likely to have some built-in templates ready for use out-of-the-box, but Cryostat also hosts its own small catalog of templates within its own storage. This catalog is stored at the path specified by the environment variable CRYOSTAT_TEMPLATE_PATH. Templates can be uploaded to Cryostat and then used to create recordings.

ARCHIVING RECORDINGS

cryostat supports a concept of "archiving" recordings. This simply means taking the contents of a recording at a point in time and saving these contents to a file to the cryostat process (as opposed to "active" recordings, which exist within the memory of the JVM target and continue to grow over time). The default directory used is /flightrecordings, but the environment variable CRYOSTAT_ARCHIVE_PATH can be used to specify a different path. To enable cryostat archive support ensure that the directory specified by CRYOSTAT_ARCHIVE_PATH (or /flightrecordings if not set) exists and has appropriate permissions. cryostat will detect the path and enable related functionality. run.sh has an example of a tmpfs volume being mounted with the default path and enabling the archive functionality.

SECURING COMMUNICATION CHANNELS

To specify the SSL certificate for HTTPS/WSS and JMX, one can set KEYSTORE_PATH to point to a .jks, .pfx or .p12 certificate file and KEYSTORE_PASS to the plaintext password to such a keystore. Alternatively, one can set KEY_PATH to a PEM encoded key file and CERT_PATH to a PEM encoded certificate file.

In the absence of these environment variables, cryostat will look for a certificate at the following locations, in an orderly fashion:

If no certificate can be found, cryostat will autogenerate a self-signed certificate and use it to secure HTTPS/WSS and JMX connections.

If HTTPS/WSS (SSL) and JMX auth credentials must be disabled then the environment variables CRYOSTAT_DISABLE_SSL=true and/or CRYOSTAT_DISABLE_JMX_AUTH=true can be set.

In case cryostat is deployed behind an SSL proxy, set the environment variable CRYOSTAT_SSL_PROXIED to a non-empty value. This informs cryostat that the URLs it reports pointing back to itself should use the secure variants of protocols, even though it itself does not encrypt the traffic. This is only required if Cryostat's own SSL is disabled as above.

If the certificate used for SSL-enabled Grafana/jfr-datasource connections is self-signed or otherwise untrusted, set the environment variable CRYOSTAT_ALLOW_UNTRUSTED_SSL to permit uploads of recordings.

Target JVMs with SSL enabled on JMX connections are also supported. In order to allow Cryostat to establish a connection, the target's certificate must be copied into Cryostat's /truststore directory before Cryostat's startup. If Cryostat attempts to connect to an SSL-enabled target and no matching trusted certificate is found then the connection attempt will fail.

USER AUTHENTICATION / AUTHORIZATION

Cryostat has multiple authz manager implementations for handling user authentication and authorization against various platforms and mechanisms. This can be controlled using an environment variable (see the RUN section above), or automatically using platform detection.

In all scenarios, the presence of an auth manager (other than NoopAuthManager) causes Cryostat to expect a token or credentials via an Authorization header on all potentially sensitive requests, ex. recording creations and downloads, report generations.

The OpenShiftPlatformClient.OpenShiftAuthManager uses token authentication. These tokens are passed through to the OpenShift API for authz and this result determines whether Cryostat accepts the request.

The BasicAuthManager uses basic credential authentication configured with a standard Java properties file at $CRYOSTAT_CONFIG_PATH/cryostat-users.properties. The credentials stored in the Java properties file are the user name and a SHA-256 sum hex of the user's password. The property file contents should look like:

user1=abc123
user2=def987

Where abc123 and def987 are substituted for the SHA-256 sum hexes of the desired user passwords. These can be obtained by ex. echo -n PASS | sha256sum | cut -d' ' -f1.

Token-based auth managers expect an HTTP Authorization: Bearer TOKEN header and a Sec-WebSocket-Protocol: base64url.bearer.authorization.cryostat.${base64(TOKEN)} WebSocket SubProtocol header. The token is never stored in any form, only kept in-memory long enough to process the external token validation.

Basic credentials-based auth managers expect an HTTP Authorization: Basic ${base64(user:pass)} header and a Sec-WebSocket-Protocol: basic.authorization.cryostat.${base64(user:pass)} WebSocket SubProtocol header.

If no appropriate auth manager is configured or can be automatically determined then the fallback is the NoopAuthManager, which does no external validation calls and simply accepts any provided token or credentials.

INCOMING JMX CONNECTION AUTHENTICATION

JMX connections into cryostat are secured using the default username "cryostat" and a randomly generated password. The environment variables CRYOSTAT_RJMX_USER and CRYOSTAT_RJMX_PASS can be used to override the default username and specify a password.

API

Cryostat exposes an HTTP API that provides the backing for its web interface, but is also intended as an automation or extension point for external clients. For details about this API see HTTP_API.md, GRAPHQL.md, and DISCOVERY_PLUGINS.md.