Add docs on using spring actuator to debug issues

[planetf1]

Spring actuator can be enabled by setting the appropriate property:

java "-Dmanagement.endpoints.web.exposure.include=*" -jar open-metadata-implementation/server-chassis/server-chassis-spring/build/libs/server-chassis-spring-4.0-SNAPSHOT.jar &

Once run here are some examples of how to list metrics. We can also gather heapdumps, and check health.

All responses are json

The prometheus endpoint can be consumed by prometheus for more sophisticated use

Capturing these as rough notes, and will refine into proper docs

[planetf1]

➜  egeria git:(issue7424) ✗ http --verify=no https://localhost:9443/actuator
HTTP/1.1 200
Connection: keep-alive
Content-Type: application/vnd.spring-boot.actuator.v3+json
Date: Mon, 20 Mar 2023 12:42:25 GMT
Keep-Alive: timeout=60
Transfer-Encoding: chunked

{
    "_links": {
        "beans": {
            "href": "https://localhost:9443/actuator/beans",
            "templated": false
        },
        "caches": {
            "href": "https://localhost:9443/actuator/caches",
            "templated": false
        },
        "caches-cache": {
            "href": "https://localhost:9443/actuator/caches/{cache}",
            "templated": true
        },
        "conditions": {
            "href": "https://localhost:9443/actuator/conditions",
            "templated": false
        },
        "configprops": {
            "href": "https://localhost:9443/actuator/configprops",
            "templated": false
        },
        "configprops-prefix": {
            "href": "https://localhost:9443/actuator/configprops/{prefix}",
            "templated": true
        },
        "env": {
            "href": "https://localhost:9443/actuator/env",
            "templated": false
        },
        "env-toMatch": {
            "href": "https://localhost:9443/actuator/env/{toMatch}",
            "templated": true
        },
        "health": {
            "href": "https://localhost:9443/actuator/health",
            "templated": false
        },
        "health-path": {
            "href": "https://localhost:9443/actuator/health/{*path}",
            "templated": true
        },
        "heapdump": {
            "href": "https://localhost:9443/actuator/heapdump",
            "templated": false
        },
        "info": {
            "href": "https://localhost:9443/actuator/info",
            "templated": false
        },
        "loggers": {
            "href": "https://localhost:9443/actuator/loggers",
            "templated": false
        },
        "loggers-name": {
            "href": "https://localhost:9443/actuator/loggers/{name}",
            "templated": true
        },
        "mappings": {
            "href": "https://localhost:9443/actuator/mappings",
            "templated": false
        },
        "metrics": {
            "href": "https://localhost:9443/actuator/metrics",
            "templated": false
        },
        "metrics-requiredMetricName": {
            "href": "https://localhost:9443/actuator/metrics/{requiredMetricName}",
            "templated": true
        },
        "prometheus": {
            "href": "https://localhost:9443/actuator/prometheus",
            "templated": false
        },
        "scheduledtasks": {
            "href": "https://localhost:9443/actuator/scheduledtasks",
            "templated": false
        },
        "self": {
            "href": "https://localhost:9443/actuator",
            "templated": false
        },
        "threaddump": {
            "href": "https://localhost:9443/actuator/threaddump",
            "templated": false
        }
    }
}

➜  egeria git:(issue7424) ✗ kill %1
➜  egeria git:(issue7424) ✗ http --verify=no https://localhost:9443/actuator/metrics
HTTP/1.1 200
Connection: keep-alive
Content-Type: application/vnd.spring-boot.actuator.v3+json
Date: Mon, 20 Mar 2023 12:43:14 GMT
Keep-Alive: timeout=60
Transfer-Encoding: chunked

{
    "names": [
        "application.ready.time",
        "application.started.time",
        "disk.free",
        "disk.total",
        "executor.active",
        "executor.completed",
        "executor.pool.core",
        "executor.pool.max",
        "executor.pool.size",
        "executor.queue.remaining",
        "executor.queued",
        "http.server.requests",
        "http.server.requests.active",
        "jvm.buffer.count",
        "jvm.buffer.memory.used",
        "jvm.buffer.total.capacity",
        "jvm.classes.loaded",
        "jvm.classes.unloaded",
        "jvm.compilation.time",
        "jvm.gc.live.data.size",
        "jvm.gc.max.data.size",
        "jvm.gc.memory.allocated",
        "jvm.gc.memory.promoted",
        "jvm.gc.overhead",
        "jvm.gc.pause",
        "jvm.info",
        "jvm.memory.committed",
        "jvm.memory.max",
        "jvm.memory.usage.after.gc",
        "jvm.memory.used",
        "jvm.threads.daemon",
        "jvm.threads.live",
        "jvm.threads.peak",
        "jvm.threads.states",
        "logback.events",
        "process.cpu.usage",
        "process.files.max",
        "process.files.open",
        "process.start.time",
        "process.uptime",
        "system.cpu.count",
        "system.cpu.usage",
        "system.load.average.1m",
        "tomcat.sessions.active.current",
        "tomcat.sessions.active.max",
        "tomcat.sessions.alive.max",
        "tomcat.sessions.created",
        "tomcat.sessions.expired",
        "tomcat.sessions.rejected"
    ]
}

➜  egeria git:(issue7424) ✗ http --verify=no https://localhost:9443/actuator/metrics/jvm.memory.used
HTTP/1.1 200
Connection: keep-alive
Content-Disposition: inline;filename=f.txt
Content-Type: application/vnd.spring-boot.actuator.v3+json
Date: Mon, 20 Mar 2023 12:44:23 GMT
Keep-Alive: timeout=60
Transfer-Encoding: chunked

{
    "availableTags": [
        {
            "tag": "area",
            "values": [
                "heap",
                "nonheap"
            ]
        },
        {
            "tag": "id",
            "values": [
                "CodeHeap 'profiled nmethods'",
                "G1 Old Gen",
                "CodeHeap 'non-profiled nmethods'",
                "G1 Survivor Space",
                "Compressed Class Space",
                "Metaspace",
                "G1 Eden Space",
                "CodeHeap 'non-nmethods'"
            ]
        }
    ],
    "baseUnit": "bytes",
    "description": "The amount of used memory",
    "measurements": [
        {
            "statistic": "VALUE",
            "value": 139875728.0
        }
    ],
    "name": "jvm.memory.used"
}

[planetf1]

In our helm charts this can be enabled by setting the 'jvmopts' property associated with many of our containers

jvmopts: "-Dmanagement.endpoints.web.exposure.include=*"

[planetf1]

On Red Hat Openshift, prometheus can monitor user applications.

See

• https://docs.openshift.com/container-platform/4.12/monitoring/configuring-the-monitoring-stack.html