provide appropriate resource attributes for OpenTelemetry

[donbourne]

In order to identify data sources, OpenTelemetry provides the ability to set resource attributes. These resource attributes are sent with the data signals to downstream consumers. We need to be able to run Liberty, with OpenTelemetry, using either our mpTelemetry-2.0 feature or with the OTel agent.

When running with the OTel java instrumentation agent, this is a typical set of resource attributes (as seen from the logging output from an OTel collector):

otel-collector  | 2024-05-31T15:00:25.173Z  info    MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "logging", "resource metrics": 1, "metrics": 12, "data points": 39}
otel-collector  | 2024-05-31T15:00:25.174Z  info    ResourceMetrics #0
otel-collector  | Resource SchemaURL: https://opentelemetry.io/schemas/1.24.0
otel-collector  | Resource attributes:
otel-collector  |      -> container.id: Str(963bb39fb7126fe0fcc45e21608cb2d6c68106b91c497bdfd1aa1701d1b263c5)
otel-collector  |      -> host.arch: Str(aarch64)
otel-collector  |      -> host.name: Str(21227821ec43)
otel-collector  |      -> os.description: Str(Linux 6.8.8-300.fc40.aarch64)
otel-collector  |      -> os.type: Str(linux)
otel-collector  |      -> process.command_args: Slice(["/opt/java/openjdk/bin/java","-javaagent:/opt/ol/wlp/bin/tools/ws-javaagent.jar","-Djava.awt.headless=true","-Djdk.attach.allowAttachSelf=true","-javaagent:/config/lib/opentelemetry-javaagent.jar","--add-exports","java.base/sun.security.action=ALL-UNNAMED","--add-exports","java.naming/com.sun.jndi.ldap=ALL-UNNAMED","--add-exports","java.naming/com.sun.jndi.url.ldap=ALL-UNNAMED","--add-exports","jdk.naming.dns/com.sun.jndi.dns=ALL-UNNAMED","--add-exports","jdk.naming.dns/com.sun.jndi.url.dns=ALL-UNNAMED","--add-exports","java.security.jgss/sun.security.krb5.internal=ALL-UNNAMED","--add-exports","jdk.attach/sun.tools.attach=ALL-UNNAMED","--add-opens","java.base/java.util=ALL-UNNAMED","--add-opens","java.base/java.lang=ALL-UNNAMED","--add-opens","java.base/java.util.concurrent=ALL-UNNAMED","--add-opens","java.base/java.io=ALL-UNNAMED","--add-opens","java.base/java.nio=ALL-UNNAMED","--add-opens","java.base/sun.nio.ch=ALL-UNNAMED","--add-opens","java.naming/javax.naming.spi=ALL-UNNAMED","--add-opens","java.naming/com.sun.naming.internal=ALL-UNNAMED","--add-opens","jdk.naming.rmi/com.sun.jndi.url.rmi=ALL-UNNAMED","--add-opens","java.naming/javax.naming=ALL-UNNAMED","--add-opens","java.rmi/java.rmi=ALL-UNNAMED","--add-opens","java.sql/java.sql=ALL-UNNAMED","--add-opens","java.management/javax.management=ALL-UNNAMED","--add-opens","java.base/java.lang.reflect=ALL-UNNAMED","--add-opens","java.desktop/java.awt.image=ALL-UNNAMED","--add-opens","java.base/java.security=ALL-UNNAMED","--add-opens","java.base/java.net=ALL-UNNAMED","--add-opens","java.base/java.text=ALL-UNNAMED","--add-opens","java.base/sun.net.www.protocol.https=ALL-UNNAMED","--add-exports","jdk.management.agent/jdk.internal.agent=ALL-UNNAMED","--add-exports","java.base/jdk.internal.vm=ALL-UNNAMED","-jar","/opt/ol/wlp/bin/tools/ws-server.jar","defaultServer"])
otel-collector  |      -> process.executable.path: Str(/opt/java/openjdk/bin/java)
otel-collector  |      -> process.pid: Int(1)
otel-collector  |      -> process.runtime.description: Str(Eclipse OpenJ9 Eclipse OpenJ9 VM openj9-0.43.0)
otel-collector  |      -> process.runtime.name: Str(IBM Semeru Runtime Open Edition)
otel-collector  |      -> process.runtime.version: Str(21.0.2+13-LTS)
otel-collector  |      -> service.instance.id: Str(4b43a183-7a28-4cb1-b2b0-4cbc16fb5b2a)
otel-collector  |      -> service.name: Str(adservice)
otel-collector  |      -> telemetry.distro.name: Str(opentelemetry-java-instrumentation)
otel-collector  |      -> telemetry.distro.version: Str(2.3.0)
otel-collector  |      -> telemetry.sdk.language: Str(java)
otel-collector  |      -> telemetry.sdk.name: Str(opentelemetry)
otel-collector  |      -> telemetry.sdk.version: Str(1.37.0)

We need to ensure we have a similarly complete list of resource attributes when running without an OTel java instrumentation agent.

For guidance on how to set some of these, see https://opentelemetry.io/docs/languages/java/resources/

---

Should this just be provided by the user of OL by setting otel.resource.attributes environment variables / system properties, or should we set these values programmatically? I'm thinking that if we can easily set them programmatically, that makes more sense, as it's difficult to get all the data for host/os/process otherwise, and it would be nicer if user just has to concern themselves with custom resource attributes they may want to add.

[pgunapal]

The following is the resource attributes we get when running Liberty, with OpenTelemetry, using mpTelemetry-2.0, where we are emitting Liberty logs via OTLP to the Open Telemetry Collector:

From the above OTel java instrumentation agent, quite a few fields are missing, like host and OS/process information. The resource attributes we get now with mpTelemetry-2.0, is just the configured service.name and information the SDK provides.

...
{
    "resourceLogs": [
        {
            "resource": {
                "attributes": [
                    {
                        "key": "service.name",
                        "value": {
                            "stringValue": "openliberty"
                        }
                    },
                    {
                        "key": "telemetry.sdk.language",
                        "value": {
                            "stringValue": "java"
                        }
                    },
                    {
                        "key": "telemetry.sdk.name",
                        "value": {
                            "stringValue": "opentelemetry"
                        }
                    },
                    {
                        "key": "telemetry.sdk.version",
                        "value": {
                            "stringValue": "1.35.0"
                        }
                    }
                ]
            },
 ...