search

open-telemetry / opentelemetry-demo

This repository contains the OpenTelemetry Astronomy Shop, a microservice-based distributed system intended to illustrate the implementation of OpenTelemetry in a near real-world environment.
https://opentelemetry.io/docs/demo/
Apache License 2.0
1.68k stars 1.06k forks source link

`docker_stats` errors #1677

Open julianocosta89 opened 1 month ago

julianocosta89 commented 1 month ago

Bug Report

9b5478b307a44da1bd6a97154b878aed92d5e717

Symptom

When building and starting the demo I'm getting the following errors on the collector logs:

2024-07-22T09:26:55.491Z    error   docker@v0.105.0/docker.go:194   Could not parse docker containerStats for container id  {"kind": "receiver", "name": "docker_stats", "data_type": "metrics", "id": "fa060dee854f4a1fc383b84ae1fc694080be8b0d207450f887ce78d8d44f723e", "error": "context canceled"}
github.com/open-telemetry/opentelemetry-collector-contrib/internal/docker.(*Client).toStatsJSON
    github.com/open-telemetry/opentelemetry-collector-contrib/internal/docker@v0.105.0/docker.go:194
github.com/open-telemetry/opentelemetry-collector-contrib/internal/docker.(*Client).FetchContainerStatsAsJSON
    github.com/open-telemetry/opentelemetry-collector-contrib/internal/docker@v0.105.0/docker.go:144
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/dockerstatsreceiver.(*metricsReceiver).scrapeV2.func1
    github.com/open-telemetry/opentelemetry-collector-contrib/receiver/dockerstatsreceiver@v0.105.0/receiver.go:92

2024-07-22T09:26:55.502Z    error   scraperhelper/scrapercontroller.go:197  Error scraping metrics  {"kind": "receiver", "name": "docker_stats", "data_type": "metrics", "error": "context canceled", "scraper": "docker_stats"}
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport
    go.opentelemetry.io/collector/receiver@v0.105.0/scraperhelper/scrapercontroller.go:197
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1
    go.opentelemetry.io/collector/receiver@v0.105.0/scraperhelper/scrapercontroller.go:173

Reproduce

Pull the latest changes and run:

docker compose build
docker compose up -d
docker logs otel-col -f
julianocosta89 commented 1 month ago

@rogercoll do you see this issue on your end as well?

rogercoll commented 1 month ago

@julianocosta89 Not on my end. I reckon the issue might be when the dockerstats receiver trying to retrieve the stats of your container with id fa060dee854f4a1fc383b84ae1fc694080be8b0d207450f887ce78d8d44f723e. What is the state of this container? (docker inspect fa060dee854f4a1fc383b84ae1fc694080be8b0d207450f887ce78d8d44f723e) Is it an opentelemetry-demo container or from another environment?

julianocosta89 commented 1 month ago

It is from the demo yes. I've re-ran the demo and checked the errors and it raises a bunch of different IDs, I've just pasted one.

Eg:

❯ docker inspect 85775
[
    {
        "Id": "857754a7442f5c1483e8e81b0ec363bc2cd0db9ada6ccc165b29b1f094e8dd96",
        "Created": "2024-07-24T08:51:48.892932637Z",
        "Path": "/app/shippingservice",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 10350,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2024-07-24T08:51:50.169065304Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:a16932aa56554f09a6abade21f418093021c0886dd3a4e04798a44d48f89058c",
        "ResolvConfPath": "/var/lib/docker/containers/857754a7442f5c1483e8e81b0ec363bc2cd0db9ada6ccc165b29b1f094e8dd96/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/857754a7442f5c1483e8e81b0ec363bc2cd0db9ada6ccc165b29b1f094e8dd96/hostname",
        "HostsPath": "/var/lib/docker/containers/857754a7442f5c1483e8e81b0ec363bc2cd0db9ada6ccc165b29b1f094e8dd96/hosts",
        "LogPath": "/var/lib/docker/containers/857754a7442f5c1483e8e81b0ec363bc2cd0db9ada6ccc165b29b1f094e8dd96/857754a7442f5c1483e8e81b0ec363bc2cd0db9ada6ccc165b29b1f094e8dd96-json.log",
        "Name": "/shipping-service",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {
                    "max-file": "2",
                    "max-size": "5m",
                    "tag": "{{.Name}}"
                }
            },
            "NetworkMode": "opentelemetry-demo",
            "PortBindings": {
                "50050/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "0"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "unless-stopped",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "ConsoleSize": [
                0,
                0
            ],
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "private",
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": [],
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 20971520,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "MemoryReservation": 0,
            "MemorySwap": 41943040,
            "MemorySwappiness": null,
            "OomKillDisable": null,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware",
                "/sys/devices/virtual/powercap"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/f97a4a48255cc4d685e8111a8df66a295061997d5ef3d8f6b7d6fc5a28c05145-init/diff:/var/lib/docker/overlay2/vkoudxzjfg26swicotvfpwz07/diff:/var/lib/docker/overlay2/ld0z3iz6tax5ery1jv9dxrpp0/diff:/var/lib/docker/overlay2/4b4lfbidxgsoafuntz7lobpdy/diff:/var/lib/docker/overlay2/b4983cc9499d7fb98a4cfd14a33a2afbc0f58d12f14ae8c5e267d7b4b2cb0bf5/diff",
                "MergedDir": "/var/lib/docker/overlay2/f97a4a48255cc4d685e8111a8df66a295061997d5ef3d8f6b7d6fc5a28c05145/merged",
                "UpperDir": "/var/lib/docker/overlay2/f97a4a48255cc4d685e8111a8df66a295061997d5ef3d8f6b7d6fc5a28c05145/diff",
                "WorkDir": "/var/lib/docker/overlay2/f97a4a48255cc4d685e8111a8df66a295061997d5ef3d8f6b7d6fc5a28c05145/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [],
        "Config": {
            "Hostname": "857754a7442f",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "ExposedPorts": {
                "50050/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "QUOTE_SERVICE_ADDR=http://quoteservice:8090",
                "OTEL_EXPORTER_OTLP_ENDPOINT=http://otelcol:4317",
                "OTEL_RESOURCE_ATTRIBUTES=service.namespace=opentelemetry-demo,service.version=1.11.0",
                "OTEL_SERVICE_NAME=shippingservice",
                "SHIPPING_SERVICE_PORT=50050",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": null,
            "Image": "ghcr.io/open-telemetry/demo:latest-shippingservice",
            "Volumes": null,
            "WorkingDir": "/app",
            "Entrypoint": [
                "/app/shippingservice"
            ],
            "OnBuild": null,
            "Labels": {
                "com.docker.compose.config-hash": "cc6d75eae8d68c2311f4f9e92debff17e0007ef15828f3f23fea72777d56f5d2",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.depends_on": "otelcol:service_started:false",
                "com.docker.compose.image": "sha256:a16932aa56554f09a6abade21f418093021c0886dd3a4e04798a44d48f89058c",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "opentelemetry-demo",
                "com.docker.compose.project.config_files": "/Users/juliano.costa/workspace/opentelemetry-demo/docker-compose.yml",
                "com.docker.compose.project.environment_file": "/Users/juliano.costa/workspace/opentelemetry-demo/.env,/Users/juliano.costa/workspace/opentelemetry-demo/.env.override",
                "com.docker.compose.project.working_dir": "/Users/juliano.costa/workspace/opentelemetry-demo",
                "com.docker.compose.service": "shippingservice",
                "com.docker.compose.version": "2.23.3"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "66c1260db5448d30f85f4f7b16aff69a4aafc4442e0f76e635f71c385b6f6179",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "50050/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "50511"
                    }
                ]
            },
            "SandboxKey": "/var/run/docker/netns/66c1260db544",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "opentelemetry-demo": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "shipping-service",
                        "shippingservice",
                        "857754a7442f"
                    ],
                    "MacAddress": "02:42:ac:12:00:0c",
                    "NetworkID": "263aa056fe48f70a72d21a193ec506c59d8d16d15a6d3dab3a11e894aef225bd",
                    "EndpointID": "05ad3f3671447b16cb1b605d8bc29cd223407a56983030b7f928c986e5eb3c80",
                    "Gateway": "172.18.0.1",
                    "IPAddress": "172.18.0.12",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "DriverOpts": null
                }
            }
        }
    }
]

But I've asked a friend that uses Ubuntu to try it out, and it is actually working fine there. It seems this is an issue with Docker on Mac 😞

rogercoll commented 1 month ago

It seems this is an issue with Docker on Mac 😞

I agree. Actually, it seems that Darwin is not supported by the receiver: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/dockerstatsreceiver#docker-stats-receiver

puckpuck commented 1 month ago

Many people develop on the Mac platform. It's important that we have a solution to disable the docker_stats receiver for them.

julianocosta89 commented 1 month ago

@puckpuck I've mentioned on the SIG meeting that the most fascinating thing is that I do get the docker metrics. I get the errors, but surprisingly, it works 🙃