Agent crashing shortly after running

grosejon commented 1 week ago

🔍 Check for existing issues

[X] Completed

How is Dozzle deployed?

Agents

📦 Dozzle version

8.4.1

✅ Command used to run Dozzle

services: dozzle-agent: image: amir20/dozzle:latest container_name: dozzle-agent-beelink command: agent networks:

socket_proxy healthcheck: test: ["CMD", "/dozzle", "healthcheck"] interval: 5s retries: 5 start_period: 5s start_interval: 5s environment:
DOZZLE_HOSTNAME=beelink
DOCKER_HOST=tcp://socket-proxy:2375 ports:
7007:7007

networks: socket_proxy: external: true

🐛 Describe the bug / provide steps to reproduce it

After running for a short time, the agent will crash. I'm running the main instance on a synology, and then two agents: one on another synology (working), and the other on a beelink mini pc running ubuntu (error). The two agents are using the same compose, but the beelink one keeps crashing.

💻 Environment

Client: Docker Engine - Community Version: 27.2.0 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.16.2 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.29.2 Path: /usr/libexec/docker/cli-plugins/docker-compose

Server: Containers: 6 Running: 5 Paused: 0 Stopped: 1 Images: 10 Server Version: 27.2.0 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: systemd Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog Swarm: inactive Runtimes: runc io.containerd.runc.v2 Default Runtime: runc Init Binary: docker-init containerd version: 472731909fa34bd7bc9c087e4c27943f9835f111 runc version: v1.1.13-0-g58aa920 init version: de40ad0 Security Options: apparmor seccomp Profile: builtin cgroupns Kernel Version: 6.8.0-41-generic Operating System: Ubuntu 24.04.1 LTS OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 15.4GiB Name: beelink Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false

📸 If applicable, add screenshots to help explain your bug

No response

📜 If applicable, attach your Dozzle logs. You many need to enable debug mode. See https://dozzle.dev/guide/debugging.

2024-09-06T03:34:16Z 2024-09-06T03:34:16Z INF 2024-09-06T03:34:16Z 2024-09-06T03:34:16Z INF 2024-09-06T03:34:16Z 2024-09-06T03:34:16Z INF 2024-09-06T03:34:16Z 2024-09-06T03:34:16Z INF Dozzle agent version v8.4.1 2024-09-06T03:34:16Z 2024-09-06T03:34:16Z INF 2024-09-06T03:34:16Z 2024-09-06T03:34:16Z INF 2024-09-06T03:34:16Z 2024-09-06T03:34:16Z INF 2024-09-06T03:34:16Z 2024-09-06T03:34:16Z INF Agent listening on [::]:7007 panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xb21fb7] goroutine 20421 [running]: github.com/amir20/dozzle/internal/agent.logEventToPb(0x0) /dozzle/internal/agent/server.go:332 +0x17 github.com/amir20/dozzle/internal/agent.(server).LogsBetweenDates(0xc00042eea0, 0xc0008587d0, {0x10cf000, 0xc0007b3920}) /dozzle/internal/agent/server.go:92 +0x3be github.com/amir20/dozzle/internal/agent/pb._AgentService_LogsBetweenDates_Handler({0xe4ce20, 0xc00042eea0}, {0x10cd4c8, 0xc0005500f0}) /dozzle/internal/agent/pb/rpc_grpc.pb.go:339 +0x107 google.golang.org/grpc.(Server).processStreamingRPC(0xc000160600, {0x10ca0c8, 0xc00066b3e0}, {0x10d0180, 0xc0002afa00}, 0xc00040fb00, 0xc00042ef30, 0x16ce300, 0x0) /go/pkg/mod/google.golang.org/grpc@v1.66.0/server.go:1687 +0x1208 google.golang.org/grpc.(Server).handleStream(0xc000160600, {0x10d0180, 0xc0002afa00}, 0xc00040fb00) /go/pkg/mod/google.golang.org/grpc@v1.66.0/server.go:1808 +0xe3a google.golang.org/grpc.(Server).serveStreams.func2.1() /go/pkg/mod/google.golang.org/grpc@v1.66.0/server.go:1029 +0x7f created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 63 /go/pkg/mod/google.golang.org/grpc@v1.66.0/server.go:1040 +0x125

grosejon commented 1 week ago

Sorry but I won't be clicking that random link

amir20 commented 1 week ago

Not sure who @brunoR1000 is. Deleting the comment.

amir20 commented 1 week ago

Looked at that error on line 332. Seems to suggest that the log is null. I'll look closer tomorrow.

I might add more logging and have you test a build since I haven't seen this error before.

amir20 commented 1 week ago

@grosejon based on the error, it is happening here https://github.com/amir20/dozzle/blob/0d58f2bfcade224262c4617c93a2a73830c0aedb/internal/agent/server.go#L332

However, based on the code, it is impossible for this to happen because the log has to exist.

I have created a pull request which can tested with amir20/dozzle:pr-3259. When message is nil it should do a fatal log like:

agent server: message is nil event={"c":"d6ce877e3ca3","id":2879538069,"m":"Egestas sed sed risus pretium quam vulputate","s":"stderr","ts":1725637426530}

Can you run this PR and let me know when you see that error? Ideally, I want to reproduce it. Most likely, one of your containers has weird log. You could see which container is producing this log with "c":"d6ce877e3ca3" part.

grosejon commented 1 week ago

@amir20 I'm not sure what happened, but it has stopped crashing. I just restarted a few times and it seems to be working for a couple days now.

amir20 commented 1 week ago

Well that's good. I suspect there as an empty log or something that I have never seen before with Dozzle. I am going to merge the PR anyway with fatal. Which means it would exit with an explicit error message. If it happens again, I would need to see what is causing it.

amir20 / dozzle