amir20 / dozzle

Realtime log viewer for docker containers.
https://dozzle.dev/
MIT License
5.7k stars 287 forks source link

No CPU and no Memory is shown #3103

Closed Mephisto090 closed 2 months ago

Mephisto090 commented 2 months ago

Describe the bug In the overview of all "external" servers, no CPU and no memory is shown. Ubuntuserver is the host on which Dozzle runs

Screenshots image

Desktop (please complete the following information):

Dozzle-Docker:

services:
  dozzle:
    container_name: Dozzle
    image: amir20/dozzle:latest   
    environment:
      DOZZLE_REMOTE_HOST: tcp://XXX.XXX.XXX.XX:2375|Nginx-Server,tcp://XXX.XXX.XXX.XX:2375|Webserver,tcp://XXX.XXX.XXX.XX0:2375|Mailserver,tcp://XXX.XXX.XXX.XX:2375|Ubuntuserver 2
      DOZZLE_HOSTNAME: Ubuntuserver
      DOZZLE_ENABLE_ACTIONS: true 
      #DOZZLE_AUTH_PROVIDER: simple
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      #- /data:/data
    ports:
      - 8000:8080
    healthcheck:
      test: ["CMD", "/dozzle", "healthcheck"]
      interval: 3s
      timeout: 30s
      retries: 5
      start_period: 30s

Socket-Proxy: (Ubuntuserver 2)

services:
  socket-proxy:
    container_name: Socket-Proxy
    image: lscr.io/linuxserver/socket-proxy:latest
    logging:
      driver: journald
    restart: unless-stopped
    tmpfs:
      - /run
    privileged: true
    ports:
      - 2375:2375
    environment:
      - ALLOW_START=1 #optional
      - ALLOW_STOP=1 #optional
      - ALLOW_RESTARTS=01 #optional
      - CONTAINERS=1  #optional
      - INFO=1 #optional
      - SYSLOGD=1 #optional
      - SYSTEM=1  #optional   
      - VERSION=1 #optional
      - EVENTS=1 #optional
      - NETWORKS=1 #optional
      - PING=1 #optional
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /dev/log:/dev/log
    healthcheck:
      test: ["CMD", "nc", "-z", "127.0.0.1", "2375"]
      interval: 5s
      retries: 5
      start_period: 5s
      start_interval: 5s
amir20 commented 2 months ago

I wouldn't know too much about how socket-proxy works. And supporting it is out of my scope. But sounds you do have INFO=1. Which should work. I get this information from docker system info so if it's not there then Docker might not have access to it.

I won't be supporting questions on socket-proxy too much because it takes significant amount of time to investigate. Agents should have the data, but if they don't, I can at least show debug information.

See discussion at #3066 as why agents are also more performant.

Mephisto090 commented 2 months ago

iI switched to the Dozzle Agent and it works. I just get the wrong RAM displayed on one server :(

image

image

amir20 commented 2 months ago

Interesting. What does docker system info say?

Mephisto090 commented 2 months ago

This :)

image

root@server:/server# docker system info
Client: Docker Engine - Community
 Version:    27.0.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.15.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.28.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 5
  Running: 2
  Paused: 0
  Stopped: 3
 Images: 7
 Server Version: 27.0.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae71819c4f5e67bb4d5ae76a6b735f29cc25774e
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.8.0-38-generic
 Operating System: Ubuntu 24.04 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 14.57GiB
 Name: server
 ID: 9f76cc40-55fe-46ce-bc48-5f0f5f77df68
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
amir20 commented 2 months ago

When I get a chance I'll add some debug logs and we can check what Docker returns. I just fetch this data so can't imagine something is happening from my side.

Mephisto090 commented 2 months ago

Is the Env variable DOZZLE_LEVEL: debug? In preparation for the debug messages. Server and Agent on Debug?

Its4Nik commented 2 months ago

I tried changing every setting from docker socket proxy to "1" and it still didnt have the right usage reporting, but i could start and stop the docker containers on that other host.

Different Host, "fin-4": image

Fin-1 Instance of dozzle image

So as we can see we would probably have to change something else in the docker socket proxy.

Mazbe we would have to mount some sort of volume or system path...

amir20 commented 2 months ago

While debugging this, I think I may have found an issue. But I'll need a few days to look at it. So keeping this open.

amir20 commented 2 months ago

Oops. Closed it early. I am away for a while, but I wonder if amir20/dozzle:master fixed it? I made a mistake and didn't create a PR. 🤐

amir20 commented 2 months ago

Use https://dozzle.dev/guide/debugging if it doesn't work

Mephisto090 commented 2 months ago

Oops. Closed it early. I am away for a while, but I wonder if amir20/dozzle:master fixed it? I made a mistake and didn't create a PR. 🤐

Should I try it now on the master to see if the RAM fix works? If so, on the server and agent?

EDIT: I tried it with master and unfortunately the RAM error is still there. image

Dozzle-Agent: image

Dozzle-Server: image

The Agent is correctly pulling out the RAM (15639183360 B ~ 15.63918336 GB). However, the server somehow gets an incorrect value (2754281472 B ~ 2.754281472 GB).

amir20 commented 2 months ago

@Its4Nik and @Mephisto090 have different issues. I think I fixed @Its4Nik issue where the stats weren't showing up at all on remote hosts.

For @Mephisto090, I am a little stumped because it looks like it is reading correctly. I use javascript to convert it to 2.7GB so there might be something wrong there. I am going to copy the exact number of bytes and try it out to see if it does the same thing for me.

amir20 commented 2 months ago

I think I have a hunch now. It's the difference between int32 and int64. 🤯

Added https://github.com/amir20/dozzle/pull/3115

@Mephisto090 can you try amir20/dozzle:pr-3115? You do need to redeploy the agent and UI since proto messages have changed.

Officially most random bug I have investigated.

Mephisto090 commented 2 months ago

image

This seems to have worked!

I go back to latest and wait for the actual update. Is master or latest further, i.e. do I have a beta version with master?

amir20 commented 2 months ago

Master is dev

amir20 commented 2 months ago

Merged. Still working on other fixes to release.

iYoungblood commented 1 month ago

Had a 3 week old version and also noticed this, but didn't think much of it. Today, I found I had a container with a memory leak and found it was an issue for months but didnt have a great way to find it. Just came looking for this, did a pull and the memory function AND host stats are working. Thank you!

amir20 commented 1 month ago

:)