search

amir20 / dozzle

Realtime log viewer for docker containers.
https://dozzle.dev/
MIT License
6.15k stars 308 forks source link

System becomes irresponsive, 100% resource utilization #3085

Closed mistrello96 closed 3 months ago

mistrello96 commented 4 months ago

Describe the bug System becomes irresponsive as soon as i start the dozzle container, with 100% CPU and RAM usage. Docker stop/down does not takes effect (probably cause the system is not responisve), only a system restart solves the problem

To Reproduce Steps to reproduce the behavior:

Simply execute the docker/docker compose up command in the Readme

Expected behavior The system should keep working normally

Desktop:

mistrello96 commented 4 months ago

I tried starting the container without mountig the docker.sock and everything is fine (OC dozzle cannot detect any docker engine).

Also triead using a proxy for the sock but same result

As soon as I try to mount the sock, even in read only, the systems freeze

amir20 commented 3 months ago

This has been running for thousand of other people so most likely something is different about your set up.

  1. How many containers do you have?
  2. Did this used to work? Did something break recently?
  3. Does it work if you try locally or on a different host?

You didn't provide enough details so I can't really help much. As the bug template suggests, please provide logs by also enabling DEBUG mode.

mistrello96 commented 3 months ago

Giving a maximum resource limit in the deploy settings semms to have fixed the problem, sorry to have bothered

amir20 commented 3 months ago

Can you show what was breaking it for future reference.

mistrello96 commented 3 months ago

I'm reopening the issue since, after a reboot the problem is back again (so the resource limit is not effective)

  1. I'm having currently 30ish container
  2. It is the first time i'm setting it up, but as i said I tried different version of dozzle and the problem remains
  3. I have no other host to try it on

Here is attached the log file in debug mode, I cannot see anything abnormal. As soon as I start the container, dozzle can be reached correctly but the resource starts to filling up until the entire system becomes irresponsive.

Seems more like the problem is not caused directly by dozzle container, but something that confict with it.

Do you have any suggestion on how to further investigate the problem?

log.txt

mistrello96 commented 3 months ago

@amir20 probably I have the answer. I had A TON of stopped container and I thought that dozzle only monitored the active ones. Digging into the option, I found the "show stopped containers" option, which gave me the idea. Cleaning the stopped containers seems to solve the issue

amir20 commented 3 months ago

@amir20 probably I have the answer. I had A TON of stopped container and I thought that dozzle only monitored the active ones.

What is a TON? I have tested with 1000s but if you have hundreds of thousands, I could see that breaking. I do loop through all the containers so I can see this being a performance issue.

Can you create a new performance issue with the details so I can reproduce?

mistrello96 commented 3 months ago

What is a TON? I have tested with 1000s but if you have hundreds of thousands, I could see that breaking. I do loop through all the containers so I can see this being a performance issue.

I wolud say something near 30K containers (cronjob running every 10 minutes for the past few months and not deleting the exited container), but i already pruned them so I cannot be more precise.

I will definelly create a perf issue and do my best to fill all the appropriate info!

amir20 commented 3 months ago

Thanks. A rough idea is good with me. I imagine 30k containers would break Dozzle 🫠.

I am personally not sure what the fix would be. But we can continue the convo on the perf ticket.