amir20 / dozzle

Realtime log viewer for docker containers.
https://dozzle.dev/
MIT License
5.02k stars 266 forks source link

Unable to access over 9 dozzle hosts/agents #3072

Closed beastarman74 closed 1 week ago

beastarman74 commented 1 week ago

Describe the bug Unable to access dozzle hosts or agents after the first 9 are added. They cannot be accessed from the host/agent list or the drop down box. This appears to be the same in dozzle mobile and desktop view.

To Reproduce Steps to reproduce the behavior:

  1. run one or more full version of dozzle on docker hosts
  2. run dozzle agent mode on over 10 Docker hosts including those running full dozzle
  3. Open the dozzle Web page on one of the host's running full dozzle
  4. Attempt to switch to the 10th or 10th plus dozzle agent, it will not be accessible unless one of the first 9 is shutdown and the full dozzle instance restarted.

Expected behavior All dozzle agent instances can be accessed from the dozzle Web page.

Screenshots Screenshot_20240708_002453 Screenshot_20240708_001149 Screenshot_20240708_000926

Desktop (please complete the following information):

If applicable include logs with --level debug and browser logs

amir20 commented 1 week ago

it will not be accessible unless one of the first 9 is shutdown

Do you mean it doesn't show up in the drop down or the UI doesn't work at all?

Are you by any chance running HTTP1.1 which is limited. See https://stackoverflow.com/questions/16852690/sseeventsource-why-no-more-than-6-connections

I'll probably can look at this in the morning but a little more clarification is needed.

Thanks for stress testing!

beastarman74 commented 1 week ago

I believe it is a interface issue. 3 of the 12 hosts running dozzle agent are not visible in the list of agents shown in the main display or the host selection drop down menu. Other than that Dozzle and the listed agents appear fully functional from a log viewing and container action perspective.

I have 12 hosts running dozzle in agent mode with 3 of which running full dozzle as well. I have 10 hosts in my homelab, 8 physical (Raspberry Pi, NanoPi and repurposed thin client) and 2 lxc containers running on proxmox on a repurposed micro business pc. I created additional test lxc containers to test further.

dockge shows the agents and full dozzle installed: Screenshot_20240708_111824

If I remove one of the listed agents and restart/recreate the full dozzle containers one of the dozzle agents will be shown in its place. I think it might use creation or start time to pick which but this could just be my perception.

Sorry my initial issue was not clear enough. It was my 4th attempt to create it, I lost it 3 times to chrome refreshing after a short period of inactivity so it likely wasnt my best effort.

Thanks for looking into this.

Paul.

amir20 commented 1 week ago

Can you check the logs when dozzle starts? Do you see any warnings?

Right now if agents fail at start up, they will not be retried.

So I am wondering if some of the clients are not connecting.

beastarman74 commented 1 week ago

I think I have led you on a merry dance. There is something in the Full Dozzle log, I cannot believe I missed this. It is the LXC containers. I have to use backup / restore method of creating the containers. So I have a LXC template setup for this purpose. It has had Dozzle running in the past, in both modes (along with other containers, portainer agent, watchtower). When I restore the backup to a new LXC Container I always ensure I tick the unique box. I believed this ensured all IDs where changed.

It looks like Dozzle is remembering a host ID. But I cannot find where this is being pulled from stored.

image

amir20 commented 1 week ago

Ah there we go. This is coming from docker system info. If two instance have the same id then they would be treated as a duplicate.

Is there a better id to use? According to Docker documentation, this is supposed to be a unique ID.

beastarman74 commented 1 week ago

I have managed to sort the issue. It needs the engine-id file that is stored in /var/lib/docker removing from any machine with a duplicate ID. Then restart docker or the host machine which will recreate it with a new value.

All the machines are now visible in Dozzle. image

I am sorry I have wasted your time on this one. Not sure if it is worth renaming this issue to something similar to the below in case someone else in 50 years time manages to create the same issue for themselves! : WARNING [0001] duplicate host ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx. Endpoint: xxx.xxx.xxx.xxx:7007 found, skipping

amir20 commented 1 week ago

Ha nice. This was helpful for me too.