🚧 Deprecation of remote host in favor of agents

[amir20]

Hello!

I am in the process of releasing a very big feature for Dozzle which will introduce agents. Agents can run remotely on a set of hosts or in swarm mode. Agents use a secured connection that only Dozzle UI and agents can connect.

Agents are much safer than exposing Docker socket. They are also more efficient since each agent does the majority of the work.

I have currently updated documentation at https://dozzle.dev/guide/remote-hosts to be deprecated.

If you have any reasons to keep remote hosts in Dozzle then use this issue to provide feedback. I'd like to hear of use cases that wouldn't be possible with agent.

Thanks!

[amir20]

Someone pointed out on reddit that they might want to control permissions to docker.sock and that now there is no way. Hmmm....not sure if I have a solution with that.

[agneevX]

Is this why there's degraded functionality here where some metrics are unavailable?

[agneevX]

The way I see it is that I already have a wireguard tunnel for secure communications between docker-socket-proxy and Dozzle. I also use this for Portainer docker host control.

So now, I'll need to deploy Dozzle to all machines??

[amir20]

Is this why there's degraded functionality here where some metrics are unavailable?

No, changes to functionality that I know of...

The way I see it is that I already have a wireguard tunnel for secure communications between docker-socket-proxy and Dozzle. I also use this for Portainer docker host control

Interesting, I didn't know Portainer supports this. Perhaps, I can keep the socket proxy support but remove TLS. Not sure yet.

Let's see what others think @agneevX. If very few people need remote host then it makes it hard for me to support it because it takes me significant amount of time to support all different options. I rather trim to what is most used and safest. For now, no change.....

[jathek]

If you have any reasons to keep remote hosts in Dozzle then use this issue to provide feedback. I'd like to hear of use cases that wouldn't be possible with agent.

When possible, I like to avoid giving the entire Docker socket directly to a container, but instead give it access to a socket-proxy with limited permissions. Using DOZZLE_REMOTE_HOST I can do that. If that goes away, then I need to give Dozzle the entire socket, which I would prefer not to do.

If I understand the new agent feature correctly, to somewhat imitate what I'm doing now (Dozzle + a sibling socket-proxy container to provide limitations), I would have to spin up Dozzle and an agent Dozzle next to it. But I would have to give the agent Dozzle full socket access, which defeats the purpose of what I was trying to limit.

[C8opmBM]

I have been using tecnativa/docker-socket-proxy giving only CONTAINERS and IMAGES permission but made the switch to agent today, and all is good. Thanks for your work, your dozzle has been evolving constantly, happy to see this. 🥇

[agneevX]

No, changes to functionality that I know of...

In that screenshot, I was referring to the unavailability of CPU and RAM info.

[bluepuma77]

Does that mean that a docker-socket-proxy is not supported anymore?

Should it currently still work? I am getting an error:

dozzle          | head
dozzle          | Usage: dozzle [--addr ADDR] [--base BASE] [--hostname HOSTNAME] [--level LEVEL] [--auth-provider AUTH-PROVIDER] [--auth-header-user AUTH-HEADER-USER] [--auth-header-email AUTH-HEADER-EMAIL] [--auth-header-name AUTH-HEADER-NAME] [--enable-actions] [--filter FILTER] [--remote-host REMOTE-HOST] [--remote-agent REMOTE-AGENT] [--no-analytics] [--mode MODE] <command> [<args>]
dozzle          | error: unknown argument --remote-host dockersocket:2375

for my little compose experiment:

services:
  dozzle:
    container_name: dozzle
    image: amir20/dozzle:latest
    networks:
      - dockersocket
      - proxy
    restart: unless-stopped
    security_opt:
      - no-new-privileges: true
    user: 10020:0
    command:
      - --remote-host dockersocket: 2375
    labels:
      - traefik.enable=true
      - traefik.http.routers.dozzle.rule=Host(`dozzle.example.com`)
      - traefik.http.routers.dozzle.service=api@internal
      - traefik.http.routers.dozzle.middlewares=dozzleauth
      - traefik.http.middlewares.dozzleauth.basicauth.users=test:$$apr1$$H6uskkkW$$IgXLP6ewTrSuBkTrqE8wj/

  dockersocket:
    image: nginx:alpine-slim
    hostname: dockersocket
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      - dockersocket
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    configs:
      - source: nginx
        target: /etc/nginx/nginx.conf
    expose:
      - 2375

networks:
  proxy:
    name: proxy
    external: true
  dockersocket:
    name: dockersocket

configs:
  nginx:
    content: |
      user root;
      events { worker_connections 1024; }
      http {
        server {
          listen 2375;
          location ~ ^/v1\.24/(events|containers|services|version|networks|tasks) {
            if ($$request_method != GET) { return 405; }
            proxy_pass http://unix:/var/run/docker.sock;
            proxy_set_header Host $$host;
            proxy_set_header X-Real-IP $$remote_addr;
            proxy_set_header X-Forwarded-For $$proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $$scheme;
            proxy_read_timeout 3600s;
          }
          location / { return 405; }
        }
      }

It would be great if a TCP connection to a docker socket is still possible.

[epd5]

Just wanted to chime in that I'd also be losing the ability to use socket-proxy with Dozzle, seems a shame to lose that feature in favor of another I wouldn't use.

[tenfourty]

I'm also using a socket proxy (this one from LinuxServer) and restricting permissions to the proxy's clients.

I like the Agent mode; it seems cool, but I would prefer a model where it coexists with the remote host option.

Also, I hope that the Agent mode eventually allows the same granular access permission as some of these proxies. The current implementation makes me feel pretty insecure, and I wouldn't want to use it.

[amir20]

@EoinD123

Just wanted to chime in that I'd also be losing the ability to use socket-proxy with Dozzle, seems a shame to lose that feature in favor of another I wouldn't use.

Can you share about your use case as why you wouldn't use it? Socket-proxy is not very secure without TLS so I am trying to understand if the only use case is limiting permissions.

@bluepuma77 I haven't removed any functionality. So it should work. I have never used your setup though. Documentation with socket proxy should still work at https://dozzle.dev/guide/remote-hosts. You are using your own setup so I am not sure.

[amir20]

@tenfourty I think the majority of people want to limit permissions which is why they use socket proxy? Is that so? I think a better solution would be to map docker.socket to another such as docker-readonly.socket by setting up a proxy. My gripe with socket-proxy is that is actually opens up docker to be able to read anything, even within the network could be a security threat.

Based on feedback, I think one thing is for sure. Removing the TLS support for remote hosts seems to be OK. I am just trying to remove features that rarely get used.

I hope that the Agent mode eventually allows the same granular access permission as some of these proxies

What would you imagine that looking like? Agents are already read only because the GRPC only supports reading logs, events, and stats. So I would be interested to know how you would feel comfortable knowing that agents only return data.

I would be happy if we could collaborate on a model with agent that makes people comfortable with agents. The remote hosts are not safe:

• socket proxy doesn't use TLS. Agents have a secure mTLS communication
• socket proxy needs to be setup correctly and I get too many pings about that. Agents just work out of the box.
• Agents distribute the load on the host whereas remote host set up brings the computation on the UI.

Personally, I wish there was a solution that remaps socket file to another file that limits non-get requests.

[dj3520-git]

Darn lol, I was using the TLS implementation for a remote host and setup some PKI for it. More than happy to give the agents a whirl though, I guess my only gripe would be having another container running on the remote host to support it, but that is a very small gripe. There are also some plus sides to this approach as you mention above.

I was starting to look into proxying the docker socket and providing access more granularly for the container on the 'master' host, same as most above. I saw #3088 today and didn't realise that the :ro didn't apply to sockets which prompted some investigation into proxying the docker socket as that seemed like the best approach.

[amir20]

@dj3520-git you must be one of three rare people? 😂 Setting up TLS with Docker is so hard that most people just don't do it. I have had so many issues opened asking how to do it that I just gave up and said use socket proxy.

More than happy to give the agents a whirl though, I guess my only gripe would be having another container running on the remote host to support it

Please do give agents a try. The agent is actually pretty light. A big advantage is that each agent does it's own processing making it lighter on the UI. There is also support for swarm mode.

Just so that everybody understand why I am pushing back on this, the majority of issues that come through are something like "hey, I setup TLS or socket proxy and it doesn't work". After a bunch of investigation, I find that they misconfigured it or some thing beyond Dozzle. Even after all of this, they have exposed Docker API to their whole network. I can't imagine anybody would prefer this knowing the risks. So my goal with agents is to have some thing super easy to setup without needing to support external dependencies that I have no control over. I just don't have a lot time to support every single use case. Having agents, makes it easier to see here is a docker-compose.yml file and it should just work without anything extra work. https://github.com/amir20/dozzle/issues/3066#issuecomment-2220532261 is an example that is beyond me investigating.

This project is taking considerable amount of time from me 💁🏼. But it sounds like people just want control permissions to Dozzle. I wonder if there is any other ways to do this. I could use some help from the community. I don't think setting up an HTTP proxy just for the sake of controlling permission is the right solution.

I saw https://github.com/amir20/dozzle/issues/3088 today and didn't realise that the :ro didn't apply to sockets which prompted some investigation into proxying the docker socket as that seemed like the best approach.

That's right. I don't why anybody ever thought :ro is means the socket is read only.

As said earlier, the right approach would be to have docker.socket proxied through another socket file. I am sure something like this exists.

[jathek]

For me, while Dozzle agents are useful, and I will implement one shortly, they don't give me the same security to the main Dozzle instance that using a socket-proxy does. Limiting access to the docker socket by Dozzle is important to me b/c I access Dozzle via reverse proxy. I have Authelia as a single sign-on authentication protecting it, but I like having the extra just-in-case security roadblock.

[epd5]

@EoinD123

Just wanted to chime in that I'd also be losing the ability to use socket-proxy with Dozzle, seems a shame to lose that feature in favor of another I wouldn't use.

Can you share about your use case as why you wouldn't use it? Socket-proxy is not very secure without TLS so I am trying to understand if the only use case is limiting permissions.

Sure! Very interesting reading the thread, a lot I dont know about the workings of docker. I'm running a hobbyist stack of ~50 containers, about 6 of them require socket access, so I run socket-proxy & the rest of them on their own internal only network. Having said that I kinda assumed anyway if some of them share another icc enabled network maybe it defeats the purpose.

[amir20]

Hey folks! Let me try to summarize.

Based on the feedback, people want to have a way to control the permissions for Dozzle. All fair. I did some research and Docker doesn't seem to have a solution. Seems like socket-proxy is the best solution. It's not great solution and I can't find anything better. Side project anyone? 🤔

Therefore, until there is a better choice to control docker.sockets's permission. I don't think it would make sense to remove this functionality to use socket-proxy.

Now speaking of the TLS support which requires someone to setup TLS in Docker and expose over HTTP I think makes less sense to support. It is much more work to setup and agents could just replace that with a safer choice. But as long I don't get issues to debug why it didn't work then no worries.

All that said, I am not planning to make any changes. So don't freak out. I'll keep this issue open for a while. I am sure at some point there will be features that will be easier implemented (like swarm mode) in agents. Overtime, I hope the use of agents would grow. For example, performance and memory is already faster in agents because everything use protobuf which requires less parsing. Where using socket proxy has everything passed around as JSON increasing CPU considerably on the UI.

I do plan to add retry when agents fail which is a common request. I can do this with agents now but I won't plan to do it with remote-hosts.

[bluepuma77]

DOZZLE_REMOTE_HOST=tcp://dockersocket:2375 works for me, shows a deprecation message.

But --remote-host tcp://dockersocket:2375 does not work:

error: unknown argument --remote-host tcp://dockersocket:2375

Doozle works with this (very simplified) inline docker-socket-proxy solution:

services:
  dockersocket:
    image: nginx:alpine-slim
    restart: unless-stopped
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    configs:
      - source: nginx
        target: /etc/nginx/nginx.conf

configs:
  nginx:
    content: |
      user root;
      events { worker_connections 1024; }
      http {
        server {
          listen 2375;
          location ~ ^/v1\.(24|46)/(info|events|containers|services|version|networks|tasks) {
          if ($$request_method !~ ^(GET|HEAD)$$) { return 405; }
            proxy_pass http://unix:/var/run/docker.sock;
            proxy_set_header Host $$host;
            proxy_set_header X-Real-IP $$remote_addr;
            proxy_set_header X-Forwarded-For $$proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $$scheme;
            proxy_read_timeout 3600s;
          }
          location /_ping {
          if ($$request_method !~ ^(GET|HEAD)$$) { return 405; }
            proxy_pass http://unix:/var/run/docker.sock;
          }
          location / { return 405; }
        }
      }

The advantage of the docker-socket-proxy is that I do not need a high trust level with the application using it. Sadly Docker does not enable restricting the socket to GET only.

With the docker-socket-proxy I can ensure that containers with access will not be wreaking havoc. I don't need high trust in the code, that a contributor might be smuggling in bad code, that someone highjacks the build pipeline or someone finds a bug to hack into the container. Effects should be minimised that way, security increased. And by not trusting another random proxy solution (but a bigger corporate brand), I only increase the attack surface minimally.

So thanks a lot for providing dozzle and for keeping the remote host 👍

[tenfourty]

Based on the feedback, people want to have a way to control the permissions for Dozzle. All fair. I did some research and Docker doesn't seem to have a solution. Seems like socket-proxy is the best solution. It's not great solution and I can't find anything better. Side project anyone? 🤔

Therefore, until there is a better choice to control docker.sockets's permission. I don't think it would make sense to remove this functionality to use socket-proxy.

Thanks, @amir20. This sounds like a pragmatic way forward. I use my Docker socket proxy for other containers to securely access my Docker socket, not just Dozzle, so I found it easier to set up one container for all my services that need it.

I don't expose my proxy directly to my LAN, it's only available to a few containers via a docker network on the same host (of course still a risk) but I think it's low given it exposes a read-only API.

[dj3520-git]

@amir20 Thank you for keeping the TLS side of things for now and for all your hard work with Dozzle. I completely understand if the support burden gets too large; as you say, agents are a simpler way of achieving the same thing with a copy and paste compose file (and I will be having a good look at this as soon as I can). Thank you also for the kind words regarding Docker and TLS! I guess the hardest part is the PKI, but once that is sorted so long as you read the documentation here and for Docker (both are really good) it all just slotted together nicely for me.

@bluepuma77 That looks like a very eloquent solution for proxying the docker socket for other containers to use https://github.com/amir20/dozzle/issues/3066#issuecomment-2222375239, thank you for sharing! I agree it's a notch above spinning up someone else's container and I like your choice of nginx too, nice work. 👍

[amir20]

@bluepuma77 created #3095. I am pretty sure you have a typo some where as it works for me.

[amir20]

@tenfourty

Thanks, @amir20. This sounds like a pragmatic way forward. I use my Docker socket proxy for other containers to securely access my Docker socket, not just Dozzle, so I found it easier to set up one container for all my services that need it.

That's a fair point. I didn't realize how many other applications require docker.socket.

[dhop90]

I'm not able to get agents to work in my kubernetes (k8s) environment. I'm using docker with containerd=/run/containerd/containerd.sock with tls certs, hence docker.socket is not available. In my scenario all docker communication is done over a tls connection to the containerd.sock. How do I get the agent to talk to containerd.sock using tls certs?

[MRobi1]

I love the addition of the agent, makes it very simple to connect my various LXC containers running docker to 1 central dozzle instance. There seems to be a limit of 7 agent connections. I can add more, but only 7 show on the UI. Is this something that will change in future releases? It'd also be great to have some sort of sort order to the agent connections.

[amir20]

@MRobi1 there is no limits to the number of agents that can be configured. You maybe running to this issue. https://github.com/amir20/dozzle/issues/3072. Make sure to look at the logs. I added a section in FAQ about this. https://dozzle.dev/guide/faq#i-am-seeing-duplicate-hosts-error-in-the-logs-how-do-i-fix-it

You can create a new issue if it's something else. ❤️

[MRobi1]

Yup, duplicate hosts was my issue since I typically deploy my LXC containers using a template that I built.

That does bring up a few additional things:

• I'd recommend linking to that section through the agent docs so it's more obvious
• I'd also recommend adding "restart: unless-stopped" to the docker compose and equivalent for the docker run commands when deploying the agent so they'll start again on reboot
• If Dozzle is started before the agent, it doesn't appear to connect without restarting dozzle. This means dozzle will need to be the absolute last thing started or require a manual reboot every time. Is there a way to implement something when the agent starts it will initiate a reconnect to dozzle? Or in dozzle have it attempt reconnects every X minutes on agents that did not connect on initial startup?

[amir20]

I'd recommend linking to that section through the agent docs so it's more obvious I'd also recommend adding "restart: unless-stopped" to the docker compose and equivalent for the docker run commands when deploying the agent so they'll start again on reboot

If you have time send PRs! I am out of the office for a while.

If Dozzle is started before the agent, it doesn't appear to connect without restarting dozzle. This means dozzle will need to be the absolute last thing started or require a manual reboot every time. Is there a way to implement something when the agent starts it will initiate a reconnect to dozzle? Or in dozzle have it attempt reconnects every X minutes on agents that did not connect on initial startup?

That's on the roadmap and tracked at https://github.com/amir20/dozzle/issues/3094

[amir20]

Added 64e8ec48 for a quick warning on duplicate hosts.

[amir20]

@dhop90 Looks like agents work in k8s and I have updated the other bug.

[dj3520-git]

Just wanted to say that after testing the agent I've migrated to it from remote host TLS on the production setup and everything's working great, thanks for the awesome feature! I will still likely experiment with the inline nginx solution at some stage (posted in https://github.com/amir20/dozzle/issues/3066#issuecomment-2222375239), so it would be good to keep the remote host feature. Though, as I have no other containers that require access right now, I will be using the agent for the foreseeable future.

[amir20]

@dj3520-git that's great. In version 8.1 I will be making agent faster and more resilient to failures. If agent is not up when Dozzle starts, won't matter anymore. I think in most cases agent is better option except when one wants to limit permission. I don't have a solution around that.

[amir20]

To everybody else, I'll probably update the warning message that remote host might not be going away in the short term but I won't be investing much time in it.

[amir20]

And v8.1.0 is released with improved agents.

[webysther]

@amir20 What you think to add a SECURITY_KEY or ALLOWED_HOST to agent to make the communication safe for edge servers?

[amir20]

Hey @webysther, communication over edge should already be safe. What makes you think it is not? Dozzle uses private mutual TLS between agents. All other connections are rejected. Would there be a need to change the certificate?

If there is a use case, then I think SECURITY_KEY would make sense to override the certs. If so, create a "feature request" with valid reason and we can discuss there. Thanks!

[damho1104]

@amir20 I am using the remote host feature to utilize Dozzle for viewing Docker container logs on more than 10 devices. If I switch to using the agent feature, I would like the UI to provide an update function for the agents.

[amir20]

@damho1104 While I agree that would be a good idea, principally it wouldn't make sense for containers to update themselves. Dozzle mostly reads and not modify any states. I am not aware of any other docker applications doing a self update. Also, agents can be ran in swarm, services, compose or just docker. Dozzle wouldn't know how to update it.

Better to use watchtower. Dozzle will reconnect if agents go down.

If there other examples of applications doing this then let me know. I can take a look at how they were implemented.

[jeromehmu]

First of all, Dozzle is pretty amazing so thank you. This agent feature has been pretty great. Question though, when you make an instance an agent it no longer has the standalone capability. Is that intentional? Is it possible to have both from a single container instance?

My rationale is that I run Dozzle on two different machines, let's say A and B, and I have the agent version running on B from which A connects to. However, there are times the connection I have from A <-> B is severed but I can still reach B through another network interface/VPN. From that network interface, it would be nice to still pull up the regular Dozzle interface from that B container running as an agent. For a workaround, I can spin up an agent and a regular Dozzle instance on B but was wondering if that may be a future feature.

[amir20]

@jeromehmu Anything is possible. But I don't think it's worth the complexity. Docker best practices is to run one application per container. This would be more complex and error prone. And probably not worth it for me to make the changes.

[amir20]

In the latest version, I have changed the warning to log to info. I have also added a comparison chart in https://dozzle.dev/guide/agent to compare the two solutions.

While no plans to remove remote hosts, I think the advantages are clear. I am going to close this issue and lock it. Any new requests should be through a new issue.