Allow override of docker engine Id from an environment variable

[lionep]

Describe the feature you would like to see

Possibility to override the docker engine id for agent host Id, to avoid duplicate host ID.

Describe how you would like to see this feature implemented

Get an environment variable like DOZZLE_DOCKER_KID to use instead of docker engine id, when provided, to avoid any conflict without changing docker engine id on host.

Describe any alternatives you've considered

The Tip from documentation suggests to remove engine-id file which does not exist on my host: (debian 11 + docker 20.10.5+dfsg1).
All hosts are installed in different VMs on a proxmox hosts, and default kid /etc/docker/key.json is the same on every instance (I guess it's computed from some hardware signature).

[amir20]

Hmm, it is not a good idea to allow for overrides. The id is injected in URLs, which introduces security issue amongst other issues. My philosophy for Dozzle is to create the best logic that works for all. Sounds like Docker's system ID might not be a good option.

I installed debian and it is there:

root@deb-test:/var/lib/docker# ls -lah
total 4.0K
drwx--x--- 1 root root 154 Jul 22 08:11 .
drwxr-xr-x 1 root root 148 Jul 22 08:11 ..
drwx--x--x 1 root root 152 Jul 22 08:11 buildkit
drwx--x--- 1 root root   0 Jul 22 08:11 containers
-rw------- 1 root root  36 Jul 22 08:11 engine-id
drwx------ 1 root root  16 Jul 22 08:11 image
drwxr-x--- 1 root root  10 Jul 22 08:11 network
drwx--x--- 1 root root   2 Jul 22 08:11 overlay2
drwx------ 1 root root  20 Jul 22 08:11 plugins
drwx------ 1 root root   0 Jul 22 08:11 runtimes
drwx------ 1 root root   0 Jul 22 08:11 swarm
drwx------ 1 root root   0 Jul 22 08:11 tmp
drwx-----x 1 root root  56 Jul 22 08:11 volumes
root@deb-test:/var/lib/docker#

Instead of introducing DOZZLE_DOCKER_KID, maybe we should use a different ID. It just needs to be consistent between restarts. I see a lot of people asking this question so it's probably worth fixing it. But there hasn't been any better solutions.

Other options could be:

• Mac Address
• Host IP
• Something else?

I'll need your help a little since I haven't been able to reproduce this.

All hosts are installed in different VMs on a proxmox hosts

I would imagine there is still a system id that can be changed. This is supposed to be a unique id from my understanding. But as shown above, there is a engine-id file.

I guess it's computed from some hardware signature

If this is true, then we have a huge problem. Because I assume it is randomly generated.

I'll need your help to investigate since in all my VMs it is unique.

[amir20]

Maybe a better option is to just create random ID and mount it with VOLUME. I am not sure if this is a feasible option though and I guess it would only work for agents.

Edit: I don't think this would work as it create a new ID with new containers. It would have to be mounted properly instead.

[lionep]

Thanks @amir20 for all suggestions and thinking of a way to fix those issues.

On my side, I installed docker with the method :

curl -fsSL https://get.docker.com -o install-docker.sh

And my /var/lib/docker directory looks like this :

docker ls -lah /var/lib/docker
total 60K
drwx--x--- 13 root root 4.0K Jul 22 13:04 .
drwxr-xr-x 26 root root 4.0K Mar  4 10:36 ..
drwx--x--x  4 root root 4.0K Mar  4 10:36 buildkit
drwx--x---  4 root root 4.0K Jun 21 10:24 containers
drwx------  3 root root 4.0K Mar  4 10:36 image
drwxr-x---  3 root root 4.0K Mar  4 10:36 network
drwx--x--- 73 root root  12K Jul 22 13:04 overlay2
drwx------  4 root root 4.0K Mar  4 10:36 plugins
drwx------  2 root root 4.0K Jul 22 13:04 runtimes
drwx------  2 root root 4.0K Mar  4 10:36 swarm
drwx------  2 root root 4.0K Jul 22 13:04 tmp
drwx------  2 root root 4.0K Mar  4 10:36 trust
drwx-----x  2 root root 4.0K Jul 22 13:04 volumes

The volume method is something that is found in some projects that do not rely on any db for data storage. If the user mount a local directory as a volume, ID would persist between multiple down / up sequences.

What would be the impact if an agent has its ID changing ?

Can't the endpoints defined in DOZZLE_REMOTE_AGENT also be used as ids ?

If this is true, then we have a huge problem. Because I assume it is randomly generated.

I setup docker with some ansible playbooks on multiple VMs, and with VM base image. I've double checked right now if two VMs with same specs on same host (proxmox) get the same ID : they don't. So the ID is indeed kind of randomized, and on my systems, it seems it has been randomized on the base image.

I also check which file is defining the docker Id on my system, and the only one is : /etc/docker/key.json, and if I manually edit the ID here and restart, docker crash when starting.

If I remove the "kid" key from key.json file, ID remains the same on docker restart, and kid is not repopulated in file.

Let me know if I can help in any way.

[lionep]

Removing completly the key.json file, would generate a new docker id, and it seems containers still doing well, after docker service restarts.

[amir20]

Removing completly the key.json file, would generate a new docker id, and it seems containers still doing well, after docker service restarts.

Nice! So it sounds like the issue was fixed?

I did find this issue https://github.com/moby/moby/issues/13278 which suggests that if engine id is not unique, it will not work properly with other Docker services. Given that Docker assumes that the system ID needs to be unique, I think this is the best option. Alternate options would be to do hacky MAC addresses or auto generate an ID.

docker 20.10.5+dfsg1

I hadn't see this before. But that explains why it is /etc/docker/key.json. You have a very old version. The latest version is 26.1.3 which has engine-id.

What would be the impact if an agent has its ID changing ?

The impact would be that when the agent restarts, the UI wouldn't be able to find it. The UI has a map of ID --> clients. If the ID changes on the agent, it would route it to the agent but then the agent wouldn't know what to do with it since it doesn't have a matching ID.

I am going to close this. Sounds like if you update your docker, you would find an engine-id. I also don't think there is a better option. Docker does assume these IDs are unique and I think that's a fair assumption for Dozzle too.

[lionep]

Thanks, nice find about the docker id that is meant to be unique.