search

burmilla / os

Tiny Linux distro that runs the entire OS as Docker containers
https://burmillaos.org
Apache License 2.0
210 stars 14 forks source link

Update or replace system-docker #28

Closed dgiebert closed 1 year ago

dgiebert commented 3 years ago

Currently the version is currently pinned to 17.06

The buildsystem the following forks with the main change linked:

Do you want to also create a fork or use patch files instead ?

olljanat commented 3 years ago

I have forked those repos under burmilla but I'm not sure if we really want upgrade system-docker because we don't really need those new features on it and binaries sizes has increased a lot of on those new versions.

docker-17.06.2-ce:
total 89M
-rwxr-xr-x 1 pi pi  18M Sep  5  2017 docker
-rwxr-xr-x 1 pi pi 9.5M Sep  5  2017 docker-containerd
-rwxr-xr-x 1 pi pi 8.6M Sep  5  2017 docker-containerd-ctr
-rwxr-xr-x 1 pi pi 3.7M Sep  5  2017 docker-containerd-shim
-rwxr-xr-x 1 pi pi  39M Sep  5  2017 dockerd
-rwxr-xr-x 1 pi pi 755K Sep  5  2017 docker-init
-rwxr-xr-x 1 pi pi 2.4M Sep  5  2017 docker-proxy
-rwxr-xr-x 1 pi pi 7.4M Sep  5  2017 docker-runc

docker-19.03.14:
total 200M
-rwxr-xr-x 1 pi pi  36M Dec  1 21:22 containerd
-rwxr-xr-x 1 pi pi 6.9M Dec  1 21:22 containerd-shim
-rwxr-xr-x 1 pi pi  19M Dec  1 21:22 ctr
-rwxr-xr-x 1 pi pi  59M Dec  1 21:22 docker
-rwxr-xr-x 1 pi pi  69M Dec  1 21:22 dockerd
-rwxr-xr-x 1 pi pi 693K Dec  1 21:22 docker-init
-rwxr-xr-x 1 pi pi 2.8M Dec  1 21:22 docker-proxy
-rwxr-xr-x 1 pi pi 9.2M Dec  1 21:22 runc

docker-20.10.0:
total 219M
-rwxr-xr-x 1 pi pi  38M Dec  8 20:59 containerd
-rwxr-xr-x 1 pi pi 7.0M Dec  8 20:59 containerd-shim
-rwxr-xr-x 1 pi pi 9.5M Dec  8 20:59 containerd-shim-runc-v2
-rwxr-xr-x 1 pi pi  21M Dec  8 20:59 ctr
-rwxr-xr-x 1 pi pi  53M Dec  8 20:59 docker
-rwxr-xr-x 1 pi pi  76M Dec  8 20:59 dockerd
-rwxr-xr-x 1 pi pi 693K Dec  8 20:59 docker-init
-rwxr-xr-x 1 pi pi 2.8M Dec  8 20:59 docker-proxy
-rwxr-xr-x 1 pi pi  13M Dec  8 20:59 runc

Also I can see that system-docker and system-dockerd binaries are smaller than same version from official Docker version so those are most probably patched somehow to only build only needed of the features.

docker-amd64-17.06-ros6:
total 67M
-rwxr-xr-x 1 pi pi 9.9M Feb 11  2019 system-docker
-rwxr-xr-x 1 pi pi 9.5M Feb 11  2019 system-docker-containerd
-rwxr-xr-x 1 pi pi 8.6M Feb 11  2019 system-docker-containerd-ctr
-rwxr-xr-x 1 pi pi 3.7M Feb 11  2019 system-docker-containerd-shim
-rwxr-xr-x 1 pi pi  25M Feb 11  2019 system-dockerd
-rwxr-xr-x 1 pi pi 755K Feb 11  2019 system-docker-init
-rwxr-xr-x 1 pi pi 2.4M Feb 11  2019 system-docker-proxy
-rwxr-xr-x 1 pi pi 7.4M Feb 11  2019 system-docker-runc

So most probably we should update ros command to all containerd directly so we can drop system-docker but I'm not sure if that is worth effort.

Alternative way is stay on 17.06 forever and only cherry-pick fixes from Moby if needed.

dgiebert commented 3 years ago

The reason for this is probably the removal of some features for system-docker:

I will try to compile and adapt this and post a size for system-docker

dgiebert commented 3 years ago

Actually after a successful built, the size did not shrink that much:

-rwxr-xr-x 1 dgiebert dgiebert  36M Dec 13 14:18 system-containerd*
-rwxr-xr-x 1 dgiebert dgiebert 6.9M Dec 13 14:18 system-containerd-shim*
-rwxr-xr-x 1 dgiebert dgiebert  19M Dec 13 14:18 system-ctr*
-rwxr-xr-x 1 dgiebert dgiebert  59M Dec 13 14:18 system-docker*
-rwxr-xr-x 1 dgiebert dgiebert  55M Dec 13 14:18 system-dockerd*
-rwxr-xr-x 1 dgiebert dgiebert 693K Dec 13 14:18 system-docker-init*
-rwxr-xr-x 1 dgiebert dgiebert 2.8M Dec 13 14:18 system-docker-proxy*
-rwxr-xr-x 1 dgiebert dgiebert 9.2M Dec 13 14:18 system-runc*

Checking the source code and diffs, a bit of google research showed the possibility of running multiple deamons: https://stackoverflow.com/questions/32334167/is-it-possible-to-start-multiple-docker-daemons-on-the-same-machine

This could be a better way to only maintain a single dockerd and just have a small wrapper script for system-docker

olljanat commented 3 years ago

Checking the source code and diffs, a bit of google research showed the possibility of running multiple deamons: https://stackoverflow.com/questions/32334167/is-it-possible-to-start-multiple-docker-daemons-on-the-same-machine

This could be a better way to only maintain a single dockerd and just have a small wrapper script for system-docker

@dgiebert first of all I would like to understand that what you are actually trying to achieve by updating system-docker? Main purpose of it is actually just works as package provider (like apt-get, yum, etc) for BurmillaOS so as long we don't have known bugs on it I don't see reason to change it.

olljanat commented 3 years ago

Go binaries growing looks to be generic problem https://github.com/golang/go/issues/6853

I can see same on https://github.com/burmilla/docker/tree/test/update-build-scripts where I just took latest build scripts from Moby project so those gets build with later version of Go. That issue can be partly fixed by optimizing build parameters but not fully.

dgiebert commented 3 years ago

Also quite visible here in the official stable builds: https://download.docker.com/linux/static/stable/x86_64/ Will try to optimize the build with the compiler flags later today, any tipps ?

Also from the linked issue, seems that in 1.15 the size could be reduced but as of now moby is on 1.13.

olljanat commented 3 years ago

They are working on to get it updated https://github.com/moby/moby/pull/40353

Also if we start working on this then we probably should target directly to 20.10.x (or even 21.03.x) because major refactoring have been done to Moby on that version (backend code have been removed from Moby used from containerd) and afaiu that work still continues.

dgiebert commented 3 years ago

Very good point indeed, changed to 20.10.1 !

Old Size (19.03.14): ~187M New Size(20.10.1) using 1.15.5: ~177M

-rwxr-xr-x 1 root root  33M Dec 15 12:30 system-containerd*
-rwxr-xr-x 1 root root 6.5M Dec 15 12:30 system-containerd-shim*
-rwxr-xr-x 1 root root  21M Dec 15 12:30 system-ctr*
-rwxr-xr-x 1 root root  49M Dec 15 12:30 system-docker*
-rwxr-xr-x 1 root root  52M Dec 15 12:30 system-dockerd*
-rwxr-xr-x 1 root root 693K Dec 15 12:30 system-docker-init*
-rwxr-xr-x 1 root root 2.8M Dec 15 12:30 system-docker-proxy*
-rwxr-xr-x 1 root root  12M Dec 15 12:30 system-runc*

This was just plain 1.13.15 -> 1.15.5 changes, so I believe more can be achieved here

dgiebert commented 3 years ago

Following this article here: https://boyter.org/posts/trimming-golang-binary-fat/

-rwxr-xr-x 1 root root  11M Dec 15 14:08 system-containerd*
-rwxr-xr-x 1 root root 2.1M Dec 15 14:08 system-containerd-shim*
-rwxr-xr-x 1 root root  21M Dec 15 14:08 system-ctr*
-rwxr-xr-x 1 root root  25M Dec 15 14:08 system-docker*
-rwxr-xr-x 1 root root  15M Dec 15 14:08 system-dockerd*
-rwxr-xr-x 1 root root 277K Dec 15 14:08 system-docker-init*
-rwxr-xr-x 1 root root 1.6M Dec 15 14:08 system-docker-proxy*
-rwxr-xr-x 1 root root 5.0M Dec 15 14:08 system-runc*

Size after UPX compression: ~81M

olljanat commented 3 years ago

Interesting that Docker CLI is bigger than Docker daemon it selves but of course that makes sense as part of logic have been moved to containerd nowadays.

This made me thinking that it might possible actually replace "system-docker" with shell script which call "docker" command with needed parameters because boot process does not need it afaik:

$ sudo docker -H unix:///var/run/system-docker.sock ps
CONTAINER ID        IMAGE                               COMMAND                  CREATED             STATUS              PORTS               NAMES
19b253c34be2        burmilla/os-docker:19.03.14         "ros user-docker"        36 hours ago        Up 36 hours                             docker
13b35a796613        burmilla/os-console:v1.9.0          "/usr/bin/ros entryp…"   36 hours ago        Up 36 hours                             console
f3e81544617a        burmilla/os-base:v1.9.0             "/usr/bin/ros entryp…"   36 hours ago        Up 36 hours                             ntp
0229f9a12949        burmilla/os-base:v1.9.0             "/usr/bin/ros entryp…"   36 hours ago        Up 36 hours                             network
0f86354032d3        burmilla/os-base:v1.9.0             "/usr/bin/ros entryp…"   36 hours ago        Up 36 hours                             udev
37ac8ebfb9df        burmilla/container-crontab:v0.5.0   "container-crontab"      36 hours ago        Up 36 hours                             system-cron
742bdbd5c705        burmilla/os-syslog:v1.9.0           "/usr/bin/entrypoint…"   36 hours ago        Up 36 hours                             syslog
2c50b2258fda        burmilla/os-acpid:v1.9.0            "/usr/bin/ros entryp…"   36 hours ago        Up 36 hours                             acpid

Also I think that we do not need "system-docker-containerd-ctr" / "system-ctr" at all and "system-docker-proxy" gets removed during build process on: https://github.com/burmilla/os/blob/93de612cdcf6b64e51fa30826434dd70215f0dac/scripts/layout-initrd#L33-L34

What comes to idea to use UPX, system-docker is actually packages inside of initrd which is already compressed so we should check how this affect size of it and iso file: https://github.com/burmilla/os/blob/93de612cdcf6b64e51fa30826434dd70215f0dac/scripts/package-initrd#L19-L27

Also boot time and memory usage are areas which we need follow when we changing these as there is already comment about high CPU usage with too much compression: https://github.com/burmilla/os/blob/93de612cdcf6b64e51fa30826434dd70215f0dac/scripts/tar-images#L47-L54

dgiebert commented 3 years ago

Yes, I think it is very easy to replace system-docker with a two line bash script in the console image.

You are correct did not think about the initrd compression, but I guess if we strip off all unneeded / duplicated executables the resulting size should not exceed the 17.06 build by much

olljanat commented 3 years ago

This could be a better way to only maintain a single dockerd and just have a small wrapper script for system-docker

Hmm. Maybe that is actually best way because if we decide that system-docker and default user docker versions are same then we can:

Only special things needed on that one would be that we need modify settings on here: https://github.com/burmilla/os/blob/4e5d2482980b614480d99ff3c4de33c4562502c5/os-config.tpl.yml#L361-L385

Then it should (on theory) works similarly like consoles was earlier that default settings will be overridden when user uses sudo ros engine switch command (as on that point it need to use version specific configuration from os-service repo) and probably we need also add "default" option to that list which always points to Docker version which is bundled to ISO file.

olljanat commented 3 years ago

@dgiebert so how you now build those 20.10.1 binaries? I'm prepairing next 2.0.0 beta version so probably we should include it already and implement those optimizations later.

dgiebert commented 3 years ago

The main issue I see with the shared dockerd, would be that an upgrade of the engine would need a reboot.

I will try to get a build working today @olljanat

dgiebert commented 3 years ago

@olljanat seems like the console is starting and tells started but its not attaching. Do you have a pointer where this could be hanging ?

olljanat commented 3 years ago

@dgiebert can you share those codes which you are now using so it is easier to guess what might be wrong?

olljanat commented 3 years ago

Instead of updating system-docker we should also consider:

olljanat commented 1 year ago

I finally managed to build working version of system-docker from sources (earlier we have used binary build by Rancher) https://github.com/burmilla/os-system-docker/releases/tag/17.06.107

What I did find out that it is possible to build system-docker even with latest Go version (draft in here) but there is some change done in Go it selves which breaks it in way that runc is not able to start console container anymore.

replacing it example with containerd and nerdctl combination

I think that this would be best long term solution. We can keep size in control by disabling all those optional components https://github.com/containerd/containerd/blob/v1.6.15/BUILDING.md#build-containerd

olljanat commented 1 year ago

It looks to be that replacing system-docker with containerd + nerdctl would need massive refactoring for code from where ros binary is build because we use https://github.com/docker/libcompose/ and nerdctl uses https://github.com/compose-spec/compose-go

However https://github.com/balena-os/balena-engine potential drop-in replacement for system-docker so will test it some day.

olljanat commented 1 year ago

Update. Replacing system-docker with balena-engine is simple but looks that new runc version makes system crashing on boot. No matter if that is used with balena or system-docker.

Test versions of those exist in https://github.com/burmilla/os/tree/migrate-to-balena and https://github.com/burmilla/os-system-docker/releases/tag/17.06.109

pwFoo commented 1 year ago

Hi @olljanat interesting topic... I heared about balena, but don't know about balena cloud / open balena... Would it possible to manage instances with balena cloud / open balena if based on balena-engine?

Are the problems related to runc or maybe test it with crun?

olljanat commented 1 year ago

Would it possible to manage instances with balena cloud / open balena if based on balena-engine?

No. balena-engine is just minimized version of docker/moby engine (swarm mode is removed, etc) so very similar than how we use system-docker, just based on more modern version of docker and maintained by Balena team (the main reason to which to it).

Are the problems related to runc or maybe test it with crun?

That is new for me. Will look about it.