search

distr1 / distri

a Linux distribution to research fast package management
https://distr1.org
Other
536 stars 26 forks source link

docker: reduce container image size #28

Open stapelberg opened 5 years ago

stapelberg commented 5 years ago

Current size is 1.07 GB uncompressed, 324 MB compressed. docker or the docker hub seems to transparently compress, so the progress output shows me that I’m uploading 1.07 GB, but when downloading the container image from the docker hub, I only need to download 324 MB.

I’m filing this so that we can track ideas, but I don’t plan to work on this actively right now. It’s unfortunate that the container is so large.

jojo243 commented 4 years ago

Maybe if you provide the Dockerfile you are using, I can look into this (I have some experience converting Applications into docker images of minimal size)

stapelberg commented 4 years ago

I’m not using a Dockerfile. I’m creating a root file system via https://github.com/distr1/distri/blob/1d68a1393d7138631ea1c3f339f2388256408c0f/cmd/distri/pack.go#L174, which is directly packed into a single-layer container.

The usual techniques of optimizing Dockerfiles won’t help here.

pwaller commented 4 years ago

You don't mention in your issue how big it is now, which might be an interesting thing to have track of :)

(Nice effort btw!)

stapelberg commented 4 years ago

Update the first comment on this issue to include the size. I suppose transferring 324 MB isn’t so bad actually.

pwaller commented 4 years ago

Yup, put that way it doesn't sound bad :)

pgaskin commented 4 years ago

As of supersilverhaze, the majority of the disk usage comes from locale-archive in glibc and the kernel modules in linux. Both of these are unnecessary in Docker images. locale-archive can be shrunk using localedef, and the Linux kernel isn't needed at all.

In addition grub2-efi and containerd aren't necessary either.

I'm still relatively new to distri (I noticed it a while ago, but I'm only looking at in depth now), so I'm not completely familiar with the build system yet.

stapelberg commented 4 years ago

Thanks for taking a look! For glibc locales, I had recently stumbled upon https://fedoraproject.org/wiki/QA:Glibc_locale_subpackaging, but haven’t investigated more details or how to adopt that strategy in distri. If you wanted to look into it, that’d be cool.

pgaskin commented 4 years ago

I had a look at that, and the overall concept seems pretty simple. Basically, you just patch glibc's Makefile to add --no-archive to the localedef command line in define build-one-locale, then you can take the individual locales as folders in {prefix}/lib/locale. To generate custom ones, it's just localedef --no-archive -i {locale} -f {encoding}, and you can find it in {prefix}/lib/locale/{locale}.{encoding}.

To do this, you'd need package generation, or it's not really feasible (unless you choose to ship all locales by default, but have a few separate packages for common ones like English)`. I'll have a look at distri later today.

pgaskin commented 4 years ago

I noticed that in here:

https://github.com/distr1/distri/blob/434b7298ad7ef8d4ae229df84dd2353badf48fa1/pkgs/base/build.textproto#L17-L18

, you were planning to split the packages into base-boot. Should this be done now?

stapelberg commented 4 years ago

To do this, you'd need package generation, or it's not really feasible (unless you choose to ship all locales by default, but have a few separate packages for common ones like English)`. I'll have a look at distri later today.

Having a common english locale by default and the others in an extra package sounds good to me.

, you were planning to split the packages into base-boot. Should this be done now?

Sure, why not :)

pgaskin commented 4 years ago

Sure, why not :)

What should be done with the linux and containerd packages?

P.S. I won't be able to get around to actually implementing and testing the locale stuff until next week.

stapelberg commented 4 years ago

What should be done with the linux and containerd packages?

How do you mean? They should only be depended on from the base packages that don’t go into the docker container. Is that what you mean, or did you have a different question?

pgaskin commented 4 years ago

How do you mean?

For containerd, I'm not completely sure what the TODO is referring to.: https://github.com/distr1/distri/blob/52cce77e16960797b8523b54bfa3e63c606a5c8e/pkgs/base/build.textproto#L30-L31

For linux, I just wanted to confirm that it isn't needed by anything else implicitly (the comment about base-boot is on the other block): https://github.com/distr1/distri/blob/52cce77e16960797b8523b54bfa3e63c606a5c8e/pkgs/base/build.textproto#L17-L28

stapelberg commented 4 years ago

For containerd, I'm not completely sure what the TODO is referring to.:

This is referring to: https://github.com/distr1/distri/blob/52cce77e16960797b8523b54bfa3e63c606a5c8e/cmd/distri/pack.go#L503-L522

For linux, I just wanted to confirm that it isn't needed by anything else implicitly (the comment about base-boot is on the other block):

If something else needs the linux package, it should have a dependency declared on it, no? :)

Don’t worry about breaking things too much. We can always revert or fix later.