Try to use multi-stage build to reduce image size

[amutake]

Most of the SATySFi users don't require ocaml & opam. I hope to be able to reduce the image size by multi-stage builds.

[y-yu]

I reduce the image size to 155MB from 1GB. https://github.com/y-yu/satysfi-docker I do the following changes.

1. Change the base image to smaller one than the official one
2. Remove OPAM dependencies after install SATySFi
   • That's the why I Install SATySFi manually. If we would install it by OPAM, it places into OPAM directory
   • However it makes me give up to install Satyrographos. Because Satyrographos has to use OPAM directories I think
   • You use Satyrographos to install fonts but it can be done download-fonts.sh so you might not mind that I give up Satyrographos

I give up install Satyrographos so I haven't submit a PR yet...

[amutake]

Thank you for your information!

• I want to install SATySFi via opam because it makes easy to install, maintain, lock version thanks to satyrographos-repo.
  • I made this repository because other SATySFi docker images don't have version tags. So the priority of tagging is higher than the priority of reducing image size.
• I noticed that official opam docker images have unnecessary compilers
  • So change the base image to smaller one is a good idea.

opam@eccfe73f39c8:~/opam-repository$ opam switch
#   switch                      compiler                                     description
->  4.07                        ocaml-base-compiler.4.07.1                   4.07
    4.07+afl                    ocaml-variants.4.07.1+afl                    4.07+afl
    4.07+default-unsafe-string  ocaml-variants.4.07.1+default-unsafe-string  4.07+default-unsafe-string
    4.07+flambda                ocaml-variants.4.07.1+flambda                4.07+flambda
    4.07+force-safe-string      ocaml-variants.4.07.1+force-safe-string      4.07+force-safe-string

• If possible, I don't want to drop Satyrographos because this is a general (not only fonts) package manager for SATySFi...
• I guess copying SATySFi binary and Satyrographos binary into new opam env via multi-stage builds solves this issue, but I have not tried yet.

[y-yu]

Thanks for your reply. I don't want to give up Satyrographos, and I want to use OPAM to install SATySFi, you too.

• It would be better looking for a function that can remove all the dependencies of the OPAM package
  • If there is the function, we can easily install SATySFi and Satyrographos keeping the Docker image size to be small
• I think we have no need to use the official Docker image of OPAM
  • You had said that, a lot of the users don't care if they can use OPAM or not.... 😢 So the small image is more important than the perfect OPAM in my opinion
• I tried to copy the Satyrographos binary into /usr/local/bin but Satyrographos uses OPAM directories. So as you said, we need
  1. First we install Satyrographos by a OPAM
  2. And then we copy the Satyrographos binary into a new OPAM
  3. Finally we delete the old OPAM using in (i)

I don't know about multi-stage builds. So please let me know about this if you don't mind 🙏

[na4zagin3]

Thanks for using Satyrographos! If my understanding is correct, the problem here is that ~/.opam directory is not removable as Satyrographos creates symlinks rather than copies, isn’t it? I'm adding -copy option to eliminate symlinks so that ~/.opam will no longer be required after satyrographos install -copy

[amutake]

@y-yu Thank you for your reply! I agree with you at all. multi-stage builds is a feature that enables to copy files that are created in a base image into another base image. By using multi-stage builds, the result image size can be reduced drastically.

ref: https://docs.docker.com/develop/develop-images/multistage-build/

Typical usecase is that (1) build an application in build environment (2) copy the application into lightweight environment.

In this case (satysfi-docker), using multi-stage builds and Satyrographos' -copy option like below, SATySFi works well and the result image size is 55MB!

FROM ocaml/opam2:ubuntu-18.04-ocaml-4.07 AS build-env

# Build SATySFi
ENV SATYSFI_VERSION=0.0.3+dev2019.03.10
RUN opam repository add satysfi-external https://github.com/gfngfn/satysfi-external-repo.git
RUN opam repository add satyrographos-repo https://github.com/na4zagin3/satyrographos-repo.git
RUN opam update
RUN opam depext satysfi.${SATYSFI_VERSION} satysfi-lib-dist.${SATYSFI_VERSION}
RUN opam install satysfi.${SATYSFI_VERSION} satysfi-lib-dist.${SATYSFI_VERSION}
RUN opam pin add https://github.com/na4zagin3/satyrographos.git # to use satyrographos.0.0.1.6
RUN eval $(opam env) && \
  satyrographos install -copy && \
  cp $(which satysfi) /home/opam/ && \
  cp $(which satyrographos) /home/opam/

FROM ubuntu:18.04

# Copy from build-env
COPY --from=build-env /home/opam/.satysfi /root/.satysfi
COPY --from=build-env /home/opam/.satyrographos /root/.satyrographos
COPY --from=build-env /home/opam/satysfi /usr/local/bin/satysfi
COPY --from=build-env /home/opam/satyrographos /usr/local/bin/satyrographos

# Setup build directory
RUN mkdir /satysfi
WORKDIR /satysfi

However, Satyrographos doesn't work as @y-yu -san said... @na4zagin3

root@aad5b213dd54:/satysfi# satyrographos
/bin/sh: 1: opam: not found
opam share dir:
Uncaught exception:

  (Failure
   "Filename.concat called with an empty string as its first argument (second argument: satysfi)")

Raised at file "stdlib.ml", line 33, characters 17-33
Called from file "src/core_filename.ml", line 25, characters 4-138
Called from file "bin/main.ml", line 46, characters 32-72

The cause of this error is that opam command is executed in satyrographos.

https://github.com/na4zagin3/satyrographos/blob/92cf9902da630f3c3f5590a803b23ae8960399fb/src/satysfiDirs.ml#L22

If opam var share fails, opam_share_dir () becomes empty string and then Filename.concat fails. To exec opam var share, opam init is required. However, opam init is a command that installs OCaml compiler and I don't want to install unnecessary OCaml compiler if possible because the size is too large...

Although SATySFi works well thanks to -copy (thank you @na4zagin3 ! :pray:), ideally I would appreciate if Satyrographos works on no-opam env. However, I don't know it is OK for the concept of Satyrographos.

[na4zagin3]

Ah, I see. Thank you for your detailed explanation, @amutake! https://github.com/na4zagin3/satyrographos/pull/22 fixed the problem where Satyrographos dies without an OPAM binary.

BTW, without an OPAM repository, Satyrographos requires a satysfi dist located at either /usr/local/share/satysfi/dist or /usr/share/satysfi/dist rather than ~/.satysfi/dist. Therefore, the copying section in the Dockerfile will be something like this. (I haven’t tested it because Docker does not yet compile in my machine by some reason :cry:)

# ...

# Copy from build-env
# COPY --from=build-env /home/opam/.satysfi /root/.satysfi
COPY --from=build-env /home/opam/.satysfi /usr/local/share/satysfi # Copy satysfi lib into /usr/share/satysfi
# COPY --from=build-env /home/opam/.satyrographos /root/.satyrographos # ~/.satyrographos is not required if no Satyrographos native packages have been installed
COPY --from=build-env /home/opam/satysfi /usr/local/bin/satysfi
COPY --from=build-env /home/opam/satyrographos /usr/local/bin/satyrographos

# ...

Though the current Satyrographos relies OPAM to resolve dependencies and to download remote files/packages, in other words, Satysfi packages are distributed as opam packages for now, it can install at least local packages.

[amutake]

Thank you again!

na4zagin3/satyrographos#22 fixed the problem where Satyrographos dies without an OPAM binary.

Thanks for the fix! But it seems that na4zagin3/satyrographos#22 is closed. Is this a mistake?

Therefore, the copying section in the Dockerfile will be something like this.

Oh, I see! I'll try it. :pray:

Satysfi packages are distributed as opam packages for now, it can install at least local packages.

Hmm.. I'm still debating which one to choose.

1. Do nothing until Satyrographos can install remote packages on opam-free environment.
   • pros: We can keep compatibility (satysfi-docker users can install remote SATySFi packages via Satyrographos easily).
   • cons: the image size is too large.
2. Switch to use multi-stage builds.
   • pros: We can reduce the image size drastically. Consequently, it makes newcomers easy to try SATySFi.
   • cons: Break compatibility (satysfi-docker users cannot install remote SATySFi packages via Satyrographos).
3. both (e.g., 0.0.3-devYYYY.MM.DD-slim means opam-free env)
   • pros: We can keep compatibility and reduce image size.
   • cons: a bit complex

I might choose 3rd option for now.

[y-yu]

@na4zagin3 thanks to your nice work!

@amutake I think it's good to give us the both choices opam-free and opam versions.

[amutake]

Thank you for your comment! I've made slim series for opam-free environment.

[amutake]

I have satysfied by the opam-slim tag (about 100MB image). Close this issue.

Thank you!