search

lichess-org / external-engine

Using engines running outside of the browser on https://lichess.org/analysis
GNU General Public License v3.0
76 stars 22 forks source link

Proposal: Docker image build optimizations #13

Closed RickyGrassmuck closed 1 year ago

RickyGrassmuck commented 2 years ago

Hello!

While working on the pull requests submitted recently I found that building the docker image to test changes took up quite a bit of time mainly due to having to build the stockfish binaries.

I ended up making some changes locally that actually made a pretty big difference in both the time it took to build things and also resulted in much smaller image sizes.

Changes

  1. Split stockfish off into it's own separate docker image since it doesn't need to be rebuilt (this saves a good amount of time on the initial build) and based the final image on a scratch container that just contains the stockfish binaries and .deb
  2. Refactored the external-engine image build to simply copy over the stockfish binaries from it's new image.
  3. Switched to using a distroless image for the final layer in external-engine.

Image Size Improvements

Switching over to the distroless image for external-engine resulted in a significant reduction in the size of the final image.

The current docker image that builds both stockfish and external-engine ends comes in around 170Mb.

❯ docker image ls | grep 'fat'
external-engine                     fat-build         b51f1903898c   37 seconds ago      169MB

Switching to the distroless image with just the binaries from external-engine and stockfish cuts that size almost in half.

Note: the distroless-slim image below only has 1 of the 8 stockfish binaries so i'm just adding the two image sizes together for a rough estimate

❯ docker image ls | grep 'distroless-slim\|lichess-org/stockfish'
lichess-org/stockfish               15                88cf4948be4d   38 minutes ago      60.9MB
external-engine                     distroless-slim   e8c28ac942eb   23 hours ago        30.1MB

Hosted Images

My thought process around splitting off the stockfish container was that since it's an external project that doesn't change often, you can remove building it from the external-engine build process by building and pushing it to the Github container registry. That way you are only ever rebuilding the external-engine project itself resulting in significantly faster turnaround time when using a docker based development workflow.

And since the Github build pipeline is already building a container image for external-engine, throwing it up on Githubs container registry will allow users like myself who will only ever deploy it with docker do so without having to build the image at deploy time.

If this is something you'd consider pulling in just let me know and i'll cleanup what I have and submit a PR.

niklasf commented 1 year ago

Just stripped down the repository to bare minimum provider, because that's easier to maintain while the protocol is in flux. Something more fully featured will be back, and then we'll definitely have to do this properly.