apache / accumulo-docker

Apache Accumulo Docker
https://accumulo.apache.org
Apache License 2.0
18 stars 25 forks source link

Make rebuilding the docker image for a new Accumulo snapshot faster. #20

Closed keith-turner closed 2 years ago

keith-turner commented 2 years ago

This PR is intentionally incomplete as I am seeking to improve a problem I see but I am not sure this is the best approach.

In the past when testing compactor and scan servers running Kubernetes I would go through the following process.

  1. clone accumulo docker
  2. manually download hadoop and zookeeper
  3. build a snapshot version of accumulo
  4. build an accumulo docker image
  5. push the docker image to a container repository that the Kubernetes cluster can pull from
  6. restart the accumulo processes running in Kubernetes
  7. run some experiments and make some changes to accumulo and then go to step 3

Step 4 above takes multiple minutes and creates a 2GB images. Because the image is so large it makes steps 5 and 6 take a while as the image is uploaded and then downloaded from the repo. This PR works around these problems by doing the following.

With the above changes I can have the following workflow.

  1. clone accumulo docker
  2. run download script to get hadoop and zookeeper files
  3. build the accumulo-base docker image that includes java, hadoop, and, zk
  4. build a snapshot version of accumulo
  5. build the accumulo docker image that extends accumulo-base and includes accumulo
  6. push the docker image to a container repository that the Kubernetes cluster can pull from
  7. restart the accumulo processes running in Kubernetes
  8. run some experiments and make some changes to accumulo and then go to step 4

Step 5 above takes a few seconds (vs a few minutes) and produces a new image where the layers on top of accumulo-base are only ~30MB (can see this with docker history command). The first times step 6 and 7 run, the large accumulo-base image will have to be uploaded and downloaded. However on subsequent runs of step 6 and 7 only ~30MB needs to be uploaded and downloaded, making those steps much much faster.

This is a huge improvement for what I am trying to do. I did just enough work to get this functioning. Before updating the readme, improving the docker file, and download script I would like to see if anyone has feedback.

dlmarion commented 2 years ago

Did you mean to target this against the next-release branch? Also, did you look at Docker multi-stage builds? You might only need to change the existing Dockerfile to achieve what you want.

keith-turner commented 2 years ago

Did you mean to target this against the next-release branch?

Oh I forgot about that branch, yeah I probably want to target against it.

Also, did you look at Docker multi-stage builds?

No I didn't, I will take a look at that.

brianloss commented 2 years ago

I'm not sure you even need a multi-stage build. Typically, as I've seen anyway, you would do a multi-stage build if you need to install a lot of dependencies or need a different base image to build some software, but then just want to use the build output in your final image.

Docker images are already layered, and general advice is to minimize commands in the Dockerfile that add layers as that makes the image larger. However, combining everything into a small number of layers will then mean any change causes the one (or few) layers to be rebuilt. You can see that with the single RUN command in the Dockerfile. Since the hadoop/zookeeper/accumulo download and accumulo native build are all combined into a single layer, if you want to rebuild accumulo then everything must be repeated. Simply splitting the RUN command into multiple run commands might achieve what you want. The earlier RUN commands should have the things that you don't expect to change often (e.g., download and install hadoop and zookeeper). Then the later RUN commands should have the things you want to iterate on.

keith-turner commented 2 years ago

Simply splitting the RUN command into multiple run commands might achieve what you want.

That might work. I was wondering how docker decides that it can reuse a cached layer, i looked and found the following.

https://medium.com/swlh/docker-caching-introduction-to-docker-layers-84f20c48060a https://stackoverflow.com/questions/60578670/why-does-docker-rebuild-all-layers-every-time-i-change-build-args

Based on those, it seems like it will really matter when accumulo is copied in. Going to try the following in the build file.

  1. copy hadoop and ZK in
  2. run to install jdk, other stuff,hadoop, and ZK
  3. copy accumulo in
  4. run to install accumulo

so hopefully if only accumulo changes then it will only rerun 3 and 4.

brianloss commented 2 years ago

Yes, my apologies--I should have been more specific. The order of the steps in the layers is critical, and not just RUN commands create a layer. COPY and ADD commands do as well, so their location within the file matters. I believe the order of steps you list should work fine.

keith-turner commented 2 years ago

replaced by #22