search

aviggiano / redis-roaring

Roaring Bitmaps for Redis
MIT License
348 stars 56 forks source link

Add versioning of dependencies to avoid implicit upgrades and also for reproducibility #98

Open itskingori opened 2 years ago

itskingori commented 2 years ago

At the moment, there's only one version of the image on Docker Hub and that's been set to :latest i.e. sha256:0c4e97a. Typically one would be able to specify the version that you want to avoid implicit upgrades (especially since we intend to use this in a production environment). I think this would be a hard requirement before https://github.com/aviggiano/redis-roaring/issues/64 can be closed.

And so I have been trying to build the Docker image for myself so that I can lock in the versions and noticed a few things:

  1. The dependencies specified as submodules in the deps/ are pointing to really old versions:
  2. The Dockerfile is always pulling the latest Redis. This implicitly means we're implicitly pulling the latest non-rc image i.e. redis:6.2 or redis:6.2.6 or 6.2.6-bullseye etc (not entirely sure how it chooses). https://github.com/aviggiano/redis-roaring/blob/4fa6e4ebebe72cea5d23b4791812e09d4973d35c/Dockerfile#L1
  3. The .configure.sh script builds the module against the outdated packaged dependencies. Since we don't have git submodule update --remote it uses the SHA checked out against the submodule. I don't think adding --remote would be the solution as that would update the submodule to HEAD of the main branch (which is another implicit upgrade). https://github.com/aviggiano/redis-roaring/blob/4fa6e4ebebe72cea5d23b4791812e09d4973d35c/configure.sh#L6
  4. And so it seems that we're the Docker build process builds the module against old dependencies but then we have an current version of the Redis server (from the upstream Docker image specified in FROM).

On that last point you can see that the packaged Redis in the image from the upstream image is current ...

$ docker run -it aviggiano/redis-roaring:latest /bin/bash
Unable to find image 'aviggiano/redis-roaring:latest' locally
latest: Pulling from aviggiano/redis-roaring
7d63c13d9b9b: Pull complete
a2c3b174c5ad: Pull complete
283a10257b0f: Pull complete
7a08c63a873a: Pull complete
0531663a7f55: Pull complete
9bf50efb265c: Pull complete
0f3cbd753d42: Pull complete
f1fa3f8174da: Pull complete
365e8424e270: Pull complete
Digest: sha256:0c4e97accbf6821b828b46aef0b77572b6d3866b6fb3b17e9f0ffd527f8cbeb3
Status: Downloaded newer image for aviggiano/redis-roaring:latest

root@1a621edf40ad:/data# redis-server --version
Redis server v=6.2.6 sha=00000000:0 malloc=jemalloc-5.1.0 bits=64 build=1527eab61b27d3bf

But the results of the build are different ...

root@1a621edf40ad:/tmp/redis-roaring# ls -l dist/
total 28424
-rwxr-xr-x 1 root root  472048 Feb 16 10:45 libredis-roaring.so
-rwxr-xr-x 1 root root 3209632 Feb 16 10:45 redis-benchmark
-rwxr-xr-x 1 root root   30688 Feb 16 10:45 redis-check-aof
-rwxr-xr-x 1 root root 7304368 Feb 16 10:45 redis-check-rdb
-rwxr-xr-x 1 root root 3402152 Feb 16 10:45 redis-cli
-rwxr-xr-x 1 root root 7304368 Feb 16 10:45 redis-sentinel
-rwxr-xr-x 1 root root 7304368 Feb 16 10:45 redis-server
-rw-r--r-- 1 root root   57819 Feb 16 10:45 redis.conf

root@1a621edf40ad:/tmp/redis-roaring# ./dist/redis-server -v
Redis server v=999.999.999 sha=58f79e2f:0 malloc=jemalloc-4.0.3 bits=64 build=8708dba438881529

root@1a621edf40ad:/tmp/redis-roaring# git log  
commit 4fa6e4ebebe72cea5d23b4791812e09d4973d35c (grafted, HEAD, origin/master, origin/HEAD, master)
Author: Antonio Viggiano <agfviggiano@gmail.com>
Date:   Wed Nov 10 09:17:12 2021 -0500

    Update README.md

I thought of using the exact SHA of the Docker image with something like this ...

FROM aviggiano/redis-roaring:1.15.3-alpine@0c4e97accbf6821b828b46aef0b77572b6d3866b6fb3b17e9f0ffd527f8cbeb3

But I'm not sure that is reliable either as I the Dockerfile used to build that image doesn't seem to be similar to the one in this repository.

Screenshot 2022-02-16 at 15 06 28

There's a step in there that makes it look like we're copying in other files into the image ...

COPY file:df205a0ef6e6df8947ce0a7ae9e37b6a5588035647f38a49b8b07321003a8a01 in /usr/local/bin/
aviggiano commented 2 years ago

Hi @itskingori

Thank you for the interest in this project and for submitting this issue I'm currently on vacation but I'll take some time to review this issue as soon as I get back. If you have a PR feel free to submit it!

Get back to you soon

itskingori commented 2 years ago

@aviggiano That's alright. Check out https://github.com/aviggiano/redis-roaring/pull/99 when you get back.

itskingori commented 2 years ago

@hadrianvalentine @zacblazic Just tagging you guys so that you're in the loop.