How to persist data with docker image ?

dan-dr commented 2 years ago

Hi! Incredible software!

But, when running the docker image (not with compose), it seems like it's memory only?

Is there a way to configure it with your image or do I have to build my own image?

nicolasff commented 2 years ago

Hi @dan-dr,

I assume you're referring to Redis persisting data, since Webdis is just a front-end for it?

If so, there are a few ways you could do this. One way would be to build your own image with Webdis as the parent image as you suggested, so starting your Dockerfile with something FROM nicolas/webdis:0.1.19 for example and configuring Redis there.

Another approach would be to use the published image as-is, but provide both the Redis config and a mounted directory from the host. This means creating a Redis config file locally with persistence enabled, mounting a local directory into the container, and replacing the default command from what's configured in CMD to a custom command that runs Redis with this config file. Alternatively you could also provide the Redis config as command line parameters instead of using a file.

Here's an example using the last approach.

First, create a directory for Redis data:

mkdir /tmp/redis-data

Then run the latest Webdis image with this new directory mounted into the container and a custom command line:

docker run --rm -ti -p127.0.0.1:7379:7379 \
    -v /tmp/redis-data:/tmp/redis-data \
    nicolas/webdis:latest /bin/sh -c \
    '/usr/bin/redis-server --daemonize yes --dir /tmp/redis-data && /usr/local/bin/webdis /etc/webdis.prod.json'

where:

--rm removes the container when it stops
--ti makes it an interactive execution with a terminal
the -p127.0.0.1:7379:7379 exposes port 7379 from inside the container to the same port on localhost (docs)
-v /tmp/redis-data:/tmp/redis-data mounts the local directory /tmp/redis-data to the same path in the container (docs)
the /bin/sh -c … is a replacement for the original command to run defined in the Dockerfile with CMD.
you need --daemonize yes, without it redis-server will *not* return and so Webdis won't start until Redis stops.

Once the container runs, it's easy to verify that it works. From another terminal, we can start with a PING:

curl -s http://127.0.0.1:7379/PING
{"PING":[true,"PONG"]}

Let's then write a key and read it back:

curl -s http://127.0.0.1:7379/SET/hello/world
{"SET":[true,"OK"]}

curl -s http://127.0.0.1:7379/GET/hello
{"GET":"world"}

Then ask Redis to persist its data:

curl -s http://127.0.0.1:7379/SAVE
{"SAVE":[true,"OK"]}

And finally, look into the .rdb file to see if we can find the key-value pair we just wrote:

$ xxd /tmp/redis-data/dump.rdb
00000000: 5245 4449 5330 3030 39fa 0972 6564 6973  REDIS0009..redis
00000010: 2d76 6572 0536 2e32 2e36 fa0a 7265 6469  -ver.6.2.6..redi
00000020: 732d 6269 7473 c040 fa05 6374 696d 65c2  s-bits.@..ctime.
00000030: dcaa 0162 fa08 7573 6564 2d6d 656d c225  ...b..used-mem.%
00000040: a00e 00fa 0c61 6f66 2d70 7265 616d 626c  .....aof-preambl
00000050: 65c0 00fe 00fb 0100 0005 6865 6c6c 6f05  e.........hello.
00000060: 776f 726c 64ff e142 4193 ae96 846c       world..BA....l

Yep! We can clearly see hello followed by world.

At this point, note that if you kill the Docker container (just hitting ctrl-C will work), and then re-run it with the same parameters, you should be able to read the same key again: just re-run the curl command with /GET, and you'll see the value come back.

If there are more than just a couple of parameters you want to pass to Redis, the best is to use an actual config file. It works the same way, although I'd recommend you store it separately from the data directory (so with a second -v volume mount). Something like this (untested):

mkdir /tmp/redis-data
mkdir /tmp/redis-config

Then edit /tmp/redis-config/redis.conf with all the custom settings you want Redis to use, and run Docker with:

docker run --rm -ti -p127.0.0.1:7379:7379 \
    -v /tmp/redis-data:/tmp/redis-data \
    -v /tmp/redis-config:/tmp/redis-config \
    nicolas/webdis:latest /bin/sh -c \
    '/usr/bin/redis-server /tmp/redis-config/redis.conf && /usr/local/bin/webdis /etc/webdis.prod.json'

Of course you can use the same technique to change the Webdis configuration by writing your own webdis.json file and mounting it into the container.

I hope this helps! Sorry this is a bit long, I figured that a detailed example could probably help others since I expect this to be a common situation. Ultimately which of these three approaches you choose (new image, command line parameters, or mounted config file) depend on you and your requirements.

nicolasff commented 2 years ago

Update: I've added this as a dedicated page in the docs.

dan-dr commented 2 years ago

@nicolasff absolutely amazing reply. Thanks for the info, I was missing the --dir command, because AFAIK redis by default should save a dump in /var/lib/redis/dump.rdb, but that didn't happen.

I tried giving it a go, but am facing an unrelated issue with a custom Dockerfile. I'll open another issue to keep this one on point.

nicolasff / webdis

How to persist data with docker image ? #214