How to populate image with default data?

[icougil]

Hi! We are starting to work with Consul and after a while we'd love to have our own image setup with a pre-default data (in our case at least some key/values already stored when we launch a container with that image). What we're trying to do is, based in your container, populate it when it boots up with our predefined data. In our case, we are using our own Dockerfile that already extends yours, something like this:

FROM consul:latest
MAINTAINER xxxx
EXPOSE 8300 8301/udp 8302 8302/udp 8400 8500 8600 8600/udp
ENV BASE /xxx
RUN mkdir -p ${BASE}/data
COPY populate_keys.sh ${BASE}/
COPY data/ ${BASE}/data
RUN chmod +x ${BASE}/populate_keys.sh
CMD ${BASE}/populate_keys.sh

In our case, the script that try to populate the server tries to run many curl PUT operations just to fill the server with our specific data. In our case, populate_keys.shis something like this:

#!/bin/sh
set -x -e
HEADERS="Content-Type: application/json;charset=UTF-8"
URL=http://localhost:8500/v1/kv/config
/usr/bin/curl -X PUT -H \""$HEADERS\"" --data @hello_kv.txt \""$URL/hello/data\""

The problem is that is seems that consul server does not start as it is in your container image.

What we are missing? How we can try to "improve" your consul docker image just by adding our own specific data and also have the consul elements running (agent, UI, etc)?

Best,

[kyhavlov]

Hi @cougil, Docker will only manage one process for a container, so when you override the CMD in your image, docker runs that script instead of starting Consul. You'd have to either make your script launch Consul in addition to doing the kv setup, or have something outside of that container be responsible for populating the keys.

[ryansch]

I went with using another container to manage the k/v store: https://github.com/outstand/consul_stockpile

Edit: Here's an example of how I'm running it locally.

consul_stockpile:
  image: outstand/consul_stockpile:0.1.5
  restart: always
  labels:
    - io.rancher.os.remove=false
  net: host
  command: start -b bucket -n backup
  volumes:
    - /opt/consul_stockpile:/fog
  environment:
    - FOG_LOCAL=true

In production it pushes and pulls from S3.

[icougil]

Hi everybody!

@kyhavlov yes, I know that Docker can only run a single script, but this does not mean that a container can not run a bunch of scripts before exiting, isn't it?. For example, in the case of the official Postgres docker image, it is possible to launch additional SQL scripts after the container has booted up. You can have a look in the official page and look on the "How to extend this image" section. At the end I understand that it consist only on "scanning" the content of a folder and if it contains some scripts, try to run them (based on the boot starter script of the same image.

@ryansch sorry, I do not understand what do you mean and how it works. How are you populating the k/v store? Using the consul_stockpile image? But how? What do you mean with pushed and pulls from S3? What it is trying to pull & push? And based on what? Sorry, but I don't get how is working ;-)

[ryansch]

@cougil So in that example I have the directory /opt/consul_stockpile mounted into the container. Inside that directory is another named bucket (it's pretending to be an S3 bucket). Inside /opt/consul_stockpile/bucket is a file named backup.json which has the following format:

[{"key":"stockpile/canary","value":"tweet"}]

You can populate consul's k/v store with whatever you want and then trigger a kv_update event like so:

docker run -it --rm --net=host consul:0.7.1 event -name 'kv_update'

That will trigger a backup to that file. When you run consul_stockpile it will automatically check for the canary key and populate the k/v store if it doesn't exist.

Does that make sense? Please let me know if you run into issues.

Edit: You can use a docker volume, host mount, or storage container. Just make sure the data is mounted at /fog in the container.

[icougil]

Ok I understand. Well, in our case I've prefered to not run any other docker image just to populate the consul image itself, also we should adapt our values to be included as a valid JSON element, which already is not, but then I should adapt it in order to create the correct backup.json file. Also we have a lot of configuration files based on different configurations and keys and I don't like to mix both of them and I'd like to be able to change it easily. What I've done is just create a little script that is responsible of booting up consul and scan an specific folder that could contain many scripts and run it in order to populate consul itself (via curl in our case). Best,

[ryansch]

@cougil Excellent! I'm glad you found a good solution.

[nileshkamani82]

Hello cougil, could you please share your solution if you do not mind ?

[nileshkamani82]

@cougil Hello cougil, could you please share your solution if you do not mind ?

[MattiNieminen]

@cougil I'm also very interested about this one!