Is there a way to add indexes and sets on docker container creation?

[kuskmen]

Hi, I am new to devops world as a programmer I would like to have image that when built uses AS base docker image and then runs some aql commands.

What I have so far is:

FROM aerospike:latest AS base

RUN aql CREATE SET set1 && \
         aql CREATE SET set2 && \
         aql CREATE INDEX index1 ON Namespace.set1 (bin1) STRING

Obviously this wouldnt work, but I want to make these sets and indexes as I am creating docker image.

Can you guys help me out?

[volmarl]

Interesting idea. Have you tried creating an aql script file with these commands and run it with an aql -f mysript.aql? You could also run aql with -c option directly in your run command.

aql -c "CREATE INDEX index1 ON Namespace.set1 (bin1) STRING"

The set would have to be created. by inserting a dummy record or can configure the set (to have a threshold or be non-evictable)

[rbotzer]

You should be able to declare indexes from a script that aql can execute, as Lucien suggested.

However, you cannot (and don't need to) declare sets. Aerospike is schemaless and doesn't have actual tables. Records are all stored within a namespace, which does have storage, replication factor and other top level parameters defined. See more about the data model.

A set is a virtual concept, a label on the record that is created on-demand when new records are written. When you first insert a record into a set, it gets created on the fly. It's really just a label on the record.

So in your example, you just need to have the one CREATE INDEX line.

[kuskmen]

I ended up doing aerospike.template.conf for my namespaces. Then some aql file for index creation.

My Dockerfile ended up looking something like that:

FROM aerospike:latest AS base # I know I will change to concrete version

COPY ./indexes.aql ./indexes.aql
COPY ./aerospike.template.conf /etc/aerospike/aerospike.template.conf

RUN mv /etc/aerospike/aerospike.template.conf /etc/aerospike/aerospike.conf # Probably can merge this and the previous line

ENTRYPOINT ["/entrypoint.sh"]
CMD ["asd"]

In the end I have one script to execute:

# Remove existing containers
docker rm $(docker stop $(docker ps -a -q --filter ancestor=settlement/aerospike --format="{{.ID}}"))

# Build settlment aerospike container
docker build -t settlement/aerospike .
docker run --name settlementAerospike -d -it -p 3006:3000 -p 3007:3001 -p 3008:3002 -p 3009:3003  settlement/aerospike
docker exec -it settlementAerospike /bin/bash -c "until aql -f indexes.aql; do echo Sleep 2 seconds...; sleep 2; done"

And that settled the deal, if you have any suggestions how to improve it please dont hesitate to tell me, after all I am still a new guy in a devops world and I'd love to hear some opinions :man_dancing:

Regards, kuskmen

[spkesan]

@kuskmen

You don't have to build a custom image for all this. Also, not a best practice to build images on top of published images unless it's really necessary.

First, let me clarify that for applying your own configuration file you could simply do this,

docker run -tid -v <PATH_TO_YOUR_CUSTOM_AEROSPIKE_CONF_ON_HOST_MACHINE>:/opt/aerospike/etc/aerospike.conf \
--name aerospike \
-p 3000:3000 \
-p 3001:3001 \
-p 3002:3002 \
-p 3003:3003 \
aerospike/aerospike-server /usr/bin/asd --foreground --config-file /opt/aerospike/etc/aerospike.conf

Second, for creating multiple secondary indices in one shot, you can do this in multiple ways.

1. Mount the indexes.aql script file using volumes. Let's modify the above command now ...

   docker run -tid \
   -v <PATH_TO_YOUR_CUSTOM_AEROSPIKE_CONF_ON_HOST_MACHINE>:/opt/aerospike/etc/aerospike.conf \
   - v <PATH_TO_INDEXES_AQL_SCRIPT>:/opt/aerospike/etc/indexes.aql \
   --name aerospike \
   -p 3000:3000 \
   -p 3001:3001 \
   -p 3002:3002 \
   -p 3003:3003 \
   aerospike/aerospike-server /usr/bin/asd --foreground --config-file /opt/aerospike/etc/aerospike.conf

docker exec -it aerospike /bin/bash -c "aql -f /opt/aerospike/etc/indexes.aql"

OR,

2. Since you are exposing aerospike service port 3000 via docker host, you can also initiate this remotely using,

aql -h <DOCKER_HOST_IP> -p 3000 -f <PATH_TO_INDEXES_AQL_FILE>

You just need AQL tool installed from where you want to run the command.

OR,

3. A better approach for such initialization on startup is to use an additional container whose job is to just create secondary indices on an Aerospike container startup.

For example, using a docker-compose.yml file,

version: '2'

services:
  aerospike:
    container_name: aerospike
    image: aerospike/aerospike-server:latest
    volumes:
      - <PATH_TO_YOUR_CUSTOM_AEROSPIKE_CONF_ON_HOST_MACHINE>:/opt/aerospike/etc/aerospike.conf
    networks:
      - test_net
    command:
      - /usr/bin/asd --foreground --config-file /opt/aerospike/etc/aerospike.conf
  init:
    container_name: init
    image: aerospike/aerospike-tools:latest
    networks:
      - test_net
    volumes:
      - <PATH_TO_INDEXES_AQL_FILE>:/opt/aerospike/etc/indexes.aql
    links:
      - aerospike
    depends_on:
      - aerospike
    command:
      - aql -h aerospike -p 3000 -f /opt/aerospike/etc/indexes.aql

networks:
  test_net:
    driver: bridge

Hope this helps.

[kuskmen]

Thanks so much @spkesan for the insights!