Columnstore support

[flopana]

Hello since MariaDB 10.5.4 Columnstore is available as a storage engine. Currently this docker image doesn't support columnstore and you have to use this one. The problem is that the columnstore image is rarely updated and is still on 10.6 while the current version is 10.9.2.

Will we get columnstore support for this image to in the near future?

[grooverdan]

@flopana, thanks for your interest in ColumnStore.

I've look this over with @LinuxJedi and yes, we are willing to get the Docker Library container to support ColumnStore. Are you (or anyone watching) willing to help create the Dockerfile and entrypoint to make this happen?

The constraints of a Docker Library project is it need to follow the guidelines. Specificity one concern per container (and not resort to the init/tini section in the FAQ) is the best policy and allows individual resource control on each running container.

The design I'd like to see is:

• this well be a separate image publish with some form of cs tags
• a separate Dockerfile.cs.template at the top level (which will generate a bunch of 10.x-cs subdirectories)
• there will be a separate entrypoint script for cs docker-entrypoint-cs.sh (the original is needed too, see later)
• the cs entrypoint will take a command corresponding to the subcomponents of ColumnStore that need to run (StorageManager, brm, controlnode, .....)

the running system will be the form of a compose file similar to:

mariadb:
  image: mariadb:10.9-cs
  command: mariadbd
mariadb-sm:
  image: mariadb:10.9-cs
  command: StorageManager
  ipc: shared
mariadb-brm:
  image: mariadb:10.9-cs
  command: brm
  ipc: shared
mariadb-controllernode:
  image: mariadb:10.9-cs
  command: controllernode
  ipc: shared
...

So this will start all components. ColumnStore used shared memory to communicate, so this needs to be created.

Basic form of Dockerfile.cs.template:

FROM mariadb:10.X

RUN apt-get update && \
        apt-get install -y mariadb-plugin-columnstore=$MARIADB_VERSION ; \
        rm -rf /var/lib/apt/lists/*

COPY docker-entrypoint-cs.sh /usr/local/bin/

ENTRYPOINT ["docker-entrypoint-cs.sh"]

I don't yet know how this works in the building process but will find out. Its a good simple starting point for the harder task of entrypoint development. The entrypoint needs to:

• identify which specific item is being executed (looking at $1)
• if mariadbd, a straight exec docker-entrypoint.sh "$@";
• other components need to conduct initialisation, if not already done, waiting for other components, and starting the service
• https://github.com/mariadb-corporation/mariadb-community-columnstore-docker/tree/master/scripts should provide some insight to the things needed.

I don't know how to configure it so that it knows where everything is. There might be a etcd service on common services. Maybe a shared config volume is needed.

This is all rather new ground. We're here to help on chat https://mariadb.zulipchat.com at various times. There may be bugs, so please report these (MCOL project) along with improvements that simplify the implementation.

I suggest start by forking this repository getting a docker compose file, a Dockerfile and entrypoint going and share the link here with instructions for people (like us) to help.

Please include notes as comments here on progress and lessons learnt.

[pjflux2001]

@grooverdan @LinuxJedi I would like to take up this issue as my project for GSOC'23

[mrn3]

Is there any progress on this? I would love to see this progress. Installing MariaDB ColumnStore is a pain, and would be so nice to have a simple Docker image for it.

[grooverdan]

Maybe you could be the one to provide it.

[drrtuy]

@grooverdan I like the approach when --ipc=shareable is used. There is a prerequisite to solve, namely the shared memory components loader must start before any other process. JFYI Columnstore team is now publishing the cluster management tool of the Columnstore. When it is done there will be a separate Columnstore image that is based on MDB 11.1 and tini.

[grooverdan]

There should be a way to probe the memory space for the started components right?

[drrtuy]

Did I get it right the probe here means per-container RAM metering and limiting @grooverdan ?

[grooverdan]

I was thinking probed the share memory for the finished startup of the memory components loader in some way, and after those probe tests, the dependent services can start.

[drrtuy]

There are multiple ways to get the containers in the correct order:

• use a shmem handover file
• use container dependencies

[grooverdan]

Just FYI for those that missed the link in the original issue - there is an image on https://hub.docker.com/r/mariadb/columnstore/ that is maintained by MariaDB plc.

[flopana]

Just FYI for those that missed the link in the original issue - there is an image on https://hub.docker.com/r/mariadb/columnstore/ that is maintained by MariaDB plc.

I cant recommend this image. It is rarely updated and poorly maintained. I've had multiple issues where the image would behave completely differently after an update. When issues about these problems were opened it has happened multiple times that they were ignored or even they didn't know what happend. Last time around the columnstore engine was broken for like a month. The whole point of this image is to provide columnstore support which they broke for everyone and didn't even gave us a chance to downgrade again since they kindly delete old images.

Edit: Here is an example https://github.com/mariadb-corporation/mariadb-columnstore-docker/issues/24#issuecomment-1378465756