Redis Stack support - Githubissues

raphaelzoellner commented 1 year ago

We would like to deploy Redis Stack with the redis-operator. Is the operator already supporting Redis Stack?

Is your feature request related to a problem? Please describe.

We would like to deploy an redis-operator managed Redis Stack instance.

Describe the solution you'd like

An additional docker image based on redis-stack configured to be deployed by the redis-operator in addition to the already existing quay.io/opstree/redis opstree/redis image.

Describe alternatives you've considered

What version of redis-operator are you using?

redis-operator version: v0.13.0

Additional context

shubham-cmyk commented 1 year ago

@raphaelzoellner Thanks for suggesting this change i would work on it.

the-wondersmith commented 1 year ago

@raphaelzoellner You can already do this without too much fuss. Use this to build the appropriate Dockerfile:

FROM quay.io/opstree/redis:v7.0.5

USER 0:0

RUN apk add --no-cache libgcc libstdc++ gcompat libc6-compat
COPY --from=docker.io/redis/redis-stack-server:latest /opt/redis-stack/lib/* /usr/lib/redis/modules/

USER 1000:1000

Then just use something like this as your deployment:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-additional-config
data:
  redis-additional.conf: |
    # repeat the line below for whichever other modules you need
    loadmodule /usr/lib/redis/modules/rejson.so
---
apiVersion: redis.redis.opstreelabs.in/v1beta1
kind: RedisCluster
metadata:
  name: cache
spec:
  clusterSize: 3
  clusterVersion: v7
  persistenceEnabled: true
  securityContext:
    runAsUser: 1000
    fsGroup: 1000
  kubernetesConfig:
    image: "{YOUR IMAGE REPO HOST}/{YOUR IMAGE NAME}:{SOME TAG}"
    imagePullPolicy: Always
    resources:
      requests:
        cpu: 1000m
        memory: 4Gi
      limits:
        cpu: 1000m
        memory: 4Gi
    imagePullSecrets:
      - name: "{YOUR IMAGE PULL SECRET NAME HERE}"
  redisLeader:
    redisConfig:
      additionalRedisConfig: redis-additional-config
  redisFollower:
    redisConfig:
      additionalRedisConfig: redis-additional-config
  storage:
    volumeClaimTemplate:
      spec:
        accessModes:
          - "ReadWriteOnce"
        resources:
          requests:
            storage: 1Gi

Twinki14 commented 1 year ago

+1, would be very beneficial

Routhinator commented 12 months ago

To keep this example as close to the offiicial opstree image as possible, the last USER line should be removed.

And then the securityContext override can be dropped if not already being used.

svabra commented 4 months ago

@shubham-cmyk @the-wondersmith I just wonder, why would we NOT use the redis-stack-server image as the default one ? The license is not an obstacle. We can always have it loaded on-demand (when starting up). I don't see a reason to maintain multiple images for the redis cluster.

svabra commented 4 months ago

@raphaelzoellner You can already do this without too much fuss. Use this to build the appropriate Dockerfile:

FROM quay.io/opstree/redis:v7.0.5

USER 0:0

RUN apk add --no-cache libgcc libstdc++ gcompat libc6-compat
COPY --from=docker.io/redis/redis-stack-server:latest /opt/redis-stack/lib/* /usr/lib/redis/modules/

USER 1000:1000

Then just use something like this as your deployment:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-additional-config
data:
  redis-additional.conf: |
    # repeat the line below for whichever other modules you need
    loadmodule /usr/lib/redis/modules/rejson.so
---
apiVersion: redis.redis.opstreelabs.in/v1beta1
kind: RedisCluster
metadata:
  name: cache
spec:
  clusterSize: 3
  clusterVersion: v7
  persistenceEnabled: true
  securityContext:
    runAsUser: 1000
    fsGroup: 1000
  kubernetesConfig:
    image: "{YOUR IMAGE REPO HOST}/{YOUR IMAGE NAME}:{SOME TAG}"
    imagePullPolicy: Always
    resources:
      requests:
        cpu: 1000m
        memory: 4Gi
      limits:
        cpu: 1000m
        memory: 4Gi
    imagePullSecrets:
      - name: "{YOUR IMAGE PULL SECRET NAME HERE}"
  redisLeader:
    redisConfig:
      additionalRedisConfig: redis-additional-config
  redisFollower:
    redisConfig:
      additionalRedisConfig: redis-additional-config
  storage:
    volumeClaimTemplate:
      spec:
        accessModes:
          - "ReadWriteOnce"
        resources:
          requests:
            storage: 1Gi

I have copied from the V7.2.0-v8-x86_64 image all the modules.

RUN apk add --no-cache libgcc libstdc++ gcompat libc6-compat
COPY --from=docker.io/redis/redis-stack-server:7.2.0-v8-x86_64 /opt/redis-stack/lib/* /usr/lib/redis/modules/

The rejson.so module can be loaded but not any other redis-stack-server modules. In this case it is the redisearch.so, which fails.

kubectl logs redis-cluster-leader-0 -n fa
Redis is running without password which is not recommended
Running without TLS mode
ACL_MODE is not true, skipping ACL file modification
Starting redis service in cluster mode.....
1:C 11 Feb 2024 20:25:46.131 # WARNING: Changing databases number from 16 to 1 since we are in cluster mode
1:C 11 Feb 2024 20:25:46.131 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 11 Feb 2024 20:25:46.131 * Redis version=7.2.3, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 11 Feb 2024 20:25:46.131 * Configuration loaded
1:M 11 Feb 2024 20:25:46.132 * monotonic clock: POSIX clock_gettime
1:M 11 Feb 2024 20:25:46.133 # Failed to write PID file: Permission denied
1:M 11 Feb 2024 20:25:46.133 * Running mode=cluster, port=6379.
1:M 11 Feb 2024 20:25:46.133 * Node configuration loaded, I'm d3efce5508954c8727b8af28dfe9c20ab4170da0
1:M 11 Feb 2024 20:25:46.133 * <ReJSON> version: 20407 git sha: a4412b3 branch: HEAD
1:M 11 Feb 2024 20:25:46.133 * <ReJSON> Exported RedisJSON_V1 API
1:M 11 Feb 2024 20:25:46.134 * <ReJSON> Exported RedisJSON_V2 API
1:M 11 Feb 2024 20:25:46.134 * <ReJSON> Exported RedisJSON_V3 API
1:M 11 Feb 2024 20:25:46.134 * <ReJSON> Enabled diskless replication
1:M 11 Feb 2024 20:25:46.134 * <ReJSON> Created new data type 'ReJSON-RL'
1:M 11 Feb 2024 20:25:46.134 * Module 'ReJSON' loaded from /usr/lib/redis/modules/rejson.so
1:M 11 Feb 2024 20:25:46.134 # Module /usr/lib/redis/modules/redisearch.so failed to load: Error relocating /usr/lib/redis/modules/redisearch.so: __printf_chk: symbol not found
1:M 11 Feb 2024 20:25:46.134 # Can't load module from /usr/lib/redis/modules/redisearch.so: server aborting

Same for the redisgraph.so module and redistimeseries.so

Module /usr/lib/redis/modules/redistimeseries.so failed to load: Error relocating /usr/lib/redis/modules/redistimeseries.so: __printf_chk: symbol not found

There is too many IFs and BUTs. Thus I recommend to choose the https://hub.docker.com/r/redis/redis-stack-server version v7.2.0 by default instead and allow to activate the modules in the operator config.

networkingana commented 4 months ago

I used the redis-stack-serve v7.2.0 but the pods are shutting down

8:C 14 Feb 2024 11:40:27.870 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
8:C 14 Feb 2024 11:40:27.870 * Redis version=7.2.4, bits=64, commit=00000000, modified=0, pid=8, just started
8:C 14 Feb 2024 11:40:27.870 * Configuration loaded
8:M 14 Feb 2024 11:40:27.870 * monotonic clock: POSIX clock_gettime
8:M 14 Feb 2024 11:40:27.871 * Running mode=standalone, port=6379.
8:M 14 Feb 2024 11:40:27.872 * Module 'RedisCompat' loaded from /opt/redis-stack/lib/rediscompat.so
8:M 14 Feb 2024 11:40:27.874 * <search> Redis version found by RedisSearch : 7.2.4 - oss
8:M 14 Feb 2024 11:40:27.874 * <search> RediSearch version 2.8.11 (Git=2.8-7368bb3)
8:M 14 Feb 2024 11:40:27.874 * <search> Low level api version 1 initialized successfully
8:M 14 Feb 2024 11:40:27.874 * <search> concurrent writes: OFF, gc: ON, prefix min length: 2, prefix max expansions: 200, query timeout (ms): 500, timeout policy: return, cursor read size: 1000, cursor max idle (ms): 300000, max doctable size: 1000000, max number of search results:  10000, search pool size: 20, index pool size: 8,
8:M 14 Feb 2024 11:40:27.874 * <search> Initialized thread pools!
8:M 14 Feb 2024 11:40:27.874 * <search> Enabled role change notification
8:M 14 Feb 2024 11:40:27.875 * Module 'search' loaded from /opt/redis-stack/lib/redisearch.so
8:M 14 Feb 2024 11:40:27.876 * <timeseries> RedisTimeSeries version 11011, git_sha=0299ac12a6bf298028859c41ba0f4d8dc842726b
8:M 14 Feb 2024 11:40:27.877 * <timeseries> Redis version found by RedisTimeSeries : 7.2.4 - oss
8:M 14 Feb 2024 11:40:27.877 * <timeseries> loaded default CHUNK_SIZE_BYTES policy: 4096
8:M 14 Feb 2024 11:40:27.877 * <timeseries> loaded server DUPLICATE_POLICY: block
8:M 14 Feb 2024 11:40:27.877 * <timeseries> Setting default series ENCODING to: compressed
8:M 14 Feb 2024 11:40:27.877 * <timeseries> Detected redis oss
8:M 14 Feb 2024 11:40:27.877 * Module 'timeseries' loaded from /opt/redis-stack/lib/redistimeseries.so
8:M 14 Feb 2024 11:40:27.878 * <ReJSON> Created new data type 'ReJSON-RL'
8:M 14 Feb 2024 11:40:27.878 * <ReJSON> version: 20609 git sha: unknown branch: unknown
8:M 14 Feb 2024 11:40:27.878 * <ReJSON> Exported RedisJSON_V1 API
8:M 14 Feb 2024 11:40:27.878 * <ReJSON> Exported RedisJSON_V2 API
8:M 14 Feb 2024 11:40:27.878 * <ReJSON> Exported RedisJSON_V3 API
8:M 14 Feb 2024 11:40:27.878 * <ReJSON> Exported RedisJSON_V4 API
8:M 14 Feb 2024 11:40:27.878 * <ReJSON> Exported RedisJSON_V5 API
8:M 14 Feb 2024 11:40:27.878 * <ReJSON> Enabled diskless replication
8:M 14 Feb 2024 11:40:27.878 * Module 'ReJSON' loaded from /opt/redis-stack/lib/rejson.so
8:M 14 Feb 2024 11:40:27.878 * <search> Acquired RedisJSON_V5 API
8:M 14 Feb 2024 11:40:27.878 * <bf> RedisBloom version 2.6.11 (Git=unknown)
8:M 14 Feb 2024 11:40:27.878 * Module 'bf' loaded from /opt/redis-stack/lib/redisbloom.so
8:M 14 Feb 2024 11:40:27.879 * <redisgears_2> Created new data type 'GearsType'
8:M 14 Feb 2024 11:40:27.879 * <redisgears_2> Detected redis oss
8:M 14 Feb 2024 11:40:27.879 # <redisgears_2> could not initialize RedisAI_InitError

8:M 14 Feb 2024 11:40:27.879 * <redisgears_2> Failed loading RedisAI API.
8:M 14 Feb 2024 11:40:27.879 * <redisgears_2> RedisGears v2.0.16, sha='9559f21d0c3fd6ab4a0edb55d8b89ab125c8cc52', build_type='release', built_for='Linux-ubuntu22.04.x86_64'.
8:M 14 Feb 2024 11:40:27.881 * <redisgears_2> Registered backend: js.
8:M 14 Feb 2024 11:40:27.881 * Module 'redisgears_2' loaded from /opt/redis-stack/lib/redisgears.so
8:M 14 Feb 2024 11:40:27.882 * Server initialized
8:M 14 Feb 2024 11:40:27.882 * <search> Loading event starts
8:M 14 Feb 2024 11:40:27.882 * <redisgears_2> Got a loading start event, clear the entire functions data.
8:M 14 Feb 2024 11:40:27.883 * Loading RDB produced by version 7.0.5
8:M 14 Feb 2024 11:40:27.883 * RDB age 2154 seconds
8:M 14 Feb 2024 11:40:27.883 * RDB memory usage when created 156.95 Mb
8:M 14 Feb 2024 11:40:29.194 * Done loading RDB, keys loaded: 20, keys expired: 4.
8:M 14 Feb 2024 11:40:29.194 # <search> Skip background reindex scan, redis version contains loaded event.
8:M 14 Feb 2024 11:40:29.194 * <search> Loading event ends
8:M 14 Feb 2024 11:40:29.194 * <redisgears_2> Loading finished, re-enable key space notificaitons.
8:M 14 Feb 2024 11:40:29.194 * DB loaded from disk: 1.312 seconds
8:M 14 Feb 2024 11:40:29.194 * Ready to accept connections tcp
8:signal-handler (1707910857) Received SIGTERM scheduling shutdown...
8:signal-handler (1707910857) Received SIGTERM scheduling shutdown...

svabra commented 4 months ago

You are almost there. Remove the RedisGear module for now to see if that is the culprit. (btw, why did you set the RDB instead of AOF?).

Anyhow, moving to the redis-stack-server image would solve this as well. Let's see what the team says before we send PRs.

networkingana commented 4 months ago

It seems like this new image does not read the /etc/redis/redis.conf, so also starts in standalone mode, I've got really confused, can you please help me a bit how this should be actually configured?

svabra commented 4 months ago

It seems like this new image does not read the /etc/redis/redis.conf, so also starts in standalone mode, I've got really confused, can you please help me a bit how this should be actually configured?

Please describe what you have don so far.

What images you are using?
Have you buildt one youself as suggested by @the-wondersmith ?
On what OS did you build the image?
what operator version you are using?
how do you load the modules? e.g. redistimeseries, redisearch, etc.
what arguments do you pass to the loading of the modules?
anything else that helps to understand what you did so far.

networkingana commented 4 months ago

I tried to use the image that i built based on this Dockerfile in my RedisCluster YAML file.

FROM docker.io/redis/redis-stack-server:7.2.0-v8-x86_64

USER 0:0

RUN rm /opt/redis-stack/lib/libredisgears_v8_plugin.so
RUN rm /opt/redis-stack/lib/redisgears.so
USER 1000:1000

I only need the redistimeseries module. I tried here with sed to remove the loadmodule command from the redis.conf, but I see that this isn't the way.

The operator version is: quay.io/opstree/redis-operator:v0.13.0

Should I go from some OS docker image and then add the redis-stack-server?

svabra commented 4 months ago

There is no point in building the your own image if you are referencing the redis-stack-server image :-) e.g. FROM docker.io/redis/redis-stack-server:7.2.0-v8-x86_64

Anyhow, it shows the problem here that merely @shubham-cmyk and the core team could solve. I doubt you can trick it with a workaround.

As you can see, your pod is starting successfully Ready to accept connections tcp. Right after it receives a SIGTERM - which is probably sent by the K8s system invoked by the opstree operator running a check, which fails.

8:M 14 Feb 2024 11:40:29.194 * Ready to accept connections tcp
8:signal-handler (1707910857) Received SIGTERM scheduling shutdown...

Anyhow, I have also tried to build an image with images, which stem from the closes release.

FROM quay.io/opstree/redis:v7.0.5
USER 0:0
RUN mkdir -p /var/run/redis && chown 1000:1000 /var/run/redis
RUN apk add --no-cache libgcc libstdc++ gcompat libc6-compat libgomp
#COPY --from=docker.io/redis/redis-stack-server:latest /opt/redis-stack/lib/* /usr/lib/redis/modules/
COPY --from=docker.io/redis/redis-stack-server:7.0.6-RC9 /opt/redis-stack/lib/* /usr/lib/redis/modules/

It always results in the same. Hinting that the library redis-stack-server:7.0.6-RC uses a different lib to compile than the quay.io/opstree/redis:v7.0.5 lib. The same applies for the combination of redis-stack-server:7.2.0-v8 and opstree/redis:v7.2.3

kc logs redis-cluster-leader-0 -n fa
Redis is running without password which is not recommended
Running without TLS mode
Starting redis service in cluster mode.....
10:C 15 Feb 2024 15:51:17.608 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
10:C 15 Feb 2024 15:51:17.608 # Redis version=7.0.5, bits=64, commit=00000000, modified=0, pid=10, just started
10:C 15 Feb 2024 15:51:17.608 # Configuration loaded
10:M 15 Feb 2024 15:51:17.609 * monotonic clock: POSIX clock_gettime
10:M 15 Feb 2024 15:51:17.609 * Node configuration loaded, I'm 70050ea8328bea19e9ee19700cc12ff002d52bad
10:M 15 Feb 2024 15:51:17.610 * Running mode=cluster, port=6379.
10:M 15 Feb 2024 15:51:17.610 # Server initialized
10:M 15 Feb 2024 15:51:17.611 * <ReJSON> version: 20407 git sha: a4412b3 branch: HEAD
10:M 15 Feb 2024 15:51:17.611 * <ReJSON> Exported RedisJSON_V1 API
10:M 15 Feb 2024 15:51:17.611 * <ReJSON> Exported RedisJSON_V2 API
10:M 15 Feb 2024 15:51:17.611 * <ReJSON> Exported RedisJSON_V3 API
10:M 15 Feb 2024 15:51:17.611 * <ReJSON> Enabled diskless replication
10:M 15 Feb 2024 15:51:17.611 * <ReJSON> Created new data type 'ReJSON-RL'
10:M 15 Feb 2024 15:51:17.611 * Module 'ReJSON' loaded from /usr/lib/redis/modules/rejson.so
10:M 15 Feb 2024 15:51:17.611 # Module /usr/lib/redis/modules/redistimeseries.so failed to load: Error relocating /usr/lib/redis/modules/redistimeseries.so: __printf_chk: symbol not found
10:M 15 Feb 2024 15:51:17.611 # Can't load module from /usr/lib/redis/modules/redistimeseries.so: server aborting

My conclusion remains, I would suggest that the team changes the base image of the opstree/redis to redis-stack-server. This way the existing users run it an also the users requiring time series, json, etc.

@shubham-cmyk How can we raise an official feature request?

svabra commented 4 months ago

In this poll you can vote for this feature? Please do, when you come across this issue. https://github.com/OT-CONTAINER-KIT/redis-operator/discussions/776

OT-CONTAINER-KIT / redis-operator

Redis Stack support #505