Unable to Create CRDB Active-Active database in kubernetes Cluster

[Deepanshu6dec]

I have a Self manage kubernetes cluster(kubeadm in Ec2 instance) Will have to deploy a Redis Enterprise into the cluster. I have successfully Deploy a three-node redis Enterprise Cluster and deploy a test database. It's working well.

NAME       NODES   VERSION     STATE     SPEC STATUS   LICENSE STATE   SHARDS LIMIT   LICENSE EXPIRATION DATE   AGE
rec   3       6.2.12-82   Running   Valid         Valid           4              2022-10-29T09:42:03Z      3h13m

Now I have to deploy an Active-Active CRDB ( Conflict Free Replication Database ). I tried following the document but it seems it was not created.

https://docs.redis.com/latest/kubernetes/re-clusters/create-aa-database/

Can any one assist me on this issue.

[laurentdroin]

Hi,

Yes, we can assist you, but we will need details. Please open a Support Ticket with Redis and provide:

• the archive generated by running the log collector script for each participating Redis Enterprise cluster (see https://docs.redis.com/latest/kubernetes/logs/collect-logs/).
• the command you ran to create the CRDB database as well as its console output.
• what is the ingress controller being used on each k8s cluster.

Laurent.

[Deepanshu6dec]

Hi,

Please find the Collector script logs.

2022-09-29 13:13:28,470 - INFO - Started Redis Enterprise k8s log collector
2022-09-29 13:13:28,683 - INFO - Namespace 'None': Collected cluster-info
2022-09-29 13:13:28,772 - INFO - Started collecting from namespace 'redis'
2022-09-29 13:13:28,980 - INFO - Namespace 'None': Collected connectivity check via curl
2022-09-29 13:13:29,102 - INFO - Namespace 'redis': Collected connectivity check via k8s cli
2022-09-29 13:13:29,185 - INFO - Trying to extract debug info from RS pods: {[u'rec-test-0', u'rec-test-1', u'rec-test-2']}

2022-09-29 13:14:17,036 - INFO - Namespace 'redis': debug info created on pod rec-test-0 in path /tmp/debuginfo.20220929-131330.tar.gz
2022-09-29 13:14:17,547 - INFO - Namespace 'redis': Collected Redis Enterprise cluster debug package from pod: rec-test-0
2022-09-29 13:14:19,347 - INFO - Namespace 'redis': Collected rs logs from pod marked as not ready, pod name: rec-test-0
2022-09-29 13:14:19,500 - INFO - Collected rs config from pod marked as not ready, pod name: rec-test-0
2022-09-29 13:14:19,661 - INFO - Namespace 'redis': Collected rs logs from pod marked as not ready, pod name: rec-test-1
2022-09-29 13:14:19,806 - INFO - Collected rs config from pod marked as not ready, pod name: rec-test-1
2022-09-29 13:14:20,519 - INFO - Namespace 'redis': Collected rs logs from pod marked as not ready, pod name: rec-test-2
2022-09-29 13:14:20,666 - INFO - Collected rs config from pod marked as not ready, pod name: rec-test-2
2022-09-29 13:14:20,793 - INFO - Namespace 'redis': Collected resources list
2022-09-29 13:14:20,866 - INFO - Namespace 'redis': Collected events
2022-09-29 13:14:20,944 - INFO - Namespace 'redis': Collecting API resources
2022-09-29 13:14:21,020 - INFO - Namespace 'redis':   + Collected RedisEnterpriseCluster
2022-09-29 13:14:21,096 - INFO - Namespace 'redis':   + Collected RedisEnterpriseDatabase
2022-09-29 13:14:21,197 - WARNING - Namespace 'redis': Failed to get RedisEnterpriseRemoteCluster resource: error: the server doesn't have a resource type "RedisEnterpriseRemoteCluster".
2022-09-29 13:14:21,488 - WARNING - Namespace 'redis': Failed to get RedisEnterpriseActiveActivePeering resource: error: the server doesn't have a resource type "RedisEnterpriseActiveActivePeering".
2022-09-29 13:14:21,749 - INFO - Namespace 'redis':   + Collected StatefulSet
2022-09-29 13:14:21,828 - INFO - Namespace 'redis':   + Collected Deployment
2022-09-29 13:14:21,908 - INFO - Namespace 'redis':   + Collected Service
2022-09-29 13:14:21,987 - INFO - Namespace 'redis':   + Collected ConfigMap
2022-09-29 13:14:22,078 - WARNING - Namespace 'redis': Failed to get Route resource: error: the server doesn't have a resource type "Route".
2022-09-29 13:14:22,345 - INFO - Namespace 'redis':   + Collected Ingress
2022-09-29 13:14:22,424 - INFO - Namespace 'redis':   + Collected Role
2022-09-29 13:14:22,500 - INFO - Namespace 'redis':   + Collected RoleBinding
2022-09-29 13:14:22,577 - INFO - Namespace 'redis':   + Collected PersistentVolume
2022-09-29 13:14:22,650 - INFO - Namespace 'redis':   + Collected PersistentVolumeClaim
2022-09-29 13:14:22,741 - INFO - Namespace 'redis':   + Collected Node
2022-09-29 13:14:22,813 - INFO - Namespace 'redis':   + Collected PodDisruptionBudget
2022-09-29 13:14:22,886 - INFO - Namespace 'redis':   + Collected ResourceQuota
2022-09-29 13:14:22,963 - INFO - Namespace 'redis':   + Collected Endpoints
2022-09-29 13:14:23,060 - INFO - Namespace 'redis':   + Collected Pod
2022-09-29 13:14:23,134 - INFO - Namespace 'redis':   + Collected NetworkPolicy
2022-09-29 13:14:23,615 - INFO - Namespace 'redis':   + Collected CustomResourceDefinition
2022-09-29 13:14:23,684 - INFO - Namespace 'redis':   + Collected CertificateSigningRequest
2022-09-29 13:14:23,758 - INFO - Namespace 'redis':   + Collected ValidatingWebhookConfiguration
2022-09-29 13:14:23,845 - WARNING - Namespace 'redis': Failed to get NamespacedValidatingType resource: error: the server doesn't have a resource type "NamespacedValidatingType".
2022-09-29 13:14:24,130 - WARNING - Namespace 'redis': Failed to get NamespacedValidatingRule resource: error: the server doesn't have a resource type "NamespacedValidatingRule".
2022-09-29 13:14:24,442 - INFO - Namespace 'redis':   + Collected ClusterRole
2022-09-29 13:14:24,540 - INFO - Namespace 'redis':   + Collected ClusterRoleBinding
2022-09-29 13:14:24,627 - WARNING - Namespace 'redis': Failed to get ClusterServiceVersion resource: error: the server doesn't have a resource type "ClusterServiceVersion".
2022-09-29 13:14:24,894 - WARNING - Namespace 'redis': Failed to get Subscription resource: error: the server doesn't have a resource type "Subscription".
2022-09-29 13:14:25,168 - WARNING - Namespace 'redis': Failed to get InstallPlan resource: error: the server doesn't have a resource type "InstallPlan".
2022-09-29 13:14:25,451 - WARNING - Namespace 'redis': Failed to get CatalogSource resource: error: the server doesn't have a resource type "CatalogSource".
2022-09-29 13:14:25,721 - WARNING - Namespace 'redis': Failed to get PodSecurityPolicy resource: error: the server doesn't have a resource type "PodSecurityPolicy".
2022-09-29 13:14:25,981 - INFO - Namespace 'redis':   + Collected ReplicaSet
2022-09-29 13:14:26,052 - INFO - Namespace 'redis':   + Collected StorageClass
2022-09-29 13:14:26,055 - INFO - Namespace 'redis': Collecting API resources description
2022-09-29 13:14:26,136 - INFO - Namespace: 'redis' + Collected RedisEnterpriseCluster
2022-09-29 13:14:26,206 - INFO - Namespace: 'redis' + Collected RedisEnterpriseDatabase
2022-09-29 13:14:26,293 - WARNING - Namespace 'redis': Failed to describe RedisEnterpriseRemoteCluster resource: error: the server doesn't have a resource type "RedisEnterpriseRemoteCluster".
2022-09-29 13:14:26,566 - WARNING - Namespace 'redis': Failed to describe RedisEnterpriseActiveActivePeering resource: error: the server doesn't have a resource type "RedisEnterpriseActiveActivePeering".
2022-09-29 13:14:26,844 - INFO - Namespace: 'redis' + Collected StatefulSet
2022-09-29 13:14:26,941 - INFO - Namespace: 'redis' + Collected Deployment
2022-09-29 13:14:27,051 - INFO - Namespace: 'redis' + Collected Service
2022-09-29 13:14:27,157 - INFO - Namespace: 'redis' + Collected ConfigMap
2022-09-29 13:14:27,247 - WARNING - Namespace 'redis': Failed to describe Route resource: error: the server doesn't have a resource type "Route".
2022-09-29 13:14:27,527 - INFO - Namespace: 'redis' + Collected Ingress
2022-09-29 13:14:27,607 - INFO - Namespace: 'redis' + Collected Role
2022-09-29 13:14:27,685 - INFO - Namespace: 'redis' + Collected RoleBinding
2022-09-29 13:14:27,787 - INFO - Namespace: 'redis' + Collected PersistentVolume
2022-09-29 13:14:27,897 - INFO - Namespace: 'redis' + Collected PersistentVolumeClaim
2022-09-29 13:14:28,081 - INFO - Namespace: 'redis' + Collected Node
2022-09-29 13:14:28,163 - INFO - Namespace: 'redis' + Collected PodDisruptionBudget
2022-09-29 13:14:28,234 - INFO - Namespace: 'redis' + Collected ResourceQuota
2022-09-29 13:14:28,332 - INFO - Namespace: 'redis' + Collected Endpoints
2022-09-29 13:14:28,447 - INFO - Namespace: 'redis' + Collected Pod
2022-09-29 13:14:28,519 - INFO - Namespace: 'redis' + Collected NetworkPolicy
2022-09-29 13:14:29,085 - INFO - Namespace: 'redis' + Collected CustomResourceDefinition
2022-09-29 13:14:29,156 - INFO - Namespace: 'redis' + Collected CertificateSigningRequest
2022-09-29 13:14:29,236 - INFO - Namespace: 'redis' + Collected ValidatingWebhookConfiguration
2022-09-29 13:14:29,325 - WARNING - Namespace 'redis': Failed to describe NamespacedValidatingType resource: error: the server doesn't have a resource type "NamespacedValidatingType".
2022-09-29 13:14:29,598 - WARNING - Namespace 'redis': Failed to describe NamespacedValidatingRule resource: error: the server doesn't have a resource type "NamespacedValidatingRule".
2022-09-29 13:14:30,092 - INFO - Namespace: 'redis' + Collected ClusterRole
2022-09-29 13:14:30,321 - INFO - Namespace: 'redis' + Collected ClusterRoleBinding
2022-09-29 13:14:30,413 - WARNING - Namespace 'redis': Failed to describe ClusterServiceVersion resource: error: the server doesn't have a resource type "ClusterServiceVersion".
2022-09-29 13:14:30,698 - WARNING - Namespace 'redis': Failed to describe Subscription resource: error: the server doesn't have a resource type "Subscription".
2022-09-29 13:14:30,986 - WARNING - Namespace 'redis': Failed to describe InstallPlan resource: error: the server doesn't have a resource type "InstallPlan".
2022-09-29 13:14:31,271 - WARNING - Namespace 'redis': Failed to describe CatalogSource resource: error: the server doesn't have a resource type "CatalogSource".
2022-09-29 13:14:31,552 - WARNING - Namespace 'redis': Failed to describe PodSecurityPolicy resource: error: the server doesn't have a resource type "PodSecurityPolicy".
2022-09-29 13:14:31,832 - INFO - Namespace: 'redis' + Collected ReplicaSet
2022-09-29 13:14:31,907 - INFO - Namespace: 'redis' + Collected StorageClass
2022-09-29 13:14:31,910 - INFO - Namespace 'redis': Collecting pods' logs:
2022-09-29 13:14:46,625 - INFO - Archived files into /home/centos/yamls/redis-en/redis_enterprise_k8s_debug_info_20220929-131328.tar.gz
2022-09-29 13:14:46,702 - INFO - Finished Redis Enterprise log collector
2022-09-29 13:14:46,702 - INFO - --- Run time: 1 minutes ---

We are using the NGINX controller and resources. As kubernetes cluster is self-managed, So we have additionally configure the Nginx Load Balancer and with the help of nodePort IP We are trying it.

I am not able to run the CRDB command just yet because I believe the initial setup is still broken.

[laurentdroin]

Hi,

The log_collector script console output is not what we will need. It does generate a tarball that you will need to attach to the Support Ticket when you open it. Also, a CRDB database involves at least two Redis Enterprise clusters. I don't see any indication that you have created a 2nd Redis Enterprise cluster on a separate k8s cluster.

Kindly open a Support Ticket as we can't use github to help you efficiently.

Laurent.

[Deepanshu6dec]

Thanks

[Deepanshu6dec]

As we are testing redis Enterprise. Can you please help me supply this values

Database name <db-name>:
•   Description: Combined with ingress suffix to create the Active-Active database hostname
REC hostname <rec-hostname>:
•   Description: Hostname used to identify your Redis Enterprise cluster in the crdb-cli command. This MUST be different from other participating clusters.
API hostname <api-hostname>:
•   Description: Hostname used to access the Redis Enterprise cluster API from outside the K8s cluster
Ingress suffix <ingress-suffix>:
•   Description: Combined with database name to create the Active-Active database hostname

In Redis Enterprise Cluster spec 
activeActive:
  apiIngressUrl: <api-hostname>
  dbIngressSuffix: <ingress-suffix>

Here is Our REC Spec YAML

apiVersion: "app.redislabs.com/v1"
kind: "RedisEnterpriseCluster"
metadata:
  name: "test-rec"
spec:
  nodes: 3
  uiServiceType: ClusterIP
  username: 
  persistentSpec:
   enabled: true
   storageClassName: "local-storage"
   volumeSize: "5Gi" 
  redisEnterpriseNodeResources:
    limits:
      cpu: 900m
      memory: 3Gi
    requests:
      cpu: 500m
      memory: 1Gi
  activeActive:
    apiIngressUrl: ???????
    dbIngressSuffix:  ?????
    ingressAnnotations:
      kubernetes.io/ingress.class: "nginx"
      nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
      nginx.ingress.kubernetes.io/ssl-passthrough: "true"  
    method: ingress

Ingress YAML

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: rec-ingress
  namespace: redis
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
spec:
  rules:
  - host: rec.k8s.my.redisdemo.com   
    http:
      paths:
      - path: /
        pathType: ImplementationSpecific
        backend:
          service:
            name: db-test
            port:
              name: redis

I will be Helpfull for setting up redis enterprise with ingress and then same we implement in our Cluster