thingsboard / thingsboard-pe-k8s

ThingsBoard Professional Edition Kubernetes scripts and docs
Apache License 2.0
21 stars 35 forks source link

Unable to deploy tb-integration-executor with redis sentinel #88

Open TheSamabo opened 2 months ago

TheSamabo commented 2 months ago

Hello, i'm facing an issue with deployment of tb-integration-executor and redis sentinel. I was able to deploy tb-node with sentinel configured (even tho some important information is missing in the TB Core component docs) with the following StatefulSet env vars:

        - name: CACHE_TYPE
          value: "redis"
        - name: REDIS_CONNECTION_TYPE
          value: "sentinel"
        - name: REDIS_MASTER
          value: "sts-redis-master"
        - name: REDIS_SENTINELS
          value: "sts-redis-node-0.sts-redis-headless.redis.svc.cluster.local:26379,sts-redis-node-1.sts-redis-headless.redis.svc.cluster.local:26379,sts-redis-node-2.sts-redis-headless.redis.svc.cluster.local:26379"
        - name: REDIS_SENTINEL_PASSWORD
          value: <Password>
        - name: REDIS_PASSWORD
          value: <Password>

So after seeing that the tb-integration-executor also specifies redis as a parameters in the minikube deployment yaml files. I thought i could just copy this configuration and it will work. But to my surprise it didn't. When is set my thingsboard.yaml for the integration-executor env vars like this:

            - name: CACHE_TYPE
              value: "redis"
            - name: REDIS_CONNECTION_TYPE
              value: "sentinel"
            - name: REDIS_MASTER
              value: "sts-redis-master"
            - name: REDIS_SENTINELS
              value: "sts-redis-node-0.sts-redis-headless.redis.svc.cluster.local:26379,sts-redis-node-1.sts-redis-headless.redis.svc.cluster.local:26379,sts-redis-node-2.sts-redis-headless.redis.svc.cluster.local:26379"
            - name: REDIS_SENTINEL_PASSWORD
              value: <Password>
            - name: REDIS_PASSWORD
              value: <Password>

the pod logs look like this :

thingsboard-tb-integration-executor-0-1725375316634848100.log

Caused by: java.lang.IllegalArgumentException: Sentinel Master Id must not be null or empty
    at org.springframework.util.Assert.hasText(Assert.java:289)
    at org.springframework.data.redis.connection.SentinelMasterId.<init>(SentinelMasterId.java:34)
    at org.springframework.data.redis.connection.RedisConfiguration$SentinelConfiguration.setMaster(RedisConfiguration.java:357)
    at org.thingsboard.server.cache.TBRedisSentinelConfiguration.loadFactory(TBRedisSentinelConfiguration.java:65)
    at org.thingsboard.server.cache.TBRedisCacheConfiguration.redisConnectionFactory(TBRedisCacheConfiguration.java:105)
    at org.thingsboard.server.cache.TBRedisSentinelConfiguration$$EnhancerBySpringCGLIB$$59e3a0.CGLIB$redisConnectionFactory$31(<generated>)
    at org.thingsboard.server.cache.TBRedisSentinelConfiguration$$EnhancerBySpringCGLIB$$59e3a0$$FastClassBySpringCGLIB$$4055d0d5.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
    at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331)
    at org.thingsboard.server.cache.TBRedisSentinelConfiguration$$EnhancerBySpringCGLIB$$59e3a0.redisConnectionFactory(<generated>)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
    ... 56 common frames omitted

So after looking in some thingsboard code i found that the key to provide redis sentinel master name is: redis.sentinel.master So i thought of env variable would look like this: REDIS_SENTINEL_MASTER, when i tried that in the env vars for the integration-executor i got the following logs:

thingsboard-tb-integration-executor-0-1725375710414817500.log

Caused by: redis.clients.jedis.exceptions.JedisConnectionException: All sentinels down, cannot determine where is sts-redis-master master is running...
    at redis.clients.jedis.JedisSentinelPool.initSentinels(JedisSentinelPool.java:293)
    at redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:217)
    at redis.clients.jedis.JedisSentinelPool.<init>(JedisSentinelPool.java:205)
    at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createRedisSentinelPool(JedisConnectionFactory.java:415)
    at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createPool(JedisConnectionFactory.java:398)
    at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.afterPropertiesSet(JedisConnectionFactory.java:348)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1863)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1800)
    ... 52 common frames omitted

Please bear in mind that this redis sentinel cluster is working fine for the tb-node. Any help is appreciated, Thank you Sam

TheSamabo commented 2 months ago

EDIT: I was able to finally deploy the integration executor with redis sentinel but i had to change the env variable keys: REDIS_MASTER -> REDIS_SENTINEL_MASTER REDIS_SENTINELS -> REDIS_SENTINEL_SENTINELS

I forgot to mention currently we are running TB 3.6.2 as a k8 deployment.

TheSamabo commented 2 months ago

The documentation of the integration executor should be updated to reflect this