search

jfrog / charts

JFrog official Helm Charts
https://jfrog.com/integration/helm-repository/
Apache License 2.0
254 stars 441 forks source link

[jfrog/distribution] distributor container fails if redis password not explicitly set #829

Closed dhdersch closed 4 years ago

dhdersch commented 4 years ago

Is this a request for help?: No

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Version of Helm and Kubernetes:

Helm Version: version.BuildInfo{Version:"v3.1.2", GitCommit:"d878d4d45863e42fd5cff6743294a11d28a9abce", GitTreeState:"clean", GoVersion:"go1.13.8"}

Kubernetes Version: 

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.9", GitCommit:"2e808b7cb054ee242b68e62455323aa783991f03", GitTreeState:"clean", BuildDate:"2020-01-18T23:33:14Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.11-gke.5", GitCommit:"a5bf731ea129336a3cf32c3375317b3a626919d7", GitTreeState:"clean", BuildDate:"2020-03-31T02:49:49Z", GoVersion:"go1.12.17b4", Compiler:"gc", Platform:"linux/amd64"}

Which chart: jfrog/distribution

What happened: When I deploy jfrog/distribution, it fails unless I explicitly provide redis.password. Specifically, the distributor container fails. I get the following error message when running kubectl logs <pod-name> -c distributor:

2020-04-17T16:33:23.453Z [jfdr ] [WARN ] [ff05c904682a3447] [tributorApplicationContext:557] [main                ] - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'foremanDistributorService': Invocation of init method failed; nested exception is redis.clients.jedis.exceptions.JedisDataException: ERR invalid password
2020-04-17T16:33:23.456Z [jfdr ] [INFO ] [ff05c904682a3447] [.c.ThreadPoolTaskScheduler:208] [main                ] - Shutting down ExecutorService 'taskScheduler'
2020-04-17T16:33:23.456Z [jfdr ] [INFO ] [ff05c904682a3447] [.c.ThreadPoolTaskScheduler:208] [main                ] - Shutting down ExecutorService 'taskScheduler'
Exception in thread "main" org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'foremanDistributorService': Invocation of init method failed; nested exception is redis.clients.jedis.exceptions.JedisDataException: ERR invalid password
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:139)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:414)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1770)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:843)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:877)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549)
        at com.jfrog.bintray.distributor.context.DistributorApplicationContext.create(DistributorApplicationContext.java:38)
        at com.jfrog.bintray.distributor.DistributorRunner.<init>(DistributorRunner.java:19)
        at com.jfrog.bintray.distributor.DistributorRunner.main(DistributorRunner.java:25)
Caused by: redis.clients.jedis.exceptions.JedisDataException: ERR invalid password
        at redis.clients.jedis.Protocol.processError(Protocol.java:117)
        at redis.clients.jedis.Protocol.process(Protocol.java:142)
        at redis.clients.jedis.Protocol.read(Protocol.java:196)
        at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:288)
        at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:187)
        at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:86)
        at redis.clients.jedis.Connection.sendCommand(Connection.java:93)
        at redis.clients.jedis.Connection.sendCommand(Connection.java:88)
        at redis.clients.jedis.BinaryClient.auth(BinaryClient.java:539)
        at redis.clients.jedis.BinaryJedis.initializeClientFromURI(BinaryJedis.java:94)
        at redis.clients.jedis.BinaryJedis.<init>(BinaryJedis.java:79)
        at redis.clients.jedis.Jedis.<init>(Jedis.java:51)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
        at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:80)
        at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:105)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:59)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:237)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:265)
        at com.jfrog.bintray.foreman.jedis.common.SingleJedisConnectionHolder.createJedisInstance(SingleJedisConnectionHolder.groovy:107)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:190)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:58)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:156)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:184)
        at com.jfrog.bintray.foreman.jedis.common.SingleJedisConnectionHolder.doConnect(SingleJedisConnectionHolder.groovy:44)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:190)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:58)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:156)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:160)
        at com.jfrog.bintray.foreman.jedis.common.SingleJedisConnectionHolder.connect(SingleJedisConnectionHolder.groovy:33)
        at com.jfrog.bintray.foreman.jedis.common.JedisConnectionHolder$connect.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:119)
        at com.jfrog.bintray.foreman.jedis.internal.JedisConnectionFactory.connect(JedisConnectionFactory.groovy:20)
        at com.jfrog.bintray.foreman.jedis.internal.JedisConnectionFactory$connect.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
        at com.jfrog.bintray.foreman.jedis.internal.TaskProcessDispatcher.<init>(TaskProcessDispatcher.groovy:43)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
        at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:80)
        at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:105)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:59)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:237)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:257)
        at com.jfrog.bintray.foreman.jedis.JedisForeman.<init>(JedisForeman.groovy:42)
        at com.jfrog.bintray.distributor.ForemanDistributorService.init(ForemanDistributorService.java:36)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:363)
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:307)
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:136)
        ... 14 more

What you expected to happen: I expected it to work

How to reproduce it (as minimally and precisely as possible): Run helm like the following:

helm upgrade --install distribution -f distribution.yaml \
--set distribution.joinKey="${JOIN_KEY}" \
--set distribution.jfrogUrl="${JFROG_URL}" \
--set distribution.masterKey=${DISTRIBUTION_MASTER_KEY} \
jfrog/distribution

with the following distribution.yaml:

replicaCount: 4

redis:
  resources:
    requests:
      memory: "512Mi"
      cpu: "1"
    limits:
      memory: "2Gi"
      cpu: "2"

distribution:
  resources:
    requests:
      memory: "2Gi"
      cpu: "1"
    limits:
      memory: "6Gi"
      cpu: "4"
  javaOpts:
    xms: "2g"
    xmx: "6g"

distributor:
  resources:
    requests:
      memory: "2Gi"
      cpu: "1"
    limits:
      memory: "6Gi"
      cpu: "4"
  javaOpts:
    xms: "2g"
    xmx: "6g"

Anything else we need to know: Additionally, even after setting the redis.password explicitly, there still seems to be an issue. I get the following error:

2020-04-17T16:36:10.685Z [jfdr ] [INFO ] [4ed923c9e985838d] [DistributionApiTokenService:58] [main                ] - Searching for Distribution api token at /opt/jfrog/distribution/var/etc/distributor/security/token
2020-04-17T16:36:10.686Z [jfdr ] [INFO ] [4ed923c9e985838d] [DistributionApiTokenService:61] [main                ] - Distribution api token file not found.
2020-04-17T16:36:10.686Z [jfdr ] [ERROR] [4ed923c9e985838d] [DistributionApiTokenService:48] [main                ] - Distribution api token is blank.
2020-04-17T16:36:10.687Z [jfdr ] [WARN ] [4ed923c9e985838d] [c.j.b.d.DistributorRunner:44  ] [main                ] - Distribution must be up and bound to an authentication provider before additional tasks can be performed.

This causes the readiness and liveness probes to fail which causes the distribution container to restart.

chukka commented 4 years ago

@dhdersch Thanks for the bug report .This is a known issue - We are working on a fix and will be available in the next chart version hopefully next week

chukka commented 4 years ago

@dhdersch distribution-5.1.2 chart has fix for above issue , Please try and share feedback

dhdersch commented 4 years ago

@chukka That has fixed the issue with postgresql, but I'm still experiencing issues with the liveness and readiness probe failing. I'll create a separate ticket for that.