frankfong0208
commented
3 years ago Not sure why the below section I submitted cannot be seen
If this is a BUG REPORT, please:
- Fill in as much of the template below as you can. If you leave out information, we can't help you as well. I attempted to setup Jfrog Platform charts with our Enterprise + license (Artifactory HA, Xray and Mission Control with external AWS RDS postgres database 12.5 and S3 V3 filestore). The rds postgres is setup in advance and test connection is fine. I generated a custom master key by following this https://github.com/jfrog/charts/blob/master/stable/artifactory-ha/README.md#create-a-unique-master-key-1 , then create a k8s secret called artifactory-master-key to store the 32 hex value and add artifactory.masterKeySecretName: artifactory-master-key in the main custom values.yaml file. When I installed using "helm upgrade --install jfrog-platform --namespace jfrog-platform jfrog/jfrog-platform --version 0.6.0 -f jfrog-platform/customvalues.yaml", the main container artifactory-ha for the jfrog-platform-artifactory-ha-primary-0 pod would fail the startup probe (Startup probe failed: HTTP probe failed with statuscode: 503). All init containers ran successfully though. If I run ' kubectl exec jfrog-platform-artifactory-ha-primary-0 -- cat /var/opt/jfrog/artifactory/etc/security/master.key', I could get the correct custom master key value I stored in the secret as d0d7dc09a7091f68b59f51fa2e7b0c336ab7f38d8d23fd18aed38bb6568164a7. When I ran this "kubectl exec jfrog-platform-artifactory-ha-primary-0 -- curl localhost:8082/router/api/v1/system/health", I got this: { "router": { "node_id": "jfrog-platform-artifactory-ha-primary-0", "state": "HEALTHY", "message": "OK" }, "services": [ { "service_id": "jfac@01fakhypawqdmn1p9yz7em19nn", "node_id": "jfrog-platform-artifactory-ha-primary-0", "state": "UNHEALTHY_PEER", "message": "Service is healthy; there is at least one unhealthy service; there are missing services: jfrt" }, { "service_id": "jfevt@01fakhypawqdmn1p9yz7em19nn", "node_id": "jfrog-platform-artifactory-ha-primary-0", "state": "UNHEALTHY_PEER", "message": "Service is healthy; there is at least one unhealthy service; there are missing services: jfrt" }, { "service_id": "jffe@000", "node_id": "jfrog-platform-artifactory-ha-primary-0", "state": "UNHEALTHY", "message": "Get \"http://localhost:8070/readiness\": dial tcp 127.0.0.1:8070: connect: connection refused" }, { "service_id": "jfmd@01fakhyvag55n029kzsyn7gvh7", "node_id": "jfrog-platform-artifactory-ha-primary-0", "state": "UNHEALTHY_PEER", "message": "Service is healthy; there is at least one unhealthy service; there are missing services: jfrt" } ] The log of the primary artifactory pod had many errors as attached. ith 404 code 2021-07-14T22:45:12.641Z [jfrou] [WARN ] [273fe4b6ee30ff64] [local_topology.go:256 ] [main ] - Readiness test failed with the following error: "required node services are missing or unhealthy" 2021-07-14T22:45:12.805Z [jffe ] [ERROR] [ ] [ ] [main ] - ArtifactoryClient::http [get] request to /api/system/ping failed with 404 code 2021-07-14T22:45:13.808Z [jffe ] [ERROR] [ ] [ ] [main ] - ArtifactoryClient::http [get] request to /api/system/ping failed with 404 code 2021-07-14T22:45:14.812Z [jffe ] [ERROR] [ ] [ ] [main ] - ArtifactoryClient::http [get] request to /api/system/ping failed with 404 code 2021-07-14T22:45:15.100Z [jfevt] [WARN ] [6791d047755e014d] [base_url_cache.go:66 ] [main ] - Base URL cache is not updated because of error: failed to fetch base URL from Artifactory (returned 404) [baseurl_cache] 2021-07-14T22:45:15.100Z [jfevt] [WARN ] [6791d047755e014d] [proxy_synchronizer.go:21 ] [main ] - Proxy cache is not updated because of error: failure to get the list of proxies from Artifactory (returned 404) [proxy_cache] 2021-07-14T22:45:15.103Z [jfevt] [WARN ] [6791d047755e014d] [domain_client.go:57 ] [main ] - Error occurred when refreshing domain cache all domain endpoint failed : Fetch domains from http://localhost:8046/artifactory/api/events/domains failed (returned 404), Fetch domains from http://localhost:8046/distribution/api/v1/events/domains failed (returned 404), [domain_client]
If this is a FEATURE REQUEST, please:
- Describe in detail the feature/behavior/change you'd like to see.
In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. -->
chukka
rahulsadanandan
Is this a request for help?: yes
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT
Version of Helm and Kubernetes: Helm v3.5.3 Kubernetes versions: Client Version: v1.19.4 Server Version: v1.18.9-eks-d1db3c Which chart: https://artifacthub.io/packages/helm/jfrog/jfrog-platform/0.7.0 I tried 0.6.0 first and then 0.7.0, same result. What happened: As described above, the jfrog-platform-artifactory-ha-primary-0 pod would not start the main artifactory-ha container due to failed startup probe. I googled for the errors and someone mentioned this could probably due to corrupted master key, which I don't quite follow as the key was generated and configured per the official doco.
What you expected to happen: The jfrog-platform-artifactory-ha-primary-0 pod should start healthy and the jfrog-platform-artifactory-ha-member-0 and jfrog-platform-artifactory-ha-member-1 pods should start as well afterwards rather than get stuck waiting for the primary pod.
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know: I also tried putting artifactory-ha.global.masterKeySecretName=artifactory-master-key, and even not using a secret, but just hard-code the master key value in this artifactory-ha.global.masterKey= or artifactory-ha.artifactory.masterKey= in the customvalues.yaml file. to no avail.
I also tried the global.masterKey or global.masterKeySecretName paths, same result.
The moment I commented out these values, the chart would use the default dummy master key which is bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb and everything worked just fine.
Every time I redo the chart installation, I would delete the artifactory_ha database from inside the rds postgres instance and even tried recreating the rds itself to ensure rds was not contaminated. however, the result was still the same.
Part of customvalues.yaml file: