search

jx3-gitops-repositories / jx3-terraform-eks

Jenkins X 3.x Infrastructure Git Template for Terraform and EKS for managing cloud resources
Apache License 2.0
9 stars 41 forks source link

Secrets not auto populating #16

Open akashsuresh opened 3 years ago

akashsuresh commented 3 years ago

I installed JX on EKS but secrets don't seem to have auto populated. Any reason why this would have happened?

Screenshot 2021-08-13 at 12 17 49 AM
dennislabajo commented 3 years ago

Hello - I have the same issue following the steps in the readme.

image

From the jx admin log it seems this may be the culprit:

Error: failed to find hmac token from secret: could not find lighthouse hmac token lighthouse-hmac-token in namespace jx: secrets "lighthouse-hmac-token" not found

Any ideas how to resolve the issue?

ankitm123 commented 3 years ago

Please update to the latest version of the eks-jx module, it's fixed there: https://github.com/jenkins-x/terraform-aws-eks-jx/releases

mrmarcsmith commented 3 years ago

I think this may be related to issue #20 , I'll repost my setup here incase it helps. I just experienced the same issue on EKS 1.15.46. here is my setup main.tf

module "eks-jx" {
  source               = "jenkins-x/eks-jx/aws"
  version              = "1.15.46"
...
}

variables.tf

variable "cluster_version" {
  description = "Kubernetes version to use for the EKS cluster."
  type        = string
  default     = "1.21"
}
variable "nginx_chart_version" {
  type        = string
  description = "nginx chart version"
  default     = "3.12.0"
}
Screen Shot 2021-09-16 at 9 33 34 PM
mrmarcsmith commented 3 years ago

I just retried with cluster_version 1.20 instead of 1.21 and I'm still running into this issue. @ankitm123 what information do you need from me to be able to debug this?

akashsuresh commented 3 years ago

Still the same issue . Also GKE version faces the same problem . Is this a terraform bug by any chance ?

testorg-mani commented 3 years ago

I do see the same problem. Downgrading versions is not helping

ankitm123 commented 3 years ago

What is the output of the external-secrets pod in the secret-infra namespace and vault pod in the jx-vault namespace?

reinbach commented 3 years ago

I appear to be having the same issue, and I'm seeing the following on the external-secrets pod in the secret-infra namespace;


{"level":30,"message_time":"2021-10-29T15:18:50.004Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"status update failed for externalsecret jx/lighthouse-oauth-token, due to modification, new poller should start"}
{"level":30,"message_time":"2021-10-29T15:19:49.667Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"running poll on the secret jx/jenkins-maven-settings"}
{"level":30,"message_time":"2021-10-29T15:19:49.668Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"running poll on the secret jx/jenkins-x-chartmuseum"}
{"level":30,"message_time":"2021-10-29T15:19:49.669Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"running poll on the secret jx-production/tekton-container-registry-auth"}
{"level":30,"message_time":"2021-10-29T15:19:49.670Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"running poll on the secret jx/jx-basic-auth-user-password"}
{"level":30,"message_time":"2021-10-29T15:19:49.671Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"running poll on the secret jx-production/tekton-container-registry-auth"}
{"level":30,"message_time":"2021-10-29T15:19:49.672Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"running poll on the secret jx/lighthouse-hmac-token"}
{"level":30,"message_time":"2021-10-29T15:19:49.673Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"running poll on the secret jx/lighthouse-hmac-token"}
{"level":30,"message_time":"2021-10-29T15:19:49.676Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"running poll on the secret jx-staging/tekton-container-registry-auth"}
{"level":30,"message_time":"2021-10-29T15:19:49.677Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"running poll on the secret jx/nexus"}
{"level":30,"message_time":"2021-10-29T15:19:49.678Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"running poll on the secret jx/tekton-container-registry-auth"}
{"level":30,"message_time":"2021-10-29T15:19:49.679Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"running poll on the secret jx/tekton-git"}
{"level":30,"message_time":"2021-10-29T15:19:49.680Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"running poll on the secret jx/lighthouse-oauth-token"}
{"level":30,"message_time":"2021-10-29T15:19:49.682Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"running poll on the secret jx/jx-basic-auth-htpasswd"}
{"level":50,"message_time":"2021-10-29T15:19:49.712Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","payload":{"response":{"statusCode":500,"body":{"errors":["claim \"iss\" is invalid"]}}},"msg":"failure while polling the secret jx/lighthouse-hmac-token"}
{"level":50,"message_time":"2021-10-29T15:19:49.714Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","payload":{"response":{"statusCode":500,"body":{"errors":["claim \"iss\" is invalid"]}}},"msg":"failure while polling the secret jx-production/tekton-container-registry-auth"}
{"level":50,"message_time":"2021-10-29T15:19:49.715Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","payload":{"response":{"statusCode":500,"body":{"errors":["claim \"iss\" is invalid"]}}},"msg":"failure while polling the secret jx/lighthouse-oauth-token"}
{"level":50,"message_time":"2021-10-29T15:19:49.723Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","payload":{"response":{"statusCode":500,"body":{"errors":["claim \"iss\" is invalid"]}}},"msg":"failure while polling the secret jx/jx-basic-auth-htpasswd"}
{"level":50,"message_time":"2021-10-29T15:19:49.731Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","payload":{"response":{"statusCode":500,"body":{"errors":["claim \"iss\" is invalid"]}}},"msg":"failure while polling the secret jx-staging/tekton-container-registry-auth"}
{"level":30,"message_time":"2021-10-29T15:19:49.746Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"stopping poller for jx/lighthouse-hmac-token"}
{"level":30,"message_time":"2021-10-29T15:19:49.747Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"starting poller for jx/lighthouse-hmac-token"}
{"level":50,"message_time":"2021-10-29T15:19:49.757Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","payload":{"response":{"statusCode":500,"body":{"errors":["claim \"iss\" is invalid"]}}},"msg":"failure while polling the secret jx/tekton-container-registry-auth"}
{"level":50,"message_time":"2021-10-29T15:19:49.758Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","payload":{"response":{"statusCode":500,"body":{"errors":["claim \"iss\" is invalid"]}}},"msg":"failure while polling the secret jx/lighthouse-hmac-token"}
{"level":50,"message_time":"2021-10-29T15:19:49.760Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","payload":{"response":{"statusCode":500,"body":{"errors":["claim \"iss\" is invalid"]}}},"msg":"failure while polling the secret jx/jenkins-maven-settings"}
{"level":50,"message_time":"2021-10-29T15:19:49.766Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","payload":{"response":{"statusCode":500,"body":{"errors":["claim \"iss\" is invalid"]}}},"msg":"failure while polling the secret jx/jx-basic-auth-user-password"}
{"level":30,"message_time":"2021-10-29T15:19:49.769Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"stopping poller for jx/jx-basic-auth-htpasswd"}
{"level":30,"message_time":"2021-10-29T15:19:49.769Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"starting poller for jx/jx-basic-auth-htpasswd"}
{"level":30,"message_time":"2021-10-29T15:19:49.770Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"stopping poller for jx-production/tekton-container-registry-auth"}
{"level":30,"message_time":"2021-10-29T15:19:49.770Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"starting poller for jx-production/tekton-container-registry-auth"}
{"level":30,"message_time":"2021-10-29T15:19:49.771Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"stopping poller for jx/lighthouse-oauth-token"}
{"level":30,"message_time":"2021-10-29T15:19:49.771Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"starting poller for jx/lighthouse-oauth-token"}
{"level":50,"message_time":"2021-10-29T15:19:49.779Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","payload":{"response":{"statusCode":500,"body":{"errors":["claim \"iss\" is invalid"]}}},"msg":"failure while polling the secret jx-production/tekton-container-registry-auth"}
{"level":50,"message_time":"2021-10-29T15:19:49.781Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","payload":{"response":{"statusCode":500,"body":{"errors":["claim \"iss\" is invalid"]}}},"msg":"failure while polling the secret jx/nexus"}
{"level":50,"message_time":"2021-10-29T15:19:49.782Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","payload":{"response":{"statusCode":500,"body":{"errors":["claim \"iss\" is invalid"]}}},"msg":"failure while polling the secret jx/tekton-git"}
{"level":30,"message_time":"2021-10-29T15:19:49.783Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"stopping poller for jx-staging/tekton-container-registry-auth"}
{"level":30,"message_time":"2021-10-29T15:19:49.784Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"starting poller for jx-staging/tekton-container-registry-auth"}
{"level":50,"message_time":"2021-10-29T15:19:49.828Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","payload":{"response":{"statusCode":500,"body":{"errors":["claim \"iss\" is invalid"]}}},"msg":"failure while polling the secret jx/jenkins-x-chartmuseum"}
{"level":30,"message_time":"2021-10-29T15:19:49.831Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"status update failed for externalsecret jx/lighthouse-hmac-token, due to modification, new poller should start"}
{"level":30,"message_time":"2021-10-29T15:19:49.838Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"stopping poller for jx/tekton-container-registry-auth"}
{"level":30,"message_time":"2021-10-29T15:19:49.839Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"starting poller for jx/tekton-container-registry-auth"}
{"level":30,"message_time":"2021-10-29T15:19:49.849Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"stopping poller for jx/jx-basic-auth-user-password"}
{"level":30,"message_time":"2021-10-29T15:19:49.850Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"starting poller for jx/jx-basic-auth-user-password"}
{"level":30,"message_time":"2021-10-29T15:19:49.854Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"stopping poller for jx/nexus"}
{"level":30,"message_time":"2021-10-29T15:19:49.855Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"starting poller for jx/nexus"}
{"level":30,"message_time":"2021-10-29T15:19:49.859Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"stopping poller for jx/jenkins-x-chartmuseum"}
{"level":30,"message_time":"2021-10-29T15:19:49.859Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"starting poller for jx/jenkins-x-chartmuseum"}
{"level":30,"message_time":"2021-10-29T15:19:49.872Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"stopping poller for jx/jenkins-maven-settings"}
{"level":30,"message_time":"2021-10-29T15:19:49.872Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"starting poller for jx/jenkins-maven-settings"}
{"level":30,"message_time":"2021-10-29T15:19:49.874Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"stopping poller for jx/tekton-git"}
{"level":30,"message_time":"2021-10-29T15:19:49.874Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"starting poller for jx/tekton-git"}
{"level":30,"message_time":"2021-10-29T15:19:49.877Z","pid":19,"hostname":"kubernetes-external-secrets-67cd55dff4-9t86n","msg":"status update failed for externalsecret jx-production/tekton-container-registry-auth, due to modification, new poller should start"}```
uny commented 2 years ago

I got the same issue. EKS: 1.20 and 1.21

I have three private subnets in advance.

vpc ```hcl data "aws_availability_zones" "available" { state = "available" } resource "aws_eip" "nat" { count = local.public_count tags = { Name = "jx-nat-${count.index}" } vpc = true } resource "aws_internet_gateway" "main" { tags = { Name = "jx" } vpc_id = aws_vpc.main.id } resource "aws_nat_gateway" "main" { allocation_id = aws_eip.nat[count.index].id count = local.public_count subnet_id = aws_subnet.public[count.index].id tags = { Name = "jx-${count.index}" } } resource "aws_route_table" "private" { count = local.private_count tags = { Name = "jx-private-${count.index}" } vpc_id = aws_vpc.main.id route { cidr_block = "0.0.0.0/0" nat_gateway_id = aws_nat_gateway.main[count.index % local.public_count].id } } resource "aws_route_table" "public" { count = local.public_count tags = { Name = "jx-public-${count.index}" } vpc_id = aws_vpc.main.id route { cidr_block = "0.0.0.0/0" gateway_id = aws_internet_gateway.main.id } } resource "aws_route_table_association" "private" { count = local.private_count route_table_id = aws_route_table.private[count.index].id subnet_id = aws_subnet.private[count.index].id } resource "aws_route_table_association" "public" { count = local.public_count route_table_id = aws_route_table.public[count.index].id subnet_id = aws_subnet.public[count.index].id } resource "aws_subnet" "private" { availability_zone = data.aws_availability_zones.available.names[count.index] cidr_block = cidrsubnet("10.0.128.0/17", 2, count.index) count = local.private_count tags = { Name = "jx-private-${count.index}" } vpc_id = aws_vpc.main.id } resource "aws_subnet" "public" { availability_zone = data.aws_availability_zones.available.names[count.index] cidr_block = cidrsubnet("10.0.0.0/17", 2, count.index) count = local.public_count tags = { Name = "jx-public-${count.index}" } vpc_id = aws_vpc.main.id } resource "aws_vpc" "main" { cidr_block = "10.0.0.0/16" tags = { Name = "jx" } } ```

And I use the latest version of eks-jx.

eks-jx ```hcl module "eks-jx" { source = "jenkins-x/eks-jx/aws" version = "1.18.11" apex_domain = "xxx.xxx" cluster_name = var.cluster_name cluster_version = var.cluster_version create_and_configure_subdomain = true create_vpc = false enable_spot_instances = true enable_tls = true region = var.region vault_user = var.vault_user is_jx2 = false jx_git_url = var.jx_git_url jx_bot_username = var.jx_bot_username jx_bot_token = var.jx_bot_token force_destroy = var.force_destroy nginx_chart_version = var.nginx_chart_version install_kuberhealthy = var.install_kuberhealthy subdomain = "jx" subnets = aws_subnet.private[*].id tls_email = "xxx@xxx" vpc_id = aws_vpc.main.id } ```

Then, jx admin log output is:

output ``` Now using namespace 'jx' on server ''. jx verify ingress --ingress-service ingress-nginx-controller now verifying docker registry ingress setup jx gitops webhook update --warn-on-fail Error: failed to find hmac token from secret: could not find lighthouse hmac token lighthouse-hmac-token in namespace jx: secrets "lighthouse-hmac-token" not found Usage: update [flags] Examples: # update all the webhooks for all SourceRepository and Environment resource: jx-gitops update # only update the webhooks for a given owner jx-gitops update --org=mycorp # use a custom hook webhook endpoint (e.g. if you are on premise using node ports or something) jx-gitops update --endpoint http://mything.com Flags: -b, --batch-mode Runs in batch mode without prompting for user input --endpoint string Don't use the endpoint from the cluster, use the provided endpoint --exact-hook-url-match Whether to exactly match the hook based on the URL (default true) --git-kind string the kind of git server to connect to --git-server string the git server URL to create the scm client --git-token string the git token used to operate on the git repository. If not specified it's loaded from the git credentials file --git-username string the git username used to operate on the git repository. If not specified it's loaded from the git credentials file -h, --help help for update --hmac string Don't use the HMAC token from the cluster, use the provided token --log-level string Sets the logging level. If not specified defaults to $JX_LOG_LEVEL -o, --owner string The name of the git organisation or user to filter on --previous-hook-url string Whether to match based on an another URL -r, --repo string The name of the repository to filter on --verbose Enables verbose output. The environment variable JX_LOG_LEVEL has precedence over this flag and allows setting the logging level to any value of: panic, fatal, error, warn, info, debug, trace --warn-on-fail If enabled lets just log a warning that we could not update the webhook error: failed to find hmac token from secret: could not find lighthouse hmac token lighthouse-hmac-token in namespace jx: secrets "lighthouse-hmac-token" not found make: *** [versionStream/src/Makefile.mk:212: gitops-webhook-update] Error 1 ```

The below are suspicious errors in pods.

kubectl logs -n jx-vault vault-0 -c vault ``` telemetry.disable_hostname has been set to false. Recommended setting is true for Prometheus to avoid poorly named metrics. ==> Vault server configuration: Cgo: disabled Go Version: go1.16.7 Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled") Log Level: debug Mlock: supported: true, enabled: true Recovery Mode: false Storage: file Version: Vault v1.8.4 Version Sha: 925bc650ad1d997e84fbb832f302a6bfe0105bbb ==> Vault server started! Log data will stream in below: 2022-04-22T06:47:34.225Z [INFO] proxy environment: http_proxy="" https_proxy="" no_proxy="" 2022-04-22T06:47:34.225Z [WARN] no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually set 2022-04-22T06:47:34.284Z [DEBUG] would have sent systemd notification (systemd not present): notification=READY=1 2022/04/22 06:47:35 [ERR] Error flushing to statsd! Err: write udp 127.0.0.1:33118->127.0.0.1:9125: write: connection refused 2022-04-22T06:47:35.553Z [INFO] core: security barrier not initialized 2022-04-22T06:47:42.583Z [INFO] core: successful mount: namespace="" path=pki/ type=pki 2022-04-22T06:48:13.016Z [DEBUG] identity: creating a new entity: alias="id:"8caf244c-18e9-76e1-7039-be4b1000fe91" canonical_id:"5764b1b5-b9a8-4649-942c-995bd4cf10a2" mount_type:"kubernetes" mount_accessor:"auth_kubernetes_72148c58" mount_path:"auth/kubernetes/" metadata:{key:"service_account_name" value:"kubernetes-external-secrets"} metadata:{key:"service_account_namespace" value:"secret-infra"} metadata:{key:"service_account_secret_name" value:""} metadata:{key:"service_account_uid" value:"b84984a4-f96c-47fd-a295-2af34fc3ad99"} name:"b84984a4-f96c-47fd-a295-2af34fc3ad99" creation_time:{seconds:1650610093 nanos:16329231} last_update_time:{seconds:1650610093 nanos:16329231} namespace_id:"root"" ```
kubectl logs -n jx-vault vault-configurer-676764577f-frbpw ``` {"level":"info","msg":"vault metrics exporter enabled: :9091/metrics","time":"2022-04-22T06:47:18Z"} {"level":"info","msg":"applying config file : /config/vault-configurer/vault-config.yml","time":"2022-04-22T06:47:18Z"} {"level":"info","msg":"checking if vault is sealed...","time":"2022-04-22T06:47:18Z"} {"level":"info","msg":"watching directory for changes: /config/vault-configurer/","time":"2022-04-22T06:47:18Z"} {"level":"error","msg":"error checking if vault is sealed: error checking status: Get \"https://vault.jx-vault:8200/v1/sys/seal-status\": dial tcp 172.20.83.98:8200: connect: connection refused, waiting 5s before trying again...","time":"2022-04-22T06:47:25Z"} {"level":"info","msg":"checking if vault is sealed...","time":"2022-04-22T06:47:30Z"} {"level":"error","msg":"error checking if vault is sealed: error checking status: Get \"https://vault.jx-vault:8200/v1/sys/seal-status\": dial tcp 172.20.83.98:8200: connect: connection refused, waiting 5s before trying again...","time":"2022-04-22T06:47:37Z"} {"level":"info","msg":"checking if vault is sealed...","time":"2022-04-22T06:47:42Z"} {"level":"info","msg":"vault is unsealed, configuring...","time":"2022-04-22T06:47:42Z"} {"level":"info","msg":"already existing mounts: map[cubbyhole/:0xc00057e900 identity/:0xc00057ec00 sys/:0xc00057f000]","time":"2022-04-22T06:47:42Z"} {"level":"info","msg":"mounting secret engine with input: api.MountInput{Type:\"kv\", Description:\"General secrets.\", Config:api.MountConfigInput{Options:map[string]string{\"version\":\"2\"}, DefaultLeaseTTL:\"\", Description:(*string)(nil), MaxLeaseTTL:\"\", ForceNoCache:false, AuditNonHMACRequestKeys:[]string(nil), AuditNonHMACResponseKeys:[]string(nil), ListingVisibility:\"\", PassthroughRequestHeaders:[]string(nil), AllowedResponseHeaders:[]string(nil), TokenType:\"\", PluginName:\"\"}, Local:false, SealWrap:false, ExternalEntropyAccess:false, Options:map[string]string{\"version\":\"2\"}, PluginName:\"\"}","time":"2022-04-22T06:47:42Z"} {"level":"info","msg":"mounted kv to secret","time":"2022-04-22T06:47:42Z"} {"level":"info","msg":"already existing mounts: map[cubbyhole/:0xc00057fb00 identity/:0xc00057fe00 secret/:0xc000600100 sys/:0xc000600b00]","time":"2022-04-22T06:47:42Z"} {"level":"info","msg":"mounting secret engine with input: api.MountInput{Type:\"pki\", Description:\"Vault PKI Backend\", Config:api.MountConfigInput{Options:map[string]string{}, DefaultLeaseTTL:\"168h\", Description:(*string)(nil), MaxLeaseTTL:\"720h\", ForceNoCache:false, AuditNonHMACRequestKeys:[]string(nil), AuditNonHMACResponseKeys:[]string(nil), ListingVisibility:\"\", PassthroughRequestHeaders:[]string(nil), AllowedResponseHeaders:[]string(nil), TokenType:\"\", PluginName:\"\"}, Local:false, SealWrap:false, ExternalEntropyAccess:false, Options:map[string]string{}, PluginName:\"\"}","time":"2022-04-22T06:47:42Z"} {"level":"info","msg":"mounted pki to pki","time":"2022-04-22T06:47:42Z"} {"level":"info","msg":"successfully configured vault","time":"2022-04-22T06:47:43Z"} ```
kubectl logs -n jx-vault vault-configurer-676764577f-frbpw ``` {"level":"info","ts":1650610007.915303,"logger":"cmd","msg":"Watched namespace: "} {"level":"info","ts":1650610008.672247,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8383"} {"level":"info","ts":1650610008.6734798,"logger":"cmd","msg":"Registering Components."} {"level":"info","ts":1650610009.4279768,"logger":"cmd","msg":"Starting the Cmd."} I0422 06:46:49.429048 1 leaderelection.go:243] attempting to acquire leader lease jx-vault/vault-operator-lock... {"level":"info","ts":1650610009.4294531,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"} I0422 06:46:49.536495 1 leaderelection.go:253] successfully acquired lease jx-vault/vault-operator-lock {"level":"info","ts":1650610009.5368915,"logger":"controller-runtime.manager.controller.vault-controller","msg":"Starting EventSource","source":"kind source: /, Kind="} {"level":"info","ts":1650610009.5370023,"logger":"controller-runtime.manager.controller.vault-controller","msg":"Starting Controller"} {"level":"info","ts":1650610009.8377054,"logger":"controller-runtime.manager.controller.vault-controller","msg":"Starting workers","worker count":1} {"level":"info","ts":1650610013.4158235,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"jx-vault","Request.Name":"vault"} {"level":"info","ts":1650610013.6617181,"logger":"controller_vault","msg":"TLS CA will be regenerated due to: ","error":"an empty CA was provided"} {"level":"error","ts":1650610028.3257236,"logger":"controller-runtime.manager.controller.vault-controller","msg":"Reconciler error","name":"vault","namespace":"jx-vault","error":"failed to distribute CA secret for vault: failed to query current secret for vault: Secret \"vault-tls\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214"} {"level":"info","ts":1650610028.331497,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"jx-vault","Request.Name":"vault"} {"level":"info","ts":1650610036.0240972,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"jx-vault","Request.Name":"vault"} {"level":"info","ts":1650610140.3777287,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"jx-vault","Request.Name":"vault"} {"level":"info","ts":1650610140.6331828,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"jx-vault","Request.Name":"vault"} {"level":"info","ts":1650610205.6644564,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"jx-vault","Request.Name":"vault"} {"level":"info","ts":1650610270.9514675,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"jx-vault","Request.Name":"vault"} {"level":"info","ts":1650610336.2386463,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"jx-vault","Request.Name":"vault"} {"level":"info","ts":1650610401.5249374,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"jx-vault","Request.Name":"vault"} {"level":"info","ts":1650610466.8118722,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"jx-vault","Request.Name":"vault"} {"level":"info","ts":1650610532.0981042,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"jx-vault","Request.Name":"vault"} {"level":"info","ts":1650610597.384733,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"jx-vault","Request.Name":"vault"} {"level":"info","ts":1650610662.6709259,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"jx-vault","Request.Name":"vault"} {"level":"info","ts":1650610727.9575841,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"jx-vault","Request.Name":"vault"} {"level":"info","ts":1650610793.244381,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"jx-vault","Request.Name":"vault"} {"level":"info","ts":1650610858.5312705,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"jx-vault","Request.Name":"vault"} {"level":"info","ts":1650610923.817538,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"jx-vault","Request.Name":"vault"} {"level":"info","ts":1650610989.1041584,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"jx-vault","Request.Name":"vault"} {"level":"info","ts":1650611054.3912048,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"jx-vault","Request.Name":"vault"} {"level":"info","ts":1650611119.678339,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"jx-vault","Request.Name":"vault"} {"level":"info","ts":1650611184.965026,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"jx-vault","Request.Name":"vault"} {"level":"info","ts":1650611250.2518773,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"jx-vault","Request.Name":"vault"} {"level":"info","ts":1650611315.5389533,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"jx-vault","Request.Name":"vault"} {"level":"info","ts":1650611380.8254879,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"jx-vault","Request.Name":"vault"} {"level":"info","ts":1650611446.1120176,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"jx-vault","Request.Name":"vault"} ```
kubectl logs -n secret-infra kubernetes-external-secrets-67cd55dff4-db8ht ``` {"level":50,"message_time":"2022-04-22T06:49:15.302Z","pid":18,"hostname":"kubernetes-external-secrets-67cd55dff4-db8ht","payload":{"response":{"statusCode":404,"body":{"errors":[]}}},"msg":"failure while polling the secret jx/jenkins-x-chartmuseum"} {"level":50,"message_time":"2022-04-22T06:49:15.329Z","pid":18,"hostname":"kubernetes-external-secrets-67cd55dff4-db8ht","payload":{"response":{"statusCode":404,"body":{"errors":[]}}},"msg":"failure while polling the secret jx-production/tekton-container-registry-auth"} {"level":50,"message_time":"2022-04-22T06:49:15.337Z","pid":18,"hostname":"kubernetes-external-secrets-67cd55dff4-db8ht","payload":{"response":{"statusCode":404,"body":{"errors":[]}}},"msg":"failure while polling the secret jx/nexus"} {"level":50,"message_time":"2022-04-22T06:49:15.418Z","pid":18,"hostname":"kubernetes-external-secrets-67cd55dff4-db8ht","payload":{"response":{"statusCode":404,"body":{"errors":[]}}},"msg":"failure while polling the secret jx-staging/tekton-container-registry-auth"} {"level":50,"message_time":"2022-04-22T06:49:15.432Z","pid":18,"hostname":"kubernetes-external-secrets-67cd55dff4-db8ht","payload":{"response":{"statusCode":404,"body":{"errors":[]}}},"msg":"failure while polling the secret jx/tekton-container-registry-auth"} {"level":50,"message_time":"2022-04-22T06:49:15.443Z","pid":18,"hostname":"kubernetes-external-secrets-67cd55dff4-db8ht","payload":{"response":{"statusCode":404,"body":{"errors":[]}}},"msg":"failure while polling the secret jx/jx-basic-auth-htpasswd"} {"level":50,"message_time":"2022-04-22T06:49:15.460Z","pid":18,"hostname":"kubernetes-external-secrets-67cd55dff4-db8ht","payload":{"response":{"statusCode":404,"body":{"errors":[]}}},"msg":"failure while polling the secret jx/jx-basic-auth-user-password"} {"level":50,"message_time":"2022-04-22T06:49:15.461Z","pid":18,"hostname":"kubernetes-external-secrets-67cd55dff4-db8ht","payload":{"response":{"statusCode":404,"body":{"errors":[]}}},"msg":"failure while polling the secret jx/tekton-git"} {"level":50,"message_time":"2022-04-22T06:49:15.469Z","pid":18,"hostname":"kubernetes-external-secrets-67cd55dff4-db8ht","payload":{"response":{"statusCode":404,"body":{"errors":[]}}},"msg":"failure while polling the secret jx/lighthouse-hmac-token"} {"level":50,"message_time":"2022-04-22T06:49:15.515Z","pid":18,"hostname":"kubernetes-external-secrets-67cd55dff4-db8ht","payload":{"response":{"statusCode":404,"body":{"errors":[]}}},"msg":"failure while polling the secret jx/jenkins-maven-settings"} ```
kubectl logs -n secret-infra pusher-wave-pusher-wave-79c4845b8c-8cjhq ``` E0422 06:47:54.424745 1 controller.go:218] controller-runtime/controller "msg"="Reconciler error" "error"="error fetching current children: error(s) encountered when geting children: Secret \"tekton-git\" not found" "controller"="deployment-controller" "request"={"Namespace":"jx","Name":"jx-pipelines-visualizer"} E0422 06:47:56.617974 1 controller.go:218] controller-runtime/controller "msg"="Reconciler error" "error"="error updating OwnerReferences: error(s) encountered updating children: error updating child: Internal error occurred: failed calling webhook \"config.webhook.pipeline.tekton.dev\": Post \"https://tekton-pipelines-webhook.tekton-pipelines.svc:443/config-validation?timeout=10s\": no endpoints available for service \"tekton-pipelines-webhook\", error updating child: Internal error occurred: failed calling webhook \"config.webhook.pipeline.tekton.dev\": Post \"https://tekton-pipelines-webhook.tekton-pipelines.svc:443/config-validation?timeout=10s\": no endpoints available for service \"tekton-pipelines-webhook\"" "controller"="deployment-controller" "request"={"Namespace":"tekton-pipelines","Name":"tekton-pipelines-controller"} E0422 06:47:57.618548 1 controller.go:218] controller-runtime/controller "msg"="Reconciler error" "error"="error fetching current children: error(s) encountered when geting children: Secret \"lighthouse-hmac-token\" not found, Secret \"lighthouse-oauth-token\" not found" "controller"="deployment-controller" "request"={"Namespace":"jx","Name":"lighthouse-webhooks"} E0422 07:12:54.313509 1 controller.go:218] controller-runtime/controller "msg"="Reconciler error" "error"="error fetching current children: error(s) encountered when geting children: Secret \"lighthouse-oauth-token\" not found, Secret \"lighthouse-hmac-token\" not found" "controller"="deployment-controller" "request"={"Namespace":"jx","Name":"lighthouse-foghorn"} E0422 07:12:55.313946 1 controller.go:218] controller-runtime/controller "msg"="Reconciler error" "error"="error fetching current children: error(s) encountered when geting children: Secret \"jenkins-x-chartmuseum\" not found" "controller"="deployment-controller" "request"={"Namespace":"jx","Name":"jenkins-x-chartmuseum"} E0422 07:12:56.314686 1 controller.go:218] controller-runtime/controller "msg"="Reconciler error" "error"="error fetching current children: error(s) encountered when geting children: Secret \"lighthouse-hmac-token\" not found, Secret \"lighthouse-oauth-token\" not found" "controller"="deployment-controller" "request"={"Namespace":"jx","Name":"lighthouse-webhooks"} E0422 07:12:57.315233 1 controller.go:218] controller-runtime/controller "msg"="Reconciler error" "error"="error fetching current children: error(s) encountered when geting children: Secret \"tekton-git\" not found" "controller"="deployment-controller" "request"={"Namespace":"jx","Name":"jx-build-controller"} E0422 07:12:58.315868 1 controller.go:218] controller-runtime/controller "msg"="Reconciler error" "error"="error fetching current children: error(s) encountered when geting children: Secret \"tekton-git\" not found" "controller"="deployment-controller" "request"={"Namespace":"jx","Name":"jx-pipelines-visualizer"} E0422 07:12:59.316895 1 controller.go:218] controller-runtime/controller "msg"="Reconciler error" "error"="error fetching current children: error(s) encountered when geting children: Secret \"nexus\" not found" "controller"="deployment-controller" "request"={"Namespace":"jx","Name":"nexus-nexus"} E0422 07:13:00.317376 1 controller.go:218] controller-runtime/controller "msg"="Reconciler error" "error"="error fetching current children: error(s) encountered when geting children: Secret \"lighthouse-oauth-token\" not found" "controller"="deployment-controller" "request"={"Namespace":"jx","Name":"lighthouse-keeper"} ```