All secrets not populated during operator install

[Raboo]

Bootstrapping jx3 with jx admin operator on a baremetal cluster fails to populate all secrets.

$ jx secret verify
SECRET                                       STATUS
jx-production/tekton-container-registry-auth valid: .dockerconfigjson/.dockerconfigjson
jx-staging/tekton-container-registry-auth    valid: .dockerconfigjson/.dockerconfigjson
jx/bucketrepo-config                         valid: config.yaml/config.yaml
jx/jenkins-maven-settings                    valid: settings.xml/settings.xml, settings-security.xml/settings-security.xml
jx/jenkins-release-gpg                       key sec-jenkins.gpg missing properties: sec-jenkins.gpg
jx/jenkins-release-gpg                       key secring.gpg missing properties: secring.gpg
jx/jenkins-release-gpg                       key trustdb.gpg missing properties: trustdb.gpg
jx/jenkins-x-bucketrepo                      valid: BASIC_AUTH_PASS/BASIC_AUTH_PASS, BASIC_AUTH_USER/BASIC_AUTH_USER
jx/jx-basic-auth-htpasswd                    key auth missing properties: auth
jx/jx-basic-auth-user-password               valid: password/password, username/username
jx/jx-local-secrets                          key secrets.yaml missing properties: secrets.yaml
jx/lighthouse-hmac-token                     valid: hmac/hmac
jx/lighthouse-oauth-token                    valid: oauth/oauth
jx/tekton-container-registry-auth            valid: .dockerconfigjson/.dockerconfigjson
jx/tekton-git                                valid: password/password, username/username

So I'm missing jx/jx-basic-auth-htpasswd

During the operator install I do get a lot of warnings like this, I've cut down repeated messages...

WARNING: failed to find referenced External Secret name nexus
WARNING: failed to find referenced External Secret name nexus
WARNING: failed to find referenced External Secret name jenkins-x-chartmuseum
WARNING: failed to find referenced External Secret name jenkins-x-chartmuseum
WARNING: failed to find secret key and property for External Secret name jenkins-x-bucketrepo
WARNING: failed to find secret key and property for External Secret name jenkins-x-bucketrepo
WARNING: failed to find referenced External Secret name sonatype
WARNING: failed to find referenced External Secret name sonatype
WARNING: failed to find referenced External Secret name docker-hub
WARNING: failed to find referenced External Secret name docker-hub
WARNING: failed to find referenced External Secret name gpg
WARNING: failed to find referenced External Secret name secret/data/jenkins-x-bucketrepo
WARNING: failed to find referenced External Secret name secret/data/jenkins-x-bucketrepo
WARNING: secret jx-basic-auth-user-password in namespace jx cannot be found - backoff loop
WARNING: secret jx-basic-auth-user-password in namespace jx cannot be found - backoff loop
WARNING: failed to create htpasswd: secret jx-basic-auth-user-password does not have username entry username in namespace jx

jx-requirements.yml:

apiVersion: core.jenkins-x.io/v4beta1
kind: Requirements
spec:
  autoUpdate:
    enabled: false
    schedule: ""
  cluster:
    chartRepository: http://bucketrepo.jx.svc.cluster.local/bucketrepo/charts
    clusterName: kind
    devEnvApprovers:
    - todo
    environmentGitOwner: todo
    gitKind: github
    gitName: github
    gitServer: https://github.com
    provider: kubernetes
    registry: ghcr.io
  environments:
  - key: dev
    owner: xxx
    repository: jx-pipeline-k8s-workloads
  - key: staging
  - key: production
  ingress:
    domain: workloads.example.net
    externalDNS: false
    kind: ingress
    namespaceSubDomain: -jx.
    tls:
      email: it@example.com
      enabled: true
      production: true
  pipelineUser:
    username: xxx
  repository: bucketrepo
  secretStorage: local
  vault: {}
  webhook: lighthouse

I might add that the boot have executed more than once. How do I work around this?

Update: I can add that basically every "Boot" job or whatever it is called seems to be running very long and waiting for some timeout for these secrets.

WARNING: secret jx-basic-auth-user-password in namespace jx cannot be found - backoff loop
WARNING: secret jx-basic-auth-user-password in namespace jx cannot be found - backoff loop
WARNING: secret jx-basic-auth-user-password in namespace jx cannot be found - backoff loop
...

[Raboo]

I'm adding other discoveries in the log that is related to secrets

Error: failed to populate secrets: failed to save properties key: pubring.gpg properties: pubring.gpg on ExternalSecret jenkins-release-gpg: failed to create Secret jenkins-release-gpg in namespace jx: namespaces "jx" not found
Usage:
  populate [flags]

Examples:
  jx-secret populate

Flags:
      --boot-secret-namespace string   the namespace to that contains the boot secret used to populate git secrets from
  -d, --dir string                     the directory to look for the .jx/secret/mapping/secret-mappings.yaml file (default ".")
  -f, --filter string                  the filter to filter on ExternalSecret names
  -h, --help                           help for populate
      --no-wait                        disables waiting for the secret store (e.g. vault) to be available
  -n, --ns string                      the namespace to filter the ExternalSecret resources
      --secret-namespace string        the namespace in which secret infrastructure resides such as Hashicorp Vault (default "jx-vault")
  -s, --source string                  the source location for the ExternalSecrets, valid values include filesystem or kubernetes (default "kubernetes")
  -w, --wait duration                  the maximum time period to wait for the vault pod to be ready if using the vault backendType (default 2h0m0s)

error: failed to populate secrets: failed to save properties key: pubring.gpg properties: pubring.gpg on ExternalSecret jenkins-release-gpg: failed to create Secret jenkins-release-gpg in namespace jx: namespaces "jx" not found
make[1]: [versionStream/src/Makefile.mk:67: fetch] Error 1 (ignored)
# lets make sure all the namespaces exist for environments of the replicated secrets
jx gitops namespace --dir-mode --dir config-root/namespaces

[Raboo]

Another discovery I made is that there is a bunch of failed jx-secrets-xxx pods.

All of which failed with same message.

FATAL: failed to list source repositories: failed to list external secrets: failed to find external secrets: externalsecrets.kubernetes-client.io is forbidden: User "system:serviceaccount:jx:jx-secrets-sa" cannot list resource "externalsecrets" in API group "kubernetes-client.io" at the cluster scope

[rahtr]

I am having same issues on my OCP 4.6 clusters.

Update: I logged it again to the cluster and this time it went through. Still checking it would work:

# they can be modified/regenerated at any time via `jx secret edit`
VAULT_ADDR=https://vault.jx-vault:8200 VAULT_NAMESPACE=jx-vault jx secret populate --secret-namespace jx-vault
loading default Secret data from helm secret folder: /tmp/secrets/jx-helm
WARNING: secret jx-basic-auth-user-password in namespace jx cannot be found - backoff loop
WARNING: secret jx-basic-auth-user-password in namespace jx cannot be found - backoff loop
WARNING: secret jx-basic-auth-user-password in namespace jx cannot be found - backoff loop
WARNING: secret jx-basic-auth-user-password in namespace jx cannot be found - backoff loop
WARNING: secret jx-basic-auth-user-password in namespace jx cannot be found - backoff loop
WARNING: secret jx-basic-auth-user-password in namespace jx cannot be found - backoff loop
WARNING: secret jx-basic-auth-user-password in namespace jx cannot be found - backoff loop
WARNING: secret jx-basic-auth-user-password in namespace jx cannot be found - backoff loop
WARNING: secret jx-basic-auth-user-password in namespace jx cannot be found - backoff loop
WARNING: secret jx-basic-auth-user-password in namespace jx cannot be found - backoff loop
WARNING: failed to create htpasswd: secret jx-basic-auth-user-password does not have username entry username in namespace jx
WARNING: secret jx-basic-auth-user-password in namespace jx cannot be found - backoff loop
WARNING: secret jx-basic-auth-user-password in namespace jx cannot be found - backoff loop
WARNING: secret jx-basic-auth-user-password in namespace jx cannot be found - backoff loop
WARNING: secret jx-basic-auth-user-password in namespace jx cannot be found - backoff loop
WARNING: secret jx-basic-auth-user-password in namespace jx cannot be found - backoff loop
WARNING: secret jx-basic-auth-user-password in namespace jx cannot be found - backoff loop
WARNING: secret jx-basic-auth-user-password in namespace jx cannot be found - backoff loop
WARNING: secret jx-basic-auth-user-password in namespace jx cannot be found - backoff loop
WARNING: secret jx-basic-auth-user-password in namespace jx cannot be found - backoff loop
WARNING: secret jx-basic-auth-user-password in namespace jx cannot be found - backoff loop
WARNING: failed to create htpasswd: secret jx-basic-auth-user-password does not have username entry username in namespace jx
changing to the jx namespace to verify
jx ns jx --quiet
Now using namespace 'jx' on server ''.
jx verify ingress --ingress-service ingress-nginx-controller
now verifying docker registry ingress setup
jx gitops webhook update --warn-on-fail
W0226 17:16:19.277458    2717 warnings.go:70] extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
Checking hooks for repository ratripathi/rahul-jenkins
echo completed > jx-boot-completed.txt
echo wrote completed file
wrote completed file
boot Job pod jx-boot-6a0743ee-b7f2-4ccc-9ee2-c1ae08fa7200-pcwhs has Succeeded
boot Job jx-boot-6a0743ee-b7f2-4ccc-9ee2-c1ae08fa7200 has Succeeded

[Raboo]

The boot job always succeeds, I believe that it just waits for some timeout and then ignores the fact that the basic-auth has not been populated.

[Raboo]

I keep getting spammed with

WARNING: secret jx-basic-auth-user-password in namespace jx cannot be found - backoff loop
WARNING: secret jx-basic-auth-user-password in namespace jx cannot be found - backoff loop

for every boot job even after I manually populated jx-basic-auth-htpasswd and jx-basic-auth-user-password. So apparently something is b0rkd.

[wowq]

same issue

[jstrachan]

is this still an issue?

[Raboo]

I don't know, I dropped jx and built my own pipeline.

[wowq]

Now it's right

[Raboo]

@wowq so this is fixed now, I should then close this issue?

[wowq]

yes