search

khuedoan / homelab

Fully automated homelab from empty disk to running services with a single command.
https://homelab.khuedoan.com
GNU General Public License v3.0
7.9k stars 705 forks source link

ExternalSecret could not get secret data from provider #115

Closed lianghuiyuan closed 1 year ago

lianghuiyuan commented 1 year ago

Brief description

1'st: Found that Gitea's 'HEALTH STATUS' is 'Degraded', 2'nd: Found "ExternalSecret could not get secret data from provider", 3'rd: vault job"generate-secrets-xxxx" Error: Unable to write secret: Put "http://vault:8200/v1/secret/data/gitea/admin": dial tcp 10.43.79.47:8200: connect: connection refused 1 4'th: Statefulset's pod 'vault-0' Degraded, Logs show "[INFO] core: security barrier not initialized" 5'th: Can not ping vault ClusterIP(10.43.79.47) from vault-configurer-6877954c5-6ltck

Details

kubectl get applications -n argocd | grep -v Healthy

[nix-shell:/home/creasy/githubs/homelab]# kubectl get applications -n argocd | grep -v Healthy
NAME                SYNC STATUS   HEALTH STATUS
dex                 OutOfSync     Degraded
gitea               OutOfSync     Degraded
grafana             OutOfSync     Degraded
registry            OutOfSync     Degraded
renovate            OutOfSync     Degraded
root                Synced        Degraded
tekton-pipelines    OutOfSync     Degraded
[nix-shell:/home/creasy/githubs/homelab]# kubectl get applications gitea -n argocd -o yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  creationTimestamp: "2023-06-19T14:05:53Z"
  finalizers:
  - resources-finalizer.argocd.argoproj.io
  generation: 4214
  name: gitea
  namespace: argocd
  ownerReferences:
  - apiVersion: argoproj.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: ApplicationSet
    name: platform
    uid: 0e9d9913-5dbc-4316-b2c9-3e7d13edaca2
  resourceVersion: "973718"
  uid: cf853233-fec7-4286-a35f-1cd061fe696e
spec:
  destination:
    name: in-cluster
    namespace: gitea
  project: default
  source:
    path: platform/gitea
    repoURL: https://github.com/lianghuiyuan/homelab
    targetRevision: master
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    retry:
      backoff:
        duration: 1m
        factor: 2
        maxDuration: 16m
      limit: 10
    syncOptions:
    - CreateNamespace=true
    - ApplyOutOfSyncOnly=true
    - ServerSideApply=true
status:
  health:
    status: Degraded
  history:
  - deployStartedAt: "2023-06-19T14:05:56Z"
    deployedAt: "2023-06-20T11:56:53Z"
    id: 0
    revision: 15d6884cc2b45db9206799613571d50b2f90fbb5
    source:
      path: platform/gitea
      repoURL: https://github.com/lianghuiyuan/homelab
      targetRevision: master
  operationState:
    finishedAt: "2023-06-20T14:10:25Z"
    message: successfully synced (all tasks run)
    operation:
      initiatedBy:
        automated: true
      retry:
        backoff:
          duration: 1m
          factor: 2
          maxDuration: 16m
        limit: 10
      sync:
        prune: true
        resources:
        - group: apps
          kind: StatefulSet
          name: gitea
        - group: external-secrets.io
          kind: ExternalSecret
          name: gitea-admin-secret
        - group: external-secrets.io
          kind: ExternalSecret
          name: gitea-webhook-secret
        - group: apps
          kind: StatefulSet
          name: gitea-postgresql
        revision: 15d6884cc2b45db9206799613571d50b2f90fbb5
        syncOptions:
        - CreateNamespace=true
        - ApplyOutOfSyncOnly=true
        - ServerSideApply=true
    phase: Succeeded
    startedAt: "2023-06-20T14:10:24Z"
    syncResult:
      resources:
      - group: apps
        hookPhase: Running
        kind: StatefulSet
        message: statefulset.apps/gitea-postgresql serverside-applied
        name: gitea-postgresql
        namespace: gitea
        status: Synced
        syncPhase: Sync
        version: v1
      - group: apps
        hookPhase: Running
        kind: StatefulSet
        message: statefulset.apps/gitea serverside-applied
        name: gitea
        namespace: gitea
        status: Synced
        syncPhase: Sync
        version: v1
      - group: external-secrets.io
        hookPhase: Running
        kind: ExternalSecret
        message: externalsecret.external-secrets.io/gitea-webhook-secret serverside-applied
        name: gitea-webhook-secret
        namespace: gitea
        status: Synced
        syncPhase: Sync
        version: v1beta1
      - group: external-secrets.io
        hookPhase: Running
        kind: ExternalSecret
        message: externalsecret.external-secrets.io/gitea-admin-secret serverside-applied
        name: gitea-admin-secret
        namespace: gitea
        status: Synced
        syncPhase: Sync
        version: v1beta1
      revision: 15d6884cc2b45db9206799613571d50b2f90fbb5
      source:
        path: platform/gitea
        repoURL: https://github.com/lianghuiyuan/homelab
        targetRevision: master
  reconciledAt: "2023-06-20T14:10:25Z"
  resources:
  - kind: ConfigMap
    name: gitea-config-source
    namespace: gitea
    status: Synced
    version: v1
  - kind: Secret
    name: gitea
    namespace: gitea
    status: Synced
    version: v1
  - kind: Secret
    name: gitea-init
    namespace: gitea
    status: Synced
    version: v1
  - kind: Secret
    name: gitea-inline-config
    namespace: gitea
    status: Synced
    version: v1
  - kind: Secret
    name: gitea-postgresql
    namespace: gitea
    status: Synced
    version: v1
  - health:
      status: Healthy
    kind: Service
    name: gitea-http
    namespace: gitea
    status: Synced
    version: v1
  - health:
      status: Healthy
    kind: Service
    name: gitea-memcached
    namespace: gitea
    status: Synced
    version: v1
  - health:
      status: Healthy
    kind: Service
    name: gitea-postgresql
    namespace: gitea
    status: Synced
    version: v1
  - health:
      status: Healthy
    kind: Service
    name: gitea-postgresql-headless
    namespace: gitea
    status: Synced
    version: v1
  - health:
      status: Healthy
    kind: Service
    name: gitea-ssh
    namespace: gitea
    status: Synced
    version: v1
  - kind: ServiceAccount
    name: gitea-memcached
    namespace: gitea
    status: Synced
    version: v1
  - group: apps
    health:
      status: Healthy
    kind: Deployment
    name: gitea-memcached
    namespace: gitea
    status: Synced
    version: v1
  - group: apps
    health:
      message: Waiting for 1 pods to be ready...
      status: Progressing
    kind: StatefulSet
    name: gitea
    namespace: gitea
    status: OutOfSync
    version: v1
  - group: apps
    health:
      message: statefulset rolling update complete 1 pods at revision gitea-postgresql-989898574...
      status: Healthy
    kind: StatefulSet
    name: gitea-postgresql
    namespace: gitea
    status: OutOfSync
    version: v1
  - group: batch
    kind: CronJob
    name: gitea-config
    namespace: gitea
    status: Synced
    syncWave: 1
    version: v1
  - group: external-secrets.io
    health:
      message: could not get secret data from provider
      status: Degraded
    kind: ExternalSecret
    name: gitea-admin-secret
    namespace: gitea
    status: OutOfSync
    version: v1beta1
  - group: external-secrets.io
    health:
      message: could not get secret data from provider
      status: Degraded
    kind: ExternalSecret
    name: gitea-webhook-secret
    namespace: gitea
    status: OutOfSync
    version: v1beta1
  - group: networking.k8s.io
    health:
      status: Healthy
    kind: Ingress
    name: gitea
    namespace: gitea
    status: Synced
    version: v1
  sourceType: Helm
  summary:
    externalURLs:
    - https://git.lhy.me/
    images:
    - docker.io/bitnami/memcached:1.6.9-debian-10-r114
    - docker.io/bitnami/postgresql:11.11.0-debian-10-r62
    - gitea/gitea:1.16.4
    - golang:1.19-alpine
  sync:
    comparedTo:
      destination:
        name: in-cluster
        namespace: gitea
      source:
        path: platform/gitea
        repoURL: https://github.com/lianghuiyuan/homelab
        targetRevision: master
    revision: 15d6884cc2b45db9206799613571d50b2f90fbb5
    status: OutOfSync

kubectl get ExternalSecret --all-namespaces

[nix-shell:/home/creasy/githubs/homelab]# kubectl get ExternalSecret --all-namespaces
NAMESPACE          NAME                    STORE   REFRESH INTERVAL   STATUS              READY
dex                dex-secrets             vault   1h                 SecretSyncedError   False
gitea              gitea-admin-secret      vault   1h                 SecretSyncedError   False
gitea              gitea-webhook-secret    vault   1h                 SecretSyncedError   False
grafana            grafana-secrets         vault   1h                 SecretSyncedError   False
registry           registry-admin-secret   vault   1h                 SecretSyncedError   False
renovate           renovate-secret         vault   1h                 SecretSyncedError   False
tekton-workflows   webhook-secret          vault   1h                 SecretSyncedError   False

kubectl get ExternalSecret gitea-admin-secret -n gitea -o yaml

[nix-shell:/home/creasy/githubs/homelab]# kubectl get ExternalSecret gitea-admin-secret -n gitea -o yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  creationTimestamp: "2023-06-19T14:05:59Z"
  generation: 1
  labels:
    argocd.argoproj.io/instance: gitea
  name: gitea-admin-secret
  namespace: gitea
  resourceVersion: "944536"
  uid: 46fed316-a399-4dba-9ded-91f5fb43d3b8
spec:
  data:
  - remoteRef:
      conversionStrategy: Default
      decodingStrategy: None
      key: /gitea/admin
      property: password
    secretKey: password
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: vault
  target:
    creationPolicy: Owner
    deletionPolicy: Retain
    template:
      data:
        password: '{{ .password }}'
        username: gitea_admin
      engineVersion: v2
status:
  conditions:
  - lastTransitionTime: "2023-06-19T14:06:14Z"
    message: could not get secret data from provider
    reason: SecretSyncedError
    status: "False"
    type: Ready

kubectl get all -n external-secrets

[nix-shell:/home/creasy/githubs/homelab]# kubectl get all -n external-secrets
NAME                                                   READY   STATUS    RESTARTS       AGE
pod/external-secrets-7f9b5b997d-5l64l                  1/1     Running   1 (153m ago)   24h
pod/external-secrets-cert-controller-c7b56c57d-9tpsk   1/1     Running   2 (153m ago)   24h
pod/external-secrets-webhook-68678fbbc5-csx6q          1/1     Running   2 (153m ago)   24h

NAME                               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/external-secrets-webhook   ClusterIP   10.43.235.169   <none>        443/TCP   24h

NAME                                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/external-secrets                   1/1     1            1           24h
deployment.apps/external-secrets-cert-controller   1/1     1            1           24h
deployment.apps/external-secrets-webhook           1/1     1            1           24h

NAME                                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/external-secrets-7f9b5b997d                  1         1         1       24h
replicaset.apps/external-secrets-cert-controller-c7b56c57d   1         1         1       24h
replicaset.apps/external-secrets-webhook-68678fbbc5          1         1         1       24h
lianghuiyuan commented 1 year ago

vault generate secrets jobs Error

[nix-shell:/home/creasy/githubs/homelab]# kubectl get jobs -n vault
NAME                        COMPLETIONS   DURATION   AGE
generate-secrets-28121390   0/1           4m13s      4m13s

[nix-shell:/home/creasy/githubs/homelab]# kubectl get pods -n vault
NAME                                   READY   STATUS             RESTARTS       AGE
generate-secrets-28121390-k2plx        0/1     Error              0              4m22s
generate-secrets-28121390-lbnbw        0/1     Error              0              3m10s
generate-secrets-28121390-rp9lw        0/1     Error              0              3m50s
generate-secrets-28121390-w28gz        0/1     Error              0              2m9s
vault-0                                1/3     CrashLoopBackOff   27 (83s ago)   115m
vault-configurer-6877954c5-6ltck       1/1     Running            2 (119m ago)   27h
vault-vault-operator-f78fbbfc4-twgtt   1/1     Running            2 (119m ago)   27h

Logs of the job's pods

2023/06/20 15:38:34 Unable to write secret: Put "http://vault:8200/v1/secret/data/gitea/admin": dial tcp 10.43.79.47:8200: connect: connection refused
1
exit status 1

kubectl get svc -n vault

[nix-shell:/home/creasy/githubs/homelab]# kubectl get svc -n vault
NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                               AGE
vault                  ClusterIP   10.43.79.47     <none>        8200/TCP,8201/TCP,9091/TCP,9102/TCP   29h
vault-0                ClusterIP   10.43.229.192   <none>        8200/TCP,8201/TCP,9091/TCP            29h
vault-configurer       ClusterIP   10.43.30.161    <none>        9091/TCP                              29h
vault-vault-operator   ClusterIP   10.43.175.66    <none>        80/TCP,8383/TCP                       29h
lianghuiyuan commented 1 year ago

IP of vault-configurer-6877954c5-rtj8r

[nix-shell:/home/creasy/githubs/homelab]# kubectl exec -i -t vault-configurer-6877954c5-rtj8r -n vault -- sh
/config $ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0@if24: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP 
    link/ether b2:a4:47:41:f2:a1 brd ff:ff:ff:ff:ff:ff
    inet 10.42.4.97/24 brd 10.42.4.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::b0a4:47ff:fe41:f2a1/64 scope link 
       valid_lft forever preferred_lft forever

ping vault ClusterIP from vault-configurer-6877954c5-6ltck

/config $ ping 10.43.79.47
PING 10.43.79.47 (10.43.79.47): 56 data bytes
^C
--- 10.43.79.47 ping statistics ---
17 packets transmitted, 0 packets received, 100% packet loss
/config $ ping baidu.com
PING baidu.com (110.242.68.66): 56 data bytes
64 bytes from 110.242.68.66: seq=0 ttl=42 time=49.257 ms
64 bytes from 110.242.68.66: seq=1 ttl=42 time=48.821 ms
^C
--- baidu.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 48.821/49.039/49.257 ms
lianghuiyuan commented 1 year ago

Redeploy fix the problem again! 😂😂