certKey.pem is missing - Githubissues

peteroneilljr commented 1 year ago

I'm trying to set this up, and it says certKey.pem is missing. But I don't see this file mentioned in the docs anywhere.

2023-07-31 16:25:30.905Z INFO[2023-07-31T16:25:30Z] Getting configuration from ./config/config.yaml  scope=Config
2023-07-31 16:25:30.905Z TRAC[2023-07-31T16:25:30Z] ./config/config.yaml exists and contains 207 bytes:
2023-07-31 16:25:30.905Z  useEST: false
2023-07-31 16:25:30.905Z defaultESTAlias:
2023-07-31 16:25:30.905Z defaultCertificateProfileName: ShortLivedProfile
2023-07-31 16:25:30.905Z defaultEndEntityProfileName: ShortLivedProfile
2023-07-31 16:25:30.905Z defaultCertificateAuthorityName: k8sRootCA
2023-07-31 16:25:30.905Z healthcheckPort: 5354
2023-07-31 16:25:30.905Z chainDepth: 4  scope=Config
2023-07-31 16:25:30.905Z INFO[2023-07-31T16:25:30Z] Successfully retrieved configuration:
2023-07-31 16:25:30.905Z  &config.ServerConfig{HealthCheckPort:"5354", DefaultCertificateProfileName:"ShortLivedProfile", DefaultEndEntityProfileName:"ShortLivedProfile", DefaultCertificateAuthorityName:"k8sRootCA", UseEST:false, DefaultESTAlias:"", ChainDepth:4}  scope=Config
2023-07-31 16:25:30.905Z INFO[2023-07-31T16:25:30Z] Getting credentials from credentials.yaml     scope=Credential
2023-07-31 16:25:30.905Z TRAC[2023-07-31T16:25:30Z] credentials.yaml exists and contains 433 bytes  scope=Credential
2023-07-31 16:25:30.905Z INFO[2023-07-31T16:25:30Z] Successfully retrieved credentials.           scope=Credential
2023-07-31 16:25:30.905Z INFO[2023-07-31T16:25:30Z] Looking in /clientcert/ for client certificates  scope=Credential
2023-07-31 16:25:30.905Z WARN[2023-07-31T16:25:30Z] read /clientcert/..data: is a directory       scope=Credential
2023-07-31 16:25:30.905Z INFO[2023-07-31T16:25:30Z] tls.crt exists and contains 1589 bytes        scope=Credential
2023-07-31 16:25:30.905Z INFO[2023-07-31T16:25:30Z] tls.key exists and contains 1703 bytes        scope=Credential
2023-07-31 16:25:30.905Z INFO[2023-07-31T16:25:30Z] Successfully retrieved client certificate     scope=Credential
2023-07-31 16:25:30.905Z DEBU[2023-07-31T16:25:30Z] Creating EJBCA client (useEST=false)          scope=Main
2023-07-31 16:25:30.905Z 2023/07/31 16:25:30 [INFO] Building new EJBCA client
2023-07-31 16:25:30.905Z 2023/07/31 16:25:30 [TRACE] Reading client certificate from certkey.pem
2023-07-31 16:25:30.905Z FATA[2023-07-31T16:25:30Z] no private key found in certkey.pem and no path to a private key specified  scope=Main
Stream closed: EOF

svenska-primekey commented 1 year ago

How did you create the ejbca-client-cert tls secret? That error looks like it's from the ejbca-client-cert secret

Are you following this tutorial? https://doc.primekey.com/ejbca/tutorials-and-guides/tutorial-deploy-ejbca-container-to-issue-certificates-to-an-istio-service-mesh

peteroneilljr commented 1 year ago

I'm creating it with this command, which I got from the Istio tutorial. kubectl -n ejbca-csr-signer create secret tls ejbca-client-cert --cert=./client.pem --key=./client.key

If this secret is missing the container will fail to boot up, so I'm sure it's there.

peteroneilljr commented 1 year ago

This is what the secret looks like

And this is what my helm values are.

replicaCount: 1

imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""

ejbca:
  image:
    repository: keyfactor/ejbca-k8s-csr-signer
    pullPolicy: IfNotPresent
    tag: 1.0.3
  useEST: false
  healthcheckPort: 5354
  defaultESTAlias: ""
  defaultCertificateProfileName: "ShortLivedProfile"
  defaultEndEntityProfileName: "ShortLivedProfile"
  defaultCertificateAuthorityName: "k8sRootCA"
  credsSecretName: ejbca-credentials
  clientCertSecretName: ejbca-client-cert
  caCertConfigmapName: ejbca-ca-cert
  chainDepth: 4
  signerNames:
    - "keyfactor.com/*"
  vault:
    enabled: false
    roleName: ejbca-cred
    vaultSecretPath: secret/data/ejbca

serviceAccount:
  # Specifies whether a service account should be created
  create: true
  # Annotations to add to the service account
  annotations: {}
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name: "ejbca-k8s"

podSecurityContext: {}
  # fsGroup: 2000

securityContext: {}
  # capabilities:
  #   drop:
  #   - ALL
  # readOnlyRootFilesystem: true
  # runAsNonRoot: true
  # runAsUser: 1000

resources: {}
  # We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  # limits:
  #   cpu: 100m
  #   memory: 128Mi
  # requests:
  #   cpu: 100m
  #   memory: 128Mi

autoscaling:
  enabled: true
  minReplicas: 1
  maxReplicas: 100
  targetCPUUtilizationPercentage: 80
  # targetMemoryUtilizationPercentage: 80

nodeSelector: {}

tolerations: []

affinity: {}

peteroneilljr commented 1 year ago

From running YAML, the secret appears to be mounted correctly.

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: 2023-07-31T16:25:16Z
  generateName: ejbca-csr-signer-helm-7b8fb7c955-
  labels:
    app.kubernetes.io/instance: ejbca-csr-signer-helm
    app.kubernetes.io/name: ejbca-csr-signer
    pod-template-hash: 7b8fb7c955
  name: ejbca-csr-signer-helm-7b8fb7c955-7lpt5
  namespace: ejbca-csr-signer
  ownerReferences:
    - apiVersion: apps/v1
      blockOwnerDeletion: true
      controller: true
      kind: ReplicaSet
      name: ejbca-csr-signer-helm-7b8fb7c955
      uid: 8c6c7d5d-af41-4c74-bd2a-32caa1ef3b63
  resourceVersion: "24936"
  uid: 8249ab0f-1f9b-4417-ba95-d57c9f8ef430
spec:
  volumes:
    - name: ejbca-credentials
      secret:
        secretName: ejbca-credentials
        defaultMode: 420
    - name: ejbca-config
      configMap:
        name: ejbca-config
        defaultMode: 420
    - name: ejbca-ca-cert
      configMap:
        name: ejbca-ca-cert
        defaultMode: 420
    - name: ejbca-client-cert
      secret:
        secretName: ejbca-client-cert
        defaultMode: 420
    - name: kube-api-access-7kvcz
      projected:
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              name: kube-root-ca.crt
              items:
                - key: ca.crt
                  path: ca.crt
          - downwardAPI:
              items:
                - path: namespace
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
        defaultMode: 420
  containers:
    - name: ejbca-csr-signer
      image: keyfactor/ejbca-k8s-csr-signer:1.0.3
      ports:
        - name: http
          containerPort: 5354
          protocol: TCP
      env:
        - name: SERVICE_NAME
          value: ejbca-csr-signer-helm
        - name: CLIENT_CERT_DIR
          value: /clientcert/
        - name: CREDENTIALS_FILE_DIR
          value: credentials/
      resources: {}
      volumeMounts:
        - name: ejbca-credentials
          mountPath: /credentials
        - name: ejbca-config
          mountPath: /config
        - name: ejbca-ca-cert
          mountPath: /etc/ssl/certs/
        - name: ejbca-client-cert
          mountPath: /clientcert
        - name: kube-api-access-7kvcz
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      livenessProbe:
        httpGet:
          path: /healthz
          port: 5354

peteroneilljr commented 1 year ago

Could this be because I'm running the community edition of EJBCA?

peteroneilljr commented 1 year ago

Looking at the logs again it does say that the client cert is there.

2023-07-31 16:25:30.905Z INFO[2023-07-31T16:25:30Z] Looking in /clientcert/ for client certificates  scope=Credential
2023-07-31 16:25:30.905Z WARN[2023-07-31T16:25:30Z] read /clientcert/..data: is a directory       scope=Credential
2023-07-31 16:25:30.905Z INFO[2023-07-31T16:25:30Z] tls.crt exists and contains 1589 bytes        scope=Credential
2023-07-31 16:25:30.905Z INFO[2023-07-31T16:25:30Z] tls.key exists and contains 1703 bytes        scope=Credential

Is it supposed to convert the tls.crt from the secret into the certkey.pem?

svenska-primekey commented 1 year ago

Could this be because I'm running the community edition of EJBCA?

No I tested this integration with ejbca-ce. All the keyfactor community videos were recorded with the CE version.

peteroneilljr commented 1 year ago

Got it, if you have any ideas on how to troubleshoot it send them my way.

svenska-primekey commented 1 year ago

Use the previous release from September, 1.0.2. It looks like the code changed in the pkg/credential/credential.go file and the helm chart file deployment.yaml. Going back a version should get you working for the time being.

peteroneilljr commented 1 year ago

Same issue with 1.0.2 and 1.0.1 says not found.

svenska-primekey commented 1 year ago

The certs are pem and no password on the private key? The problem seems to be with the secret and/or your cert files. I just went through the steps again and it worked for me.

[user@microk8-01 ejbca-csr-signer]$ kubectl -n ejbca-csr-signer logs $(kubectl get pods --template '{{range .items}}{{.metadata.name}}{{end}}' -n ejbca-csr-signer) -f
INFO[2023-08-02T16:55:14Z] Getting configuration from ./config/config.yaml  scope=Config
TRAC[2023-08-02T16:55:14Z] ./config/config.yaml exists and contains 211 bytes:
 useEST: false
defaultESTAlias:
defaultCertificateProfileName: ShortLivedProfile
defaultEndEntityProfileName: ShortLivedProfile
defaultCertificateAuthorityName: MyPKISubCA-G1
healthcheckPort: 5354
chainDepth: 4  scope=Config
INFO[2023-08-02T16:55:14Z] Successfully retrieved configuration:
 &config.ServerConfig{HealthCheckPort:"5354", DefaultCertificateProfileName:"ShortLivedProfile", DefaultEndEntityProfileName:"ShortLivedProfile", DefaultCertificateAuthorityName:"MyPKISubCA-G1", UseEST:false, DefaultESTAlias:"", ChainDepth:4}  scope=Config
INFO[2023-08-02T16:55:14Z] Getting credentials from credentials.yaml     scope=Credential
TRAC[2023-08-02T16:55:14Z] credentials.yaml exists and contains 452 bytes  scope=Credential
INFO[2023-08-02T16:55:14Z] Successfully retrieved credentials.           scope=Credential
INFO[2023-08-02T16:55:14Z] Looking in /clientcert/ for client certificates  scope=Credential
WARN[2023-08-02T16:55:14Z] read /clientcert/..data: is a directory       scope=Credential
INFO[2023-08-02T16:55:14Z] tls.crt exists and contains 1602 bytes        scope=Credential
INFO[2023-08-02T16:55:14Z] tls.key exists and contains 1704 bytes        scope=Credential
INFO[2023-08-02T16:55:14Z] Successfully retrieved client certificate     scope=Credential
DEBU[2023-08-02T16:55:14Z] Creating EJBCA client (useEST=false)          scope=Main
2023/08/02 16:55:14 [INFO] Building new EJBCA client
2023/08/02 16:55:14 [TRACE] Reading client certificate from certkey.pem
2023/08/02 16:55:14 [TRACE] Found CERTIFICATE
2023/08/02 16:55:14 [TRACE] Private key is not protected
2023/08/02 16:55:14 [TRACE] Found PRIVATE KEY
2023/08/02 16:55:14 [DEBUG] Found client certificate and private key
2023/08/02 16:55:14 [TRACE] System Cert Pool contains 1 certificates
2023/08/02 16:55:14 0:
0B10U

     ManagementCA10U

Keyfactor Community1
                    0   USE
2023/08/02 16:55:14 [INFO] Getting EJBCA V1 Certificate Endpoint Status
2023/08/02 16:55:14 [INFO] Preparing a GET request to path 'https://ejbca-internal.ejbca-k8s/ejbca/ejbca-rest-api/v1/certificate/status'
2023/08/02 16:55:14 [DEBUG] GET succeeded with response code 200
2023/08/02 16:55:14 [INFO] Connected to instance EJBCA 7.12.0 Enterprise (a117112306c180b66840d68c7fa8ada79cb296c6) with status OK
TRAC[2023-08-02T16:55:14Z] Got kubernetes config in cluster: &rest.Config{Host:"https://10.152.183.1:443", APIPath:"", ContentConfig:rest.ContentConfig{AcceptContentTypes:"", ContentType:"", GroupVersion:(*schema.GroupVersion)(nil), NegotiatedSerializer:runtime.NegotiatedSerializer(nil)}, Username:"", Password:"", BearerToken:"--- REDACTED ---", BearerTokenFile:"/var/run/secrets/kubernetes.io/serviceaccount/token", Impersonate:rest.ImpersonationConfig{UserName:"", UID:"", Groups:[]string(nil), Extra:map[string][]string(nil)}, AuthProvider:<nil>, AuthConfigPersister:rest.AuthProviderConfigPersister(nil), ExecProvider:<nil>, TLSClientConfig:rest.sanitizedTLSClientConfig{Insecure:false, ServerName:"", CertFile:"", KeyFile:"", CAFile:"/var/run/secrets/kubernetes.io/serviceaccount/ca.crt", CertData:[]uint8(nil), KeyData:[]uint8(nil), CAData:[]uint8(nil), NextProtos:[]string(nil)}, UserAgent:"", DisableCompression:false, Transport:http.RoundTripper(nil), WrapTransport:(transport.WrapperFunc)(nil), QPS:0, Burst:0, RateLimiter:flowcontrol.RateLimiter(nil), WarningHandler:rest.WarningHandler(nil), Timeout:0, Dial:(func(context.Context, string, string) (net.Conn, error))(nil), Proxy:(func(*http.Request) (*url.URL, error))(nil)}  scope=Main
INFO[2023-08-02T16:55:14Z] Created in-cluster Kubernetes client          scope=Main
INFO[2023-08-02T16:55:14Z] Creating new Certificate Controller called 'ejbca-csr-signer-helm'  scope=CertificateSigner
INFO[2023-08-02T16:55:14Z] Starting health check service at: [::]:5354   scope=CertificateSigner-Handler
TRAC[2023-08-02T16:55:14Z] Finished configuring Certificate Controller called 'ejbca-csr-signer-helm'  scope=CertificateSigner
INFO[2023-08-02T16:55:14Z] Starting certificate controller "ejbca-csr-signer-helm"  scope=CertificateSigner
I0802 16:55:14.157820       1 shared_informer.go:255] Waiting for caches to sync for certificate-ejbca-csr-signer-helm
I0802 16:55:14.258797       1 shared_informer.go:262] Caches are synced for certificate-ejbca-csr-signer-helm
INFO[2023-08-02T16:55:14Z] Certificate controller started for ejbca-csr-signer-helm  scope=CertificateSigner

svenska-primekey commented 1 year ago

Can you also confirm you have a private key in the file and didn't accidentally put the certificate in there too.

Keyfactor / ejbca-k8s-csr-signer

certKey.pem is missing #12