Importing outside CA certs: documentation and validation code issues

[mstrent]

After much experimenting I have realized that importing your own CA to the Java truststore does not seem to work as documented. The README on this repo is a bit ambiguous. The method in the comments in the example config in this repo, and the recently updated one in the Forgeops repo doesn't work: https://github.com/ForgeRock/forgeops/commit/b8895d9d3b92913da6ddf073af95301d0abe9ac0#diff-0093e182746c768dfa92629b6dca2e99b5c872f66d4f7884226d896b527b76e4

The comments indicate simply creating a K8s secret and referencing the object/key name, like so: kubectl create secret generic org-ca-certs --from-file=org-intca-01.pem=./org-intca-01.pem --from-file=org-rootca-01.pem=./org-rootca-01.pem

$ kubectl get secret org-ca-certs -o json
{
    "apiVersion": "v1",
    "data": {
        "org-intca-01": "xxxxxxx",
        "org-rootca-01": "xxxxxxxx"
    },
    "kind": "Secret",
    "metadata": {
        "creationTimestamp": "2021-08-17T16:28:18Z",
        "labels": {
            "managed-by-secret-agent": "true",
            "secret-agent-configuration-name": "forgerock-sac"
        },
        "name": "org-ca-certs",
        "namespace": "forgerock",
        "ownerReferences": [
            {
                "apiVersion": "secret-agent.secrets.forgerock.io/v1alpha1",
                "blockOwnerDeletion": true,
                "controller": true,
                "kind": "SecretAgentConfiguration",
                "name": "forgerock-sac",
                "uid": "628cfcc1-73b6-4d0e-bcc3-1fb16bbe35f9"
            }
        ],
        "resourceVersion": "58806090",
        "uid": "50eb78a1-0680-4cb3-b43a-cba89152465e"
    },
    "type": "Opaque"
}

  - name: truststore-pem
    keys:
      - name: cacerts
        type: truststore
        spec:
          pemFormat: true
          truststoreImportPaths: ["platform-ca/ca", "org-ca-certs/org-intca-01", "org-ca-certs/org-rootca-01"]

However, when trying to create or patch the SAC to mention those extra CA certs, the following error is returned every time: Error from server (Key: 'SecretAgentConfigurationSpec.Secrets.truststore-pem.cacerts' Error:Field validation for 'Secrets.truststore-pem.cacerts' failed on the 'truststoreImportPathsNotFound' tag): error when creating "STDIN": admission webhook "vsecretagentconfiguration.kb.io" denied the request: Key: 'SecretAgentConfigurationSpec.Secrets.truststore-pem.cacerts' Error:Field validation for 'Secrets.truststore-pem.cacerts' failed on the 'truststoreImportPathsNotFound' tag

[mstrent]

What actually works:

Creating the secrets from the cert files with kubectl as documented/specified above PLUS defining "type: literal" secrets without ".pem" prefixes on the key names in the SAC.

OR if you wanted to skip the manual creation of secrets step and let Secret Agent create the secrets from the "type: literal" "value: |" block PEM data you specified in the SAC, you actually have to define each key twice in the SAC, once without the ".pem" suffix on the key name and once with.

This is because there's an issue with the naming. The truststoreImportPaths validation code looks for secret/key names to be defined in the SAC without the ".pem" suffix, but the code that creates secrets doesn't add a ".pem" suffix to the K8s key names, and the code that grabs the k8s secrets and merges it into the PEM format truststore DOES look for the .pem suffix.

[maxres-fr]

Looks like when I added that notes in ForgeOps I tested without the validating webhook enabled.

That validation can probably be removed it's older and probably isn't required atleast for the truststore type.

If you need an immediate work around disable the webhooks by adding an environment variable to the controller container. ENABLE_WEBHOOKS=true

[maxres-fr]

fixed in PR#205

❯ mkcert issue204
Using the local CA at "/home/max/.local/share/mkcert" ✨

Created a new certificate valid for the following names 📜
 - "issue204"

The certificate is at "./issue204.pem" and the key at "./issue204-key.pem" ✅

❯ kubectl create secret generic external-trust --from-file=exttrust.pem=./issue204.pem
❯ cat external-trust-test.yaml
---
apiVersion: secret-agent.secrets.forgerock.io/v1alpha1
kind: SecretAgentConfiguration
metadata:
  name: forgerock-sac

# CRD settings
spec:
  appConfig:
    createKubernetesObjects: true

  secrets:
    - name: external-pem
      keys:
        - name: cacerts
          type: truststore
          spec:
            pemFormat: true
            truststoreImportPaths: ["external-trust/exttrust"]

(⎈ |k3d-k3s-default:default)
projects/secret-agent [gh-issue-203●]
❯ k apply -f external-trust-test.yaml
secretagentconfiguration.secret-agent.secrets.forgerock.io/forgerock-sac configured

(⎈ |k3d-k3s-default:default)
projects/secret-agent [gh-issue-203●]
❯ kgp secrets -w
Error from server (NotFound): pods "secrets" not found

(⎈ |k3d-k3s-default:default)
projects/secret-agent [gh-issue-203●]
❯ k get secrets -w
NAME                  TYPE                                  DATA   AGE
default-token-gcmqj   kubernetes.io/service-account-token   3      12d
external-trust        Opaque                                1      66m
external-pem          Opaque                                1      14s
^C%
(⎈ |k3d-k3s-default:default)
projects/secret-agent [gh-issue-203●]
❯ k get secrets external-trust -o json | jq '.data'
{
  "exttrust.pem": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVORENDQXB5Z0F3SUJBZ0lRVU9aNVV5S2dZNW1BYUJzT1FjYnRqekFOQmdrcWhraUc5dzBCQVFzRkFEQngKTVI0d0hBWURWUVFLRXhWdGEyTmxjblFnWkdWMlpXeHZjRzFsYm5RZ1EwRXhJekFoQmdOVkJBc01HbTFoZUVCbQpiM0puWlcxaGVDQW9UV0Y0SUZKbGMyNXBZMnNwTVNvd0tBWURWUVFERENGdGEyTmxjblFnYldGNFFHWnZjbWRsCmJXRjRJQ2hOWVhnZ1VtVnpibWxqYXlrd0hoY05NVGt3TmpBeE1EQXdNREF3V2hjTk16RXdPREU0TVRjeE16UTAKV2pCT01TY3dKUVlEVlFRS0V4NXRhMk5sY25RZ1pHVjJaV3h2Y0cxbGJuUWdZMlZ5ZEdsbWFXTmhkR1V4SXpBaApCZ05WQkFzTUdtMWhlRUJtYjNKblpXMWhlQ0FvVFdGNElGSmxjMjVwWTJzcE1JSUJJakFOQmdrcWhraUc5dzBCCkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQXhXMU1jMFZRZUJoUHRQZzZyS0pWT0kvOXFweDRaam4wZjA0Mmpia0EKMFR3Q2U2TDBXd0c2QmVaV2tvSE5GOGRhOEIvd0RJbmRrbWV0UWZMQ3UyNnhVT1B0MUkzdVA4UGFrYktXYXVWdApZZWhIanVxLzVNOFcvcFYvWkd4YUNFb3pjOWVIRWhXbEdKVFVDMEszeHlBSllEdWxzb1hmSUNpbHA0QjhjUFo5CnllbTFUbWQ5VDFtVEwyQ3ZiR1NXMkF6QTBta1NlQkpTekErV2UyczJDRHFyS3VoYlo5NVVQQ0dVSkNlOC9rSFcKZTZVV1ZLdmJJelBYS0tsQndIb3YwcE94Qm9UMU5kVU5WSHZxaHlyUXlaOGcreFZnQ3Z4ck01ZFBkZ29GWUxndQpCVG43bHA2UitrM1RMc1dHK0dYWmM1VFhtT041MnU4aE1STk5WTkNlSTFZRW5RSURBUUFCbzJzd2FUQU9CZ05WCkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0V3REFZRFZSMFRBUUgvQkFJd0FEQWYKQmdOVkhTTUVHREFXZ0JUNlpMRDZhNC9rR0lWNytpQVVZY0VTbjVqWnZEQVRCZ05WSFJFRUREQUtnZ2hwYzNOMQpaVEl3TkRBTkJna3Foa2lHOXcwQkFRc0ZBQU9DQVlFQWRhUzBHSE5ycE02R0ZRN2pWWFVBYzhraDZDTGxCR3FjCnlCNjkyZVJkR3hGYVprYmhrQW94V1h1RXluSVMzTWtqWEY3OVNBUiszeXAwZngvZkt4UEpqUE5rYTVGMUVaOW0KV25lam9keWN3M0ZraGhYZkRFMXdoeXNINzY2VWV0TWg0bkJtUnpVREowVkt4TU54Uk5NVEdWWXI5Rm1McTRFdQpLeFJyQXlWNGtxYllyTDJyQ0FNSnNoTVNpNlBYWjZ6RGNkK2ZTYmJadlZkKzhqRkRjdzYyelIxbThyRTR0YTBNCkM1U0MxUHBIZ2FsVVdTZVJzaitnS1VVZWpkbWMrNlMwb2toc1B0UHNSNjVlalg2Mjc1RDdRNFRGVkVZSE9GajAKaGkwRExYci9XQ0tzYkcxLzNhdXBwdlFxYXUxMXpNVnBEUlorWG5FMWIrZHNLdGpJSlo0YzUrcmgrZ0NzZEJzQQoyY0hVMm8ySnNzbHFOMUJXeEl0VE9tQ1VUMWRuZURqcmd1Vm5sZU85dTJOajZ1S3lYOG9uSkpEMXJIRG5nbENxClNIKzByZHBSczdGK09iNFdIZ1RtOWt5NzlNcTEyUWU1V0RKSWV4b0c5cEN5UUNFc0FUSklVWDVGcmtCQ0p5SjkKU2lzMUVOaUk2RThrcFlmcVQ2SkxCZ0VpWUFpVzNQb2kKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo="
}

(⎈ |k3d-k3s-default:default)
projects/secret-agent [gh-issue-203●]
❯ k get secrets external-pem -o json | jq '.data'
{
  "cacerts": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVORENDQXB5Z0F3SUJBZ0lRVU9aNVV5S2dZNW1BYUJzT1FjYnRqekFOQmdrcWhraUc5dzBCQVFzRkFEQngKTVI0d0hBWURWUVFLRXhWdGEyTmxjblFnWkdWMlpXeHZjRzFsYm5RZ1EwRXhJekFoQmdOVkJBc01HbTFoZUVCbQpiM0puWlcxaGVDQW9UV0Y0SUZKbGMyNXBZMnNwTVNvd0tBWURWUVFERENGdGEyTmxjblFnYldGNFFHWnZjbWRsCmJXRjRJQ2hOWVhnZ1VtVnpibWxqYXlrd0hoY05NVGt3TmpBeE1EQXdNREF3V2hjTk16RXdPREU0TVRjeE16UTAKV2pCT01TY3dKUVlEVlFRS0V4NXRhMk5sY25RZ1pHVjJaV3h2Y0cxbGJuUWdZMlZ5ZEdsbWFXTmhkR1V4SXpBaApCZ05WQkFzTUdtMWhlRUJtYjNKblpXMWhlQ0FvVFdGNElGSmxjMjVwWTJzcE1JSUJJakFOQmdrcWhraUc5dzBCCkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQXhXMU1jMFZRZUJoUHRQZzZyS0pWT0kvOXFweDRaam4wZjA0Mmpia0EKMFR3Q2U2TDBXd0c2QmVaV2tvSE5GOGRhOEIvd0RJbmRrbWV0UWZMQ3UyNnhVT1B0MUkzdVA4UGFrYktXYXVWdApZZWhIanVxLzVNOFcvcFYvWkd4YUNFb3pjOWVIRWhXbEdKVFVDMEszeHlBSllEdWxzb1hmSUNpbHA0QjhjUFo5CnllbTFUbWQ5VDFtVEwyQ3ZiR1NXMkF6QTBta1NlQkpTekErV2UyczJDRHFyS3VoYlo5NVVQQ0dVSkNlOC9rSFcKZTZVV1ZLdmJJelBYS0tsQndIb3YwcE94Qm9UMU5kVU5WSHZxaHlyUXlaOGcreFZnQ3Z4ck01ZFBkZ29GWUxndQpCVG43bHA2UitrM1RMc1dHK0dYWmM1VFhtT041MnU4aE1STk5WTkNlSTFZRW5RSURBUUFCbzJzd2FUQU9CZ05WCkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0V3REFZRFZSMFRBUUgvQkFJd0FEQWYKQmdOVkhTTUVHREFXZ0JUNlpMRDZhNC9rR0lWNytpQVVZY0VTbjVqWnZEQVRCZ05WSFJFRUREQUtnZ2hwYzNOMQpaVEl3TkRBTkJna3Foa2lHOXcwQkFRc0ZBQU9DQVlFQWRhUzBHSE5ycE02R0ZRN2pWWFVBYzhraDZDTGxCR3FjCnlCNjkyZVJkR3hGYVprYmhrQW94V1h1RXluSVMzTWtqWEY3OVNBUiszeXAwZngvZkt4UEpqUE5rYTVGMUVaOW0KV25lam9keWN3M0ZraGhYZkRFMXdoeXNINzY2VWV0TWg0bkJtUnpVREowVkt4TU54Uk5NVEdWWXI5Rm1McTRFdQpLeFJyQXlWNGtxYllyTDJyQ0FNSnNoTVNpNlBYWjZ6RGNkK2ZTYmJadlZkKzhqRkRjdzYyelIxbThyRTR0YTBNCkM1U0MxUHBIZ2FsVVdTZVJzaitnS1VVZWpkbWMrNlMwb2toc1B0UHNSNjVlalg2Mjc1RDdRNFRGVkVZSE9GajAKaGkwRExYci9XQ0tzYkcxLzNhdXBwdlFxYXUxMXpNVnBEUlorWG5FMWIrZHNLdGpJSlo0YzUrcmgrZ0NzZEJzQQoyY0hVMm8ySnNzbHFOMUJXeEl0VE9tQ1VUMWRuZURqcmd1Vm5sZU85dTJOajZ1S3lYOG9uSkpEMXJIRG5nbENxClNIKzByZHBSczdGK09iNFdIZ1RtOWt5NzlNcTEyUWU1V0RKSWV4b0c5cEN5UUNFc0FUSklVWDVGcmtCQ0p5SjkKU2lzMUVOaUk2RThrcFlmcVQ2SkxCZ0VpWUFpVzNQb2kKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo="
}

[maxres-fr]

@mstrent note in your initial description of the issue your get secret output doesn't match the kubectl create secret input. So maybe that object is from you experiments, or your shell did something with .pem in your keyname when you ran the create.

[mstrent]

@mstrent note in your initial description of the issue your get secret output doesn't match the kubectl create secret input. So maybe that object is from you experiments, or your shell did something with .pem in your keyname when you ran the create.

Good catch. Probably copy & pasted the wrong iteration of experiments, or made a mistake in my obfuscation of hostnames. But I definitely tried the ".pem" suffix key name. :)

Anyway, thanks for your quick attention to this! My hack of defining both .pem suffix and non-.pem suffix key names got me going just fine, but I'm a huge fan of promptly backing out resolved workarounds!