Jenkins helm values.yaml - failed to pass the basicSSHUserPrivateKey via terraform data.aws_secretsmanager_secret_version

avnerv commented 2 years ago

Jenkins and plugins versions report

Environment

```text Jenkins: 2.332.2 OS: Linux - 5.10.109 --- ace-editor:1.1 ansicolor:1.0.1 antisamy-markup-formatter:2.7 apache-httpcomponents-client-4-api:4.5.13-1.0 authentication-tokens:1.4 aws-credentials:191.vcb_f183ce58b_9 aws-java-sdk:1.12.163-315.v2b_716ec8e4df aws-java-sdk-cloudformation:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-codebuild:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-ec2:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-ecr:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-ecs:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-elasticbeanstalk:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-iam:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-logs:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-minimal:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-ssm:1.12.246-349.v96b_b_f7eb_a_c3c bootstrap5-api:5.1.3-7 bouncycastle-api:2.26 branch-api:2.1046.v0ca_37783ecc5 caffeine-api:2.9.3-65.v6a_47d0f4d1fe checks-api:1.7.4 cloudbees-folder:6.729.v2b_9d1a_74d673 command-launcher:84.v4a_97f2027398 configuration-as-code:1414.v878271fc496f credentials:1087.1089.v2f1b_9a_b_040e4 credentials-binding:523.vd859a_4b_122e6 datadog:4.0.0 display-url-api:2.3.6 docker-commons:1.19 docker-java-api:3.2.13-37.vf3411c9828b9 docker-plugin:1.2.7 docker-workflow:1.28 durable-task:496.va67c6f9eefa7 ec2:1.68 echarts-api:5.3.3-1 extended-choice-parameter:346.vd87693c5a_86c font-awesome-api:6.1.1-1 git:4.10.3 git-client:3.11.0 git-server:1.11 github:1.34.3 github-api:1.303-400.v35c2d8258028 github-branch-source:1598.v91207e9f9b_4a_ handlebars:3.0.8 jackson2-api:2.13.3-285.vc03c0256d517 javax-activation-api:1.2.0-3 javax-mail-api:1.6.2-6 jaxb:2.3.6-1 jdk-tool:1.0 jjwt-api:0.11.5-77.v646c772fddb_0 jnr-posix-api:3.1.7-3 job-dsl:1.79 jquery:1.12.4-1 jquery3-api:3.6.0-4 jsch:0.1.55.2 junit:1119.1121.vc43d0fc45561 kubernetes:1.31.3 kubernetes-cli:1.10.3 kubernetes-client-api:5.12.2-193.v26a_6078f65a_9 kubernetes-credentials:0.9.0 list-git-branches-parameter:0.0.11 lockable-resources:2.15 mailer:414.vcc4c33714601 matrix-auth:3.1.5 matrix-project:772.v494f19991984 metrics:4.1.6.2 mina-sshd-api-common:2.8.0-21.v493b_6b_db_22c6 mina-sshd-api-core:2.8.0-21.v493b_6b_db_22c6 momentjs:1.1.1 node-iterator-api:1.5.1 okhttp-api:4.9.3-105.vb96869f8ac3a parameterized-trigger:2.44 pipeline-aws:1.43 pipeline-build-step:2.18 pipeline-github:2.8-138.d766e30bb08b pipeline-github-lib:36.v4c01db_ca_ed16 pipeline-graph-analysis:195.v5812d95a_a_2f9 pipeline-groovy-lib:593.va_a_fc25d520e9 pipeline-input-step:449.v77f0e8b_845c4 pipeline-milestone-step:101.vd572fef9d926 pipeline-model-api:2.2097.v33db_b_de764b_e pipeline-model-definition:2.2097.v33db_b_de764b_e pipeline-model-extensions:2.2097.v33db_b_de764b_e pipeline-rest-api:2.24 pipeline-stage-step:293.v200037eefcd5 pipeline-stage-tags-metadata:2.2097.v33db_b_de764b_e pipeline-stage-view:2.24 plain-credentials:1.8 plugin-util-api:2.17.0 popper2-api:2.11.5-2 python:1.3 resource-disposer:0.19 role-strategy:3.2.0 saml:2.296.v0016349946db_ scm-api:608.vfa_f971c5a_a_e9 script-security:1175.v4b_d517d6db_f0 slack:608.v19e3b_44b_b_9ff snakeyaml-api:1.30.2-76.vc104f7ce9870 ssh-credentials:277.v95c2fec1c047 ssh-slaves:1.821.vd834f8a_c390e sshd:3.242.va_db_9da_b_26a_c3 structs:318.va_f3ccb_729b_71 terraform:1.0.10 token-macro:293.v283932a_0a_b_49 trilead-api:1.66.v49c6758b_b_360 uno-choice:2.6.1 variant:1.4 workflow-aggregator:2.6 workflow-api:1188.v0016b_4f29881 workflow-basic-steps:969.vc4ec3e4854b_f workflow-cps:2729.vea_17b_79ed57a_ workflow-cps-global-lib:588.v576c103a_ff86 workflow-durable-task-step:1174.v73a_9a_17edce0 workflow-job:1189.va_d37a_e9e4eda_ workflow-multibranch:716.vc692a_e52371b_ workflow-scm-step:400.v6b_89a_1317c9a_ workflow-step-api:625.vd896b_f445a_f8 workflow-support:833.va_1c71061486b_ ws-cleanup:0.42 xml-job-to-job-dsl:0.1.13 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

The Jenkins running on EKS cluster and deployed by Helm chart with Terraform "helm_release"

Reproduction steps

Create AWS secret manager with ssh-key for GitHub credentials as below:

Use data to call the secret:

data "aws_secretsmanager_secret_version" "github_token" {
secret_id = "github"
}

Define the SSH_PRIVATE_KEY in the values.yaml -

  credentials:
    system:
      domainCredentials:
        - credentials:
          - basicSSHUserPrivateKey:
              scope: GLOBAL
              id: github
              username: github
              description: "Credentials for GitHub repo"
              privateKeySource:
                directEntry:
                  privateKey: |
                    "${SSH_PRIVATE_KEY}"

Pass the secret to the values.yaml with terraform templatefile function -

resource "helm_release" "jenkins" {
name            = "jenkins"
namespace  = kubernetes_namespace.jenkins.metadata[0].name
repository    = "https://charts.jenkins.io"
chart            = "jenkins"
version         = var.chart_version
values = [
templatefile("${path.module}/${var.values}.yaml", {
    SSH_PRIVATE_KEY : data.aws_secretsmanager_secret_version.github_token.secret_string
  }
)
]
}

Expected Results

Helm upgrade action to be completed successfully and the ssh-key configured as expected in the Jenkins credentials.

Actual Results

Received the below error: Error: ---> error converting YAML to JSON: yaml: line 433: could not find expected ':' # Default values for jenkins. on this section:

       credentials:
         system:
           domainCredentials:
             - credentials:
               - basicSSHUserPrivateKey:
                   scope: GLOBAL
                   id: github
                   username: github
                   description: "Credentials for public-cloud-infrastructure GitHub repo"
                   privateKeySource:
                     directEntry:
                       privateKey: |
                         "-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEABCDEBG5vbmUAAAAEbm9uZQAAAAAAAAABAAABlwAAAAdzc2gtcn
NhAAAAAwEAAQAAAYEAxs6i6gjyvbqIMboLC7zQ3RB0UhGORL5idCIPhpupuQIMySUmZ1S+
-----END OPENSSH PRIVATE KEY-----"

Anything else?

I tried to use this link to fix the syntax error of the yaml with no luck

jetersen commented 2 years ago

Gotta be careful with yaml: https://yaml-multiline.info/

This should do and ensure the key saved in secret manager has an final newline.

privateKey: "${SSH_PRIVATE_KEY}"

avnerv commented 2 years ago

Gotta be careful with yaml: https://yaml-multiline.info/

This should do and ensure the key saved in secret manager has an final newline.
privateKey: "${SSH_PRIVATE_KEY}"

Thanks for the quick response.

In this case, the helm upgrade is completed correctly, but the secret is created as one line which is wrong as the ssh-key is sensitive and should be created (pass) to the values.yaml as below:

      credentials:
        system:
          domainCredentials:
            - credentials:
              - basicSSHUserPrivateKey:
                  scope: GLOBAL
                  id: github-public-cloud-infrastructure
                  username: github
                  description: "Credentials for public-cloud-infrastructure GitHub repo"
                  privateKeySource:
                    directEntry:
                      privateKey: |
                        -----BEGIN OPENSSH PRIVATE KEY-----
                        b3BlbnNzaC1rZXktdjEABCDBG5vbmUAAAAEbm9uZQAAAAAAAAABAAABlwAAAAdzc2gtcn
                        NhAAAAAwEAAQAAAYEAxs6i6gjyvbqIMboLC7zQ3RB0UhGORL5idCIPhpupuQIMySUmZ1S+
                        -----END OPENSSH PRIVATE KEY-----

jetersen commented 2 years ago

Secret manager and terraform should perserve the multiline.

So will the expanded secret

Yes you can have multiline in yaml with quotes by escaping with \n in a literal.

Also the expanded secret will also perserve newlines by using \n in the raw binary to preserve the newlines.

jetersen commented 2 years ago

Also why use terraform templating to expand secrets?

You do know you can use Jenkins and JCasC to directly connect to secret manager: https://plugins.jenkins.io/aws-secrets-manager-credentials-provider/#plugin-content-configuration-as-code-casc

Don't store secrets in helm revisions 😉

jetersen commented 2 years ago

Sorry this is the plugin you want: https://github.com/jenkinsci/aws-secrets-manager-secret-source-plugin

avnerv commented 2 years ago

Sorry, this is the plugin you want: https://github.com/jenkinsci/aws-secrets-manager-secret-source-plugin

IIRC there is no option to use IAM assume role with the plugin, right? as we manage all our secrets on different AWS account

avnerv commented 2 years ago

Secret manager and terraform should perserve the multiline.

So will the expanded secret

Yes you can have multiline in yaml with quotes by escaping with \n in a literal.

Also the expanded secret will also perserve newlines by using \n in the raw binary to preserve the newlines.

By adding the /n the issue is solved. @jetersen Thanks a lot for your help

jetersen commented 2 years ago

IIRC there is no option to use IAM assume role with the plugin, right? as we manage all our secrets on different AWS account

There is, it uses the AWS Java SDK it supports assume role. So simply make sure your Jenkins server assumes the appropriate role

wesleung commented 2 years ago

This is related to this subject. I can't seem to get credentials to work in the values.yaml file. I can get it to work if I upload to container filesystem and do a JCasC "Replace configuration source with:"

What is the proper syntax for credentials in the values.yaml? I have the below code in my values.yaml and the AuthorizedStrategy and SecurityRealms work, but not Credentials

  JCasC:
    enabled: true
    credentials:
      system:
        domainCredentials:
        - credentials:
          - string:
              description: "XXXXXXXXXXXXXXXX"
              id: "XXXXXXXXXXXXXXXXXXXX"
              scope: GLOBAL
              secret: "{XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX}"
    authorizationStrategy: |-
      roleBased:
        roles:
          global:
          - assignments:
          ...

jetersen commented 2 years ago

@wesleung not at all related, however please see docs: https://github.com/jenkinsci/configuration-as-code-plugin/blob/master/docs/features/secrets.adoc#passing-secrets-through-variables

Can also look at demos: https://github.com/jenkinsci/configuration-as-code-plugin/tree/master/demos/credentials

cyclelabs-ryanberger commented 2 years ago

Hi,

Sorry to piggy back on this but my issue is closely related to this.

I am having a similar issue when trying to pass an SSH private key (value, not from a file) from terraform building an entire Jenkins environment, and then trying to automatically configure the EC2 plugin so that environment is already hooked into AWS, all within a jenkins.yaml file, that is then consumed by a cloud-init script when building the Jenkins manager. I've tried every syntax I can think of to make sure that the syntax is right and I keep getting errors when the JCasC gets parsed and its related to me trying to create the basicSSHUserPrivateKey credential. I send over the values into my cloud-config and then use them to insert values into my JCasC config (as I create that file with cloud-config).

For some reason, I keep having issues with the private key pem that I am inserting into the credentials area in the JCasC file. I've tried a ton of diff syntaxes, but no dice. "${ec2_private_key}" , ${"$"}{decodeBase64:${ec2_private_key}}, the list goes on. @jetersen I've seen a ton of comments from you that are SORT of around this subject but not exactly related. Do you have any ideas of something I could try?

This attribute that I am using as the ec2_private_key value is a typical PEM file - so I am not sure what I am doing wrong. It must all be in the syntax of how I am trying to inject it. I've reviewed all of the documentation and all of the Git issues I could find and I am stuck.

-----BEGIN RSA PRIVATE KEY----- MIIJKgIBAAKCAgEAsnJWD4Qr3r/i6ZGnYRuP5/BhOWkvg6IwO6lfxPxvo/yBfeUl ... /X602AYAC6KYLuIsUo4x0Idp1EOD+vQeEhskmBVgkoerSd4tvXcgo6O8TUZbGdko -----END RSA PRIVATE KEY-----

jetersen commented 2 years ago

@ryanberger-az I don't think it is relevant to post here.

Seems your issue is basic yaml syntax. Which could be caused by your templating the wrong yaml.

cyclelabs-ryanberger commented 2 years ago

@jetersen Do you have any advice on how I could begin to troubleshoot this? The value that is being sent into the JCasC file is the exact value the documentation says it supports - so I am not sure why I am receiving the error message that I am when it is parsed. Sorry, I am not an expert at YAML syntaxing but I am trying to learn.

jetersen commented 2 years ago

Can you share your template? Since that should not contain secrets.

Or the section around line 64 and 65. Read the error message 😉

This might also be helpful, I think your issue is multiline: https://yaml-multiline.info/

cyclelabs-ryanberger commented 2 years ago

@jetersen I got this resolved. This was an indentation issue and I ended up having to use the terraform indent function. Your indentation count may vary, in my case I build my jenkins.yaml files using a cloud-init script, so my indentations were a little farther over because of the nested automation.

            - basicSSHUserPrivateKey:
                description: "Private key for the Windows Agents keypair"
                id: "aws-ec2-windows-agents-pk"
                privateKeySource:
                  directEntry:
                    privateKey: |
                      ${indent(22, ec2_private_key)}
                scope: GLOBAL
                username: "administrator"

jenkinsci / configuration-as-code-plugin