Jenkins not utilising IRSA

[Trozz]

Describe the bug After upgrading from chart version 2.13.1 to 3.1.9 Jenkins no longer utilises IRSA

User: arn:aws:sts::XXXX:assumed-role/cluster-XXXX/i-XXXX is being used instead of the role defined via the Service Account annotation.

    serviceAccount:
      create: true
      # The name of the service account is autogenerated by default
      name: server
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::XXX:role/eks/mgmt/cluster-XXX-external-jenkins-master

    serviceAccountAgent:
      # Specifies whether a ServiceAccount should be created
      create: true
      # The name of the ServiceAccount to use.
      # If not set and create is true, a name is generated using the fullname template
      name: agent
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::XXX:role/eks/mgmt/cluster-XXX-external-jenkins-master

Helm Version:

❯ helm version
version.BuildInfo{Version:"v3.5.2", GitCommit:"167aac70832d3a384f65f9745335e9fb40169dc2", GitTreeState:"dirty", GoVersion:"go1.15.7"}

Kubernetes Version:

❯ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-14T05:15:04Z", GoVersion:"go1.15.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.9-eks-d1db3c", GitCommit:"d1db3c46e55f95d6a7d3e5578689371318f95ff9", GitTreeState:"clean", BuildDate:"2020-10-20T22:18:07Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

Which version of the chart: 3.1.9

What happened: Jenkins is not using the Service Account Role for interacting with AWS

What you expected to happen: Jenkins to use the Service Account Role for interacting with AWS

How to reproduce it (as minimally and precisely as possible): Deploy Helm chart to EKS

values.yaml

controller:
  hostNetworking: false
  adminUser: "admin"
  admin:
    userKey: jenkins-admin-user
    passwordKey: jenkins-admin-password
  jenkinsHome: "/var/jenkins_home"
  jenkinsRef: "/usr/share/jenkins/ref"
  jenkinsWar: "/usr/share/jenkins/jenkins.war"
  resources:
    requests:
      cpu: "50m"
      memory: "256Mi"
    limits:
      cpu: "2000m"
      memory: "4096Mi"

  jenkinsUrl: "XXXX"

  installPlugins:
    - kubernetes:1.29.0
    - workflow-aggregator:2.6
    - git:4.6.0
    - configuration-as-code:1.47

  additionalPlugins:
    - workflow-multibranch:2.22
    - durable-task:1.35
    - matrix-project:1.18
    - popper-api:1.16.1-1
    - resource-disposer:0.14
    - dashboard-view:2.14
    - credentials-binding:1.24
    - script-security:1.76
    - jackson2-api:2.12.1
    - workflow-scm-step:2.11
    - font-awesome-api:5.15.2-1
    - okhttp-api:3.14.9
    - blueocean-display-url:2.4.1
    - blueocean-events:1.24.4
    - readonly-parameters:1.0.0
    - plugin-util-api:1.7.1
    - git-client:3.6.0
    - blueocean-rest:1.24.4
    - trilead-api:1.0.13
    - blueocean-dashboard:1.24.4
    - display-url-api:2.3.4
    - matrix-auth:2.6.5
    - blueocean-commons:1.24.4
    - aws-global-configuration:1.6
    - authentication-tokens:1.4
    - blueocean:1.24.4
    - cloudbees-folder:6.15
    - command-launcher:1.5
    - jquery-detached:1.2.1
    - kubernetes-credentials:0.8.0
    - gitlab-api:1.0.6
    - mercurial:2.12
    - ace-editor:1.1
    - apache-httpcomponents-client-4-api:4.5.13-1.0
    - gitlab-branch-source:1.5.4
    - pipeline-model-extensions:1.8.4
    - plain-credentials:1.7
    - blueocean-jwt:1.24.4
    - pipeline-stage-view:2.19
    - lockable-resources:2.10
    - cloudbees-disk-usage-simple:0.10
    - aws-secrets-manager-credentials-provider:0.5.3
    - kubernetes-client-api:4.13.2-1
    - blueocean-git-pipeline:1.24.4
    - branch-api:2.6.2
    - blueocean-pipeline-editor:1.24.4
    - favorite:2.3.2
    - docker-workflow:1.25
    - jenkins-design-language:1.24.4
    - extended-read-permission:3.2
    - blueocean-core-js:1.24.4
    - junit:1.48
    - metrics:4.0.2.7
    - ssh-agent:1.20
    - blueocean-autofavorite:1.2.4
    - bootstrap4-api:4.6.0-1
    - pipeline-rest-api:2.19
    - blueocean-jira:1.24.4
    - handy-uri-templates-2-api:2.1.8-1.0
    - gitlab-merge-request-jenkins:2.0.0
    - htmlpublisher:1.25
    - role-strategy:3.1
    - blueocean-personalization:1.24.4
    - workflow-support:3.7
    - slack:2.45
    - blueocean-i18n:1.24.4
    - pipeline-stage-tags-metadata:1.8.4
    - pipeline-input-step:2.12
    - mailer:1.32.1
    - pipeline-model-definition:1.8.4
    - workflow-job:2.40
    - pipeline-stage-step:2.5
    - timestamper:1.11.8
    - blueocean-rest-impl:1.24.4
    - pubsub-light:1.13
    - pipeline-utility-steps:2.6.1
    - config-file-provider:3.7.0
    - ws-cleanup:0.38
    - workflow-basic-steps:2.23
    - pipeline-build-step:2.13
    - workflow-cps-global-lib:2.17
    - checks-api:1.5.0
    - workflow-durable-task-step:2.37
    - token-macro:2.13
    - jira:3.2
    - artifact-manager-s3:1.12
    - handlebars:1.1.1
    - workflow-cps:2.88
    - ssh-credentials:1.18.1
    - cloudbees-bitbucket-branch-source:2.9.7
    - bouncycastle-api:2.18
    - blueocean-github-pipeline:1.24.4
    - jdk-tool:1.4
    - login-theme:1.1
    - gitlab-logo:1.0.5
    - snakeyaml-api:1.27.0
    - pipeline-graph-analysis:1.10
    - violation-comments-to-gitlab:2.46
    - aws-java-sdk:1.11.930
    - workflow-step-api:2.23
    - structs:1.21
    - docker-commons:1.17
    - scm-api:2.6.4
    - aws-credentials:1.28
    - momentjs:1.1.1
    - pipeline-model-api:1.8.4
    - sse-gateway:1.24
    - variant:1.4
    - blueocean-web:1.24.4
    - pipeline-milestone-step:1.3.2
    - ansicolor:0.7.5
    - jaxb:2.3.0.1
    - jsch:0.1.55.2
    - job-dsl:1.77
    - gitlab-plugin:1.5.13
    - workflow-api:2.41
    - blueocean-config:1.24.4
    - active-directory:2.23
    - pipeline-aws:1.43
    - blueocean-pipeline-scm-api:1.24.4
    - simple-theme-plugin:0.6
    - blueocean-bitbucket-pipeline:1.24.4
    - jquery3-api:3.5.1-2
    - prometheus:2.0.8
    - echarts-api:4.9.0-3
    - blueocean-pipeline-api-impl:1.24.4
    - credentials:2.3.15
    - jjwt-api:0.11.2-8.82737cbfa6f5
    - gitlab-oauth:1.10

  JCasC:
    defaultConfig: true
    configScripts:
      welcome-message: |
        jenkins:
          systemMessage: |
            This Jenkins is configured and managed 'as code'
            Any chance may be lost unless you update the git repo
      security: |
        jenkins:
          securityRealm:
            activeDirectory:
              domains:
                - name: "XXX.local"
                  servers: "XXXX:3268"
                  site: ""
                  bindName: "XX"
                  bindPassword: "XX"
                  tlsConfiguration: TRUST_ALL_CERTIFICATES
              groupLookupStrategy: "RECURSIVE"
              removeIrrelevantGroups: true
              customDomain: true
              startTls: false
      role-strategy: |
        jenkins:
          authorizationStrategy:
            roleBased:
              roles:
                global:
                  - name: "admin"
                    description: "Jenkins administrators"
                    permissions:
                      - "Overall/Administer"
                    assignments:
                      - "admin"
                      - "Jenkins Admins"
                  - name: "developers"
                    description: "Users in the Developer group, allows View on Jobs"
                    permissions:
                      - "Job/Read"
                    assignments:
                      - "Jenkins Developers"
                  - name: "readonly"
                    description: "Read-only users"
                    permissions:
                      - "Overall/Read"
                      - "Job/Read"
                    assignments:
                      - "authenticated"
      awssecrets: |
        unclassified:
          awsCredentialsProvider:
            endpointConfiguration:
              serviceEndpoint: https://secretsmanager.eu-west-1.amazonaws.com
              signingRegion: eu-west-1
      theme: |
        unclassified:
          simple-theme-plugin:
            elements:
              - cssUrl:
                  url: "https://cdn.rawgit.com/djonsson/jenkins-atlassian-theme/gh-pages/theme-min.css"
              - jsUrl:
                  url: "https://cdn.rawgit.com/djonsson/jenkins-atlassian-theme/gh-pages/theme.js"
              - faviconUrl:
                  url: "https://vignette.wikia.nocookie.net/deadpool/images/6/64/Favicon.ico"

  ingress:
    enabled: true
    apiVersion: "networking.k8s.io/v1beta1"
    annotations:
      cert-manager.io/cluster-issuer: internal
      kubernetes.io/ingress.class: internal
    hostName: AAAA
    tls:
      - secretName: internal
        hosts:
          - AAAA

  # Expose Prometheus metrics
  prometheus:
    enabled: true

  httpsKeyStore:
    enable: false

agent:
  enabled: true

persistence:
  enabled: true
  size: 50Gi

networkPolicy:
  enabled: false

## Install Default RBAC roles and bindings
rbac:
  create: true
  readSecrets: false

serviceAccount:
  create: true
  # The name of the service account is autogenerated by default
  name: server
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::715997381065:role/eks/mgmt/cluster-mgmt-external-jenkins-master

serviceAccountAgent:
  # Specifies whether a ServiceAccount should be created
  create: true
  # The name of the ServiceAccount to use.
  # If not set and create is true, a name is generated using the fullname template
  name: agent
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::715997381065:role/eks/mgmt/cluster-mgmt-external-jenkins-master

helm install infra jenkins/jenkins --version 3.1.9 --values values.yaml

Anything else we need to know:

[torstenwalter]

Can you describe what should be changed in the yaml templates?

[Trozz]

at the moment not sure, I am still trying to debug why Jenkins is attempting to use the instance profile instead of the service account

[sherifabdlnaby]

@Trozz Have you figured this out? I think it is related to fsGroup, reference:

1. https://github.com/aws/aws-eks-best-practices/issues/27

But whenever I set fsGroup to anything but 1000 the container crashes because it runs with USER 1000.

Any idea ?

[Trozz]

That sounds like a possibility, could you try using this fsGroup: 65534 I'll try this evening when I am able to get online properly

[sherifabdlnaby]

~No, using fsGroup only isn't working, the pod still uses the node's instance profile~ I was using curl http://169.254.169.254/latest/meta-data/iam/info to check the Pod's effective profile which was wrong.

@Trozz Using the correct fsGroup the access token was mounted and accessible successfully. and AWS CLI v2, and Plugins using the recent AWS SDK has successfully used the IRSA role. 👍🏻

[torstenwalter]

@sherifabdlnaby Which fsGroup did you use at the end?