Credentials for key vault are not detected on the initial run

mkrzywanski commented 2 years ago

Version report

Jenkins and plugins versions report:

Jenkins: 2.317
OS: Linux - 5.4.0-1062-azure
---
ace-editor:1.1
apache-httpcomponents-client-4-api:4.5.13-1.0
azure-credentials:198.vf9c2fdfde55c
azure-keyvault:131.v867845ef6ae9
azure-sdk:61.v6a8af1f5f5b6
azure-vm-agents:797.v31f530348574
bootstrap4-api:4.6.0-3
bootstrap5-api:5.1.1-1
bouncycastle-api:2.25
branch-api:2.7.0
build-timestamp:1.0.3
build-user-vars-plugin:1.8
caffeine-api:2.9.2-29.v717aac953ff3
checks-api:1.7.2
cloud-stats:0.27
cloudbees-folder:6.16
command-launcher:1.6
configuration-as-code:1.54
credentials:2.6.2
credentials-binding:1.27
display-url-api:2.3.5
durable-task:1.39
echarts-api:5.2.1-2
extended-read-permission:3.2
font-awesome-api:5.15.4-1
git:4.10.0
git-client:3.10.0
git-server:1.10
handlebars:3.0.8
jackson2-api:2.13.0-230.v59243c64b0a5
jdk-tool:1.5
jquery-detached:1.2.1
jquery3-api:3.6.0-2
jsch:0.1.55.2
junit:1.53
ldap:2.7
lockable-resources:2.12
mailer:1.34
matrix-auth:2.6.8
matrix-project:1.19
momentjs:1.1.1
pipeline-build-step:2.15
pipeline-graph-analysis:1.11
pipeline-input-step:2.12
pipeline-milestone-step:1.3.2
pipeline-model-api:1.9.2
pipeline-model-declarative-agent:1.1.1
pipeline-model-definition:1.9.2
pipeline-model-extensions:1.9.2
pipeline-rest-api:2.19
pipeline-stage-step:2.5
pipeline-stage-tags-metadata:1.9.2
pipeline-stage-view:2.19
plain-credentials:1.7
plugin-util-api:2.5.1
popper-api:1.16.1-2
popper2-api:2.10.2-1
resource-disposer:0.16
role-strategy:3.2.0
scm-api:2.6.5
script-security:1.78
snakeyaml-api:1.29.1
ssh-credentials:1.19
sshd:3.1.0
structs:1.23
throttle-concurrents:2.4
timestamper:1.13
trilead-api:1.0.13
workflow-aggregator:2.6
workflow-api:2.47
workflow-basic-steps:2.24
workflow-cps:2633.v6baeedc13805
workflow-cps-global-lib:545.v7b28cce323cf
workflow-durable-task-step:2.40
workflow-job:2.42
workflow-multibranch:2.26
workflow-scm-step:2.13
workflow-step-api:2.24
workflow-support:3.8
ws-cleanup:0.39

What Operating System are you using (both controller, and any agents involved in the problem)?

Jenkins docker image `jenkins/jenkins:2.317-jdk8`

Reproduction steps

I have a Jenkins as a code configuration. I want to configure key vault access and a Service Principal that is used to read secrets from it. The configuration looks similiar to it :

- credentials:
      - azure:
          azureEnvironmentName: "Azure"
          clientId: "..."
          clientSecret: "..."
          description: "keyvault-access"
          id: "keyvault-access"
          scope: GLOBAL
          subscriptionId: "..."
          tenant: "..."
...
unclassified:
  azureKeyVault:
    credentialID: "keyvault-access"
    keyVaultURL: "..."

When running fresh jenkins instance there is an error that placeholders cannot be read from the vault (for example ldap password) as there is no credentials with keyvault-access id :

No AzureKeyVault credentials found, skipping jcasc secret resolution

However when I restart the container with same configurations and same volume - placeholders are resolved to variables from vault.

Results

Expected result: I expect that credentials are available before key vault plugin tries to use them for the first time.

Actual result: Credentials for Service Principal defined in Jenkins Configurations as Code seem to be created after azure keyvault plugin configuration is performed by JaaC plugin. I expect that credentials from JaaC are configured before JaaC keyvault tries to use them.

timja commented 2 years ago

There’s system properties available for this: https://github.com/jenkinsci/azure-keyvault-plugin#via-system-properties

managed identity is recommended

mkrzywanski commented 2 years ago

Guess will have to go with the env variables solution for SP as it seems that it is expected behaviour? Managed identity is not an option as jenkins is running in docker container.

timja commented 2 years ago

Guess will have to go with the env variables solution for SP as it seems that it is expected behaviour? Managed identity is not an option as jenkins is running in docker container.

As long as it's on an Azure VM managed identity should be ok? Depending on your docker networking but I would expect it to be fine.

(we run on K8s and it works really well there)

mkrzywanski commented 2 years ago

I just checked and you are right that a container that is deployed on Azure VM can use the VM managed identity. But i thought that example that is provided in the docs should work out of the box - but it does not as described above.

hazzik commented 2 years ago

We have this issue intermittently. Started happening a few weeks ago. I had to do an emergency upgrade to UAMI. Before that was using CasC for configuring KV. The problem is that in my case jenkins refuses to start and pod stucks in crash loopback mode.

timja commented 1 year ago

Documented in https://github.com/jenkinsci/azure-keyvault-plugin/pull/140

jenkinsci / azure-keyvault-plugin