operator unable to apply new settings when Azure AD authentication is enabled

crespo-obie commented 9 months ago

Describe the bug operator is unable to reload Jenkins Pod when new values are applied if Azure AD authentication is enabled.

To Reproduce

deploy operator using helm
use values and config map to configure Azure AD
wait to deploy
verify Azure AD configuration is working
make additional changes to values
upgrade helm release with new values
see logs below for reference

Additional information

Kubernetes version: 1.28 Jenkins Operator version: 0.7.0 Jenkins Image version : jenkins/jenkins:lts (2.426.2)

Workaround : CAUTION! this will not initiate backup and will restore the last working backup

delete jenkins pod
```
kubectl delete pods jenkins-<cr-name>
```

Add error logs about the problem here (operator logs and Kubernetes events).

operator logs :

2023-12-23T03:25:04.447Z        DEBUG   controller-jenkins      Jenkins master pod is present   {"cr": "jenkins"}
2023-12-23T03:25:04.447Z        DEBUG   controller-jenkins      Jenkins master pod is ready     {"cr": "jenkins"}
2023-12-23T03:25:04.871Z        WARN    controller-jenkins      Reconcile loop failed 10 times with the same errors, giving up: couldn't poll data from Jenkins API, invalid status code returned: 500
github.com/jenkinsci/kubernetes-operator/pkg/client.newClient
    /workspace/pkg/client/jenkins.go:186
github.com/jenkinsci/kubernetes-operator/pkg/client.NewUserAndPasswordAuthorization
    /workspace/pkg/client/jenkins.go:138
github.com/jenkinsci/kubernetes-operator/pkg/configuration.(*Configuration).GetJenkinsClientFromSecret
    /workspace/pkg/configuration/configuration.go:275
github.com/jenkinsci/kubernetes-operator/pkg/configuration.(*Configuration).GetJenkinsClient

jenkins logs :

2023-12-23 03:25:04.757+0000 [id=19]    WARNING h.i.i.InstallUncaughtExceptionHandler#handleException: Caught unhan ID 25c6cda7-d196-4693-8464-4333ce08c83d
org.springframework.security.core.userdetails.UsernameNotFoundException: Cannot find user: jenkins-operator
    at com.microsoft.jenkins.azuread.AzureSecurityRealm.lambda$createSecurityComponents$4(AzureSecurityRealm.ja        at jenkins.security.ImpersonatingUserDetailsService2.loadUserByUsername(ImpersonatingUserDetailsService2.ja        at hudson.model.User.getUserDetailsForImpersonation2(User.java:407)
    at jenkins.security.BasicHeaderApiTokenAuthenticator.authenticate2(BasicHeaderApiTokenAuthenticator.java:36Caused: javax.servlet.ServletException
    at jenkins.security.BasicHeaderApiTokenAuthenticator.authenticate2(BasicHeaderApiTokenAuthenticator.java:44        at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:83)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersis7)

lhupfeldt commented 8 months ago

Would it be possible for you to share your configuration for Azure AD authentication? Are you running the operator and Jenkins in different namespaces? I'm currently struggling to get GitHub authentication working, we may want to move to Azure AD authentication later, but since both are external (not Jenkins built in) authentication, your configuration may help me understand why GH auth is not working.

crespo-obie commented 8 months ago

For AzureAD Authentication:

follow the instructions as provided in the plugin documentation - https://plugins.jenkins.io/azure-ad/

Configuration is done as follows :

using Helm or Jenkins CR :

add the following values

plugins:
  - name: azure-ad
  - version: "449.v92b_39a_d8e523"

on user CasC :

unclassified:
  location:
    url: <callback-FQDN-configured-in-AzureAD-app-registration>
jenkins:
  authorizationStrategy:
    azureAdMatrix:
      entries:
      - group:
          name: "jenkins-administrators"
          permissions:
          - "Overall/Administer"
          - "Overall/Read"
      - group:
          name: "jenkins-users"
          permissions:
          - "Agent/Build"
  securityRealm:
    azure:
      clientId: "${AZURE_AD_CLIENT_ID}"
      clientSecret: "${AZURE_AD_CLIENT_SECRET}"
      tenant: "${AZURE_AD_TENANT_ID}"

Introduce AzureAD App Registration credentials via kubernetes secrets.

Detailed instructions on User customization: https://jenkinsci.github.io/kubernetes-operator/docs/getting-started/latest/customizing-jenkins/

Possible issue you may encounter with external authentication is the callback URLs, if you are running jenkins behind reverse proxy. Ensure that the required redirection is enabled.

inspect logs on both operator and jenkins pods to identify issues as follows :

operator logs :

kubectl logs --tail 100 -f <operator-pod-name>

jenkins logs :

kubectl logs --tail 100 -f jenkins-<cr-name>

I was able to resolve the impersonation issue by implementing the following :

creating an operator account within the AzureAD.
modify the operator user credentials in kubernetes secrets

kubectl edit secret jenkins-operator-credentials-<cr-name>

lhupfeldt commented 8 months ago

Thank you for this. I don't see anything which would explain our problem, but the information may be useful for us late.

brokenpip3 commented 8 months ago

@crespo-obie thanks for reporting back the solution, I'm closing since this is a well know issue see: https://github.com/jenkinsci/kubernetes-operator/issues/963#issuecomment-1925683691

if someone wants to create a PR for documenting this will be great, thanks.

jenkinsci / kubernetes-operator

operator unable to apply new settings when Azure AD authentication is enabled #949

Possible issue you may encounter with external authentication is the callback URLs, if you are running jenkins behind reverse proxy. Ensure that the required redirection is enabled.