OIDC metadata and JWKS URLs give 404 when `aws-secrets-manager-credentials-provider` installed

iwarapter commented 1 year ago

Jenkins and plugins versions report

Environment

```text Jenkins: 2.332.1 OS: Linux - 5.10.104-linuxkit --- ace-editor:1.1 amazon-ecr:1.7 amazon-ecs:1.41 ansicolor:1.0.1 ant:1.13 antisamy-markup-formatter:2.7 apache-httpcomponents-client-4-api:4.5.13-1.0 async-http-client:1.9.40.0 authentication-tokens:1.4 aws-credentials:191.vcb_f183ce58b_9 aws-java-sdk:1.12.163-315.v2b_716ec8e4df aws-java-sdk-cloudformation:1.12.163-315.v2b_716ec8e4df aws-java-sdk-codebuild:1.12.163-315.v2b_716ec8e4df aws-java-sdk-ec2:1.12.163-315.v2b_716ec8e4df aws-java-sdk-ecr:1.12.163-315.v2b_716ec8e4df aws-java-sdk-ecs:1.12.163-315.v2b_716ec8e4df aws-java-sdk-elasticbeanstalk:1.12.163-315.v2b_716ec8e4df aws-java-sdk-iam:1.12.163-315.v2b_716ec8e4df aws-java-sdk-logs:1.12.163-315.v2b_716ec8e4df aws-java-sdk-minimal:1.12.163-315.v2b_716ec8e4df aws-java-sdk-ssm:1.12.163-315.v2b_716ec8e4df aws-lambda:0.5.10 aws-secrets-manager-credentials-provider:1.0.0 azure-ad:191.vfc8019068670 azure-sdk:106.v552de1e64d56 bitbucket-filter-project-trait:1.0 bitbucket-oauth:0.12 bitbucket-push-and-pull-request:2.8.1 block-queued-job:0.2.0 blueocean:1.25.3 blueocean-autofavorite:1.2.5 blueocean-bitbucket-pipeline:1.25.3 blueocean-commons:1.25.3 blueocean-config:1.25.3 blueocean-core-js:1.25.3 blueocean-dashboard:1.25.3 blueocean-display-url:2.4.1 blueocean-events:1.25.3 blueocean-git-pipeline:1.25.3 blueocean-github-pipeline:1.25.3 blueocean-i18n:1.25.3 blueocean-jira:1.25.3 blueocean-jwt:1.25.3 blueocean-personalization:1.25.3 blueocean-pipeline-api-impl:1.25.3 blueocean-pipeline-editor:1.25.3 blueocean-pipeline-scm-api:1.25.3 blueocean-rest:1.25.3 blueocean-rest-impl:1.25.3 blueocean-web:1.25.3 bootstrap4-api:4.6.0-3 bootstrap5-api:5.1.3-6 bouncycastle-api:2.25 branch-api:2.7.0 build-notifications:1.5.0 caffeine-api:2.9.2-29.v717aac953ff3 checks-api:1.7.2 cloudbees-bitbucket-branch-source:757.vddedc5f2589a_ cloudbees-folder:6.714.v79e858ef76a_2 command-launcher:1.6 conditional-buildstep:1.4.2 config-file-provider:3.9.0 configuration-as-code:1414.v878271fc496f configuration-as-code-secret-ssm:1.0.1 confluence-publisher:126.v1750f291eac9 credentials:1074.v60e6c29b_b_44b_ credentials-binding:1.27.1 dashboard-view:2.19 dependency-check-jenkins-plugin:5.1.2 discard-old-build:1.05 display-url-api:2.3.6 docker-commons:1.19 docker-workflow:1.28 durable-task:493.v195aefbb0ff2 ec2:1.68 echarts-api:5.3.0-2 email-ext:2.87 embeddable-build-status:2.0.3 external-monitor-job:191.v363d0d1efdf8 favorite:2.4.1 font-awesome-api:6.0.0-1 git:4.10.3 git-client:3.11.0 git-server:1.10 github:1.34.3 github-api:1.301-378.v9807bd746da5 github-branch-source:1583.v18d333ef7379 global-slack-notifier:1.5 google-oauth-plugin:1.0.6 handlebars:3.0.8 handy-uri-templates-2-api:2.1.8-1.0 htmlpublisher:1.29 jackson2-api:2.13.2-260.v43d711474c77 javadoc:217.v905b_86277a_2a_ javax-activation-api:1.2.0-2 javax-mail-api:1.6.2-5 jaxb:2.3.0 jdk-tool:1.5 jenkins-design-language:1.25.3 jira:3.7 jjwt-api:0.11.2-9.c8b45b8bb173 jnr-posix-api:3.1.7-3 job-dsl:1.77 jquery:1.12.4-1 jquery-detached:1.2.1 jquery3-api:3.6.0-2 jsch:0.1.55.2 junit:1.56 kubernetes:3568.vde94f6b_41b_c8 kubernetes-client-api:5.12.1-187.v577c3e368fb_6 kubernetes-credentials:0.9.0 ldap:2.8 lockable-resources:2.14 mailer:408.vd726a_1130320 managed-scripts:1.5.4 mapdb-api:1.0.9.0 matrix-auth:3.1 matrix-project:758.v7a_ea_491852f3 maven-plugin:3.18 mercurial:2.16 metrics:4.1.6.1 momentjs:1.1.1 node-iterator-api:1.5.1 oauth-credentials:0.5 oidc-provider:39.vb_a_d851b_03d30 okhttp-api:4.9.3-105.vb96869f8ac3a p4:1.12.2 pam-auth:1.7 parameterized-trigger:2.44 pipeline-aws:1.43 pipeline-build-step:2.16 pipeline-graph-analysis:188.v3a01e7973f2c pipeline-input-step:446.vf27b_0b_83500e pipeline-milestone-step:100.v60a_03cd446e1 pipeline-model-api:2.2064.v5eef7d0982b_e pipeline-model-definition:2.2064.v5eef7d0982b_e pipeline-model-extensions:2.2064.v5eef7d0982b_e pipeline-multibranch-defaults:2.1 pipeline-rest-api:2.23 pipeline-stage-step:291.vf0a8a7aeeb50 pipeline-stage-tags-metadata:2.2064.v5eef7d0982b_e pipeline-stage-view:2.23 pipeline-utility-steps:2.12.0 plain-credentials:1.8 plugin-util-api:2.15.0 popper-api:1.16.1-2 popper2-api:2.11.4-1 pubsub-light:1.16 rebuild:1.33 run-condition:1.5 saml:2.296.v0016349946db_ scm-api:595.vd5a_df5eb_0e39 script-security:1145.vb_cf6cf6ed960 slack:608.v19e3b_44b_b_9ff snakeyaml-api:1.29.1 sonar:2.14 sse-gateway:1.25 ssh:2.6.1 ssh-agent:1.24.1 ssh-credentials:1.19 ssh-slaves:1.806.v2253cedd3295 sshd:3.1.0 structs:308.v852b473a2b8c subversion:2.15.3 support-core:1130.vb_eef6015fc37 tap:2.3 thinBackup:1.10 timestamper:1.17 token-macro:280.v97a_82642793c trilead-api:1.0.13 variant:1.4 vsphere-cloud:2.26 windows-slaves:1.8 workflow-aggregator:2.7 workflow-api:1143.v2d42f1e9dea_5 workflow-basic-steps:941.vdfe1b_a_132c64 workflow-cps:2682.va_473dcddc941 workflow-cps-global-lib:564.ve62a_4eb_b_e039 workflow-durable-task-step:1121.va_65b_d2701486 workflow-job:1174.vdcb_d054cf74a_ workflow-multibranch:711.vdfef37cda_816 workflow-scm-step:2.13 workflow-step-api:622.vb_8e7c15b_c95a_ workflow-support:815.vd60466279fc8 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

FROM jenkins/jenkins:2.332.1
ENV JAVA_OPTS -Djenkins.install.runSetupWizard=false
ENV CASC_JENKINS_CONFIG /var/jenkins_home/casc.yaml
COPY plugins.txt /usr/share/jenkins/ref/plugins.txt
RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins.txt
COPY jenkins.yaml /var/jenkins_home/casc.yaml

JCASC - jenkins.yaml

jenkins:
  systemMessage: "Playing with Jenkins Configuration as Code (JCasC) Deployed to GKE and Minikube. "
  agentProtocols:
    - "JNLP4-connect"
    - "Ping"
credentials:
  system:
    domainCredentials:
    - credentials:
      - idTokenFile:
          id: aws-jwt
          scope: GLOBAL
          audience: sts.amazonaws.com

Reproduction steps

Check credentials added via jcasc

def creds = com.cloudbees.plugins.credentials.CredentialsProvider.lookupCredentials(
      com.cloudbees.plugins.credentials.Credentials.class
)

for (c in creds) {
  println "${c.id} ${c.class}"
}

aws-jwt class io.jenkins.plugins.oidc_provider.IdTokenFileCredentials

➜  jenkins-poc curl http://localhost:8080/oidc/.well-known/openid-configuration -v 

*   Trying 127.0.0.1:8080...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /oidc/.well-known/openid-configuration HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.79.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Date: Sun, 06 Nov 2022 21:15:50 GMT
< X-Content-Type-Options: nosniff
< Content-Length: 0
< Server: Jetty(9.4.43.v20210629)
< 
* Connection #0 to host localhost left intact
➜  jenkins-poc curl http://localhost:8080/oidc/jwks -v                             

*   Trying 127.0.0.1:8080...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /oidc/jwks HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.79.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Date: Sun, 06 Nov 2022 21:15:58 GMT
< X-Content-Type-Options: nosniff
< Content-Length: 0
< Server: Jetty(9.4.43.v20210629)
< 
* Connection #0 to host localhost left intact

Expected Results

relevant json returned

Actual Results

404 not found

Anything else?

System logs:

Nov 06, 2022 9:07:34 PM FINE io.jenkins.plugins.oidc_provider.Keys findIssuer
found RootIssuer[http://localhost:8080/oidc] but has no credentials with default issuer; not advertising existence of a folder
Nov 06, 2022 9:07:38 PM FINE io.jenkins.plugins.oidc_provider.Keys findIssuer
looking up issuer for 
Nov 06, 2022 9:07:38 PM FINE io.jenkins.plugins.oidc_provider.Keys findIssuer
found RootIssuer[http://localhost:8080/oidc] but has no credentials with default issuer; not advertising existence of a folder
Nov 06, 2022 9:15:42 PM FINE io.jenkins.plugins.oidc_provider.Keys findIssuer
looking up issuer for 
Nov 06, 2022 9:15:47 PM FINE io.jenkins.plugins.oidc_provider.Keys findIssuer
found RootIssuer[http://localhost:8080/oidc] but has no credentials with default issuer; not advertising existence of a folder
Nov 06, 2022 9:15:50 PM FINE io.jenkins.plugins.oidc_provider.Keys findIssuer
looking up issuer for 
Nov 06, 2022 9:15:50 PM FINE io.jenkins.plugins.oidc_provider.Keys findIssuer
found RootIssuer[http://localhost:8080/oidc] but has no credentials with default issuer; not advertising existence of a folder
Nov 06, 2022 9:15:58 PM FINE io.jenkins.plugins.oidc_provider.Keys findIssuer
looking up issuer for 
Nov 06, 2022 9:15:58 PM FINE io.jenkins.plugins.oidc_provider.Keys findIssuer
found RootIssuer[http://localhost:8080/oidc] but has no credentials with default issuer; not advertising existence of a folder

iwarapter commented 1 year ago

I did some trail and error - it looks like its aws-secrets-manager-credentials-provider:1.0.0

jglick commented 1 year ago

https://github.com/jenkinsci/oidc-provider-plugin/blob/bad851b03d3093a7224fe28f72f920282188b6bd/src/main/java/io/jenkins/plugins/oidc_provider/Keys.java#L123 so I suspect https://github.com/jenkinsci/oidc-provider-plugin/blob/bad851b03d3093a7224fe28f72f920282188b6bd/src/main/java/io/jenkins/plugins/oidc_provider/Issuer.java#L64-L67 should not be returning early but rather aggregating results.

jenkinsci / oidc-provider-plugin