OpenShift Client Plugin Token Credentials Unusable

chriscarpenter12 commented 2 years ago

Description When using the sync plugin for OpenShift Client Plugin Token credentials they are unavailable to select from the Client Plugin configuration. Secrets are correctly synced to Jenkins credentials, but there's now two different credential types for OpenShift Token for OpenShift Client Plugin in Jenkins.

Expectation Opaque secrets with data key openshift-client-token are available to the OpenShift Client Plugin configration.

Analysis From what I can see the opaque secrets with openshift-client-token are synced correctly to Jenkins credentials, but the credential class type is wrong in credentials.xml making the credentials not selectable to the client plugin.

Manually creating a OpenShift Token for OpenShift Client Plugin credential in Jenkins works correctly.

kind: Secret
apiVersion: v1
metadata:
  name: openshift-client-token-sync
  namespace: jenkins-pipeline
  labels:
    credential.sync.jenkins.openshift.io: 'true'
data:
  openshift-client-token: dGVzdA==
type: Opaque

Upon inspecting the credentials.xml file it seems the synced credentials have a class of io.fabric8.jenkins.openshiftsync.OpenShiftTokenCredentials instead of the expected com.openshift.jenkins.plugins.OpenShiftTokenCredentials.

credentials.xml

<com.openshift.jenkins.plugins.OpenShiftTokenCredentials plugin="openshift-client@1.0.35">
  <scope>GLOBAL</scope>
  <id>openshift-client-token-manual</id>
  <description></description>
  <secret>{AQAAABAAAAAQVuLJ2BepgVlVCXWXMQnDiVAB97s/VzApyPNc7lCYAyc=}</secret>
</com.openshift.jenkins.plugins.OpenShiftTokenCredentials>
<io.fabric8.jenkins.openshiftsync.OpenShiftTokenCredentials plugin="openshift-sync@1.0.51">
  <scope>GLOBAL</scope>
  <id>jenkins-pipeline-openshift-client-token-sync</id>
  <description>jenkins-pipeline-openshift-client-token-sync</description>
  <secret>{AQAAABAAAAAQN/3yCFoHcIIuf5Lz6pJjiMflnFmv1jnGSbazK624mhk=}</secret>
</io.fabric8.jenkins.openshiftsync.OpenShiftTokenCredentials>

It seems that it was intended to remove a dependency on the openshift client plugin, but it doesn't seem to be working for me.

Versions

Server Version: 4.8.12
Kubernetes Version: v1.21.1+d8043e1
Jenkins 2.303.3
openshift-client: 1.0.35
openshift-login: 1.0.26
openshift-sync: 1.0.51

registry.redhat.io/openshift4/ose-jenkins@sha256:a624b19e15d09f5519a2b68545ba910e37a5d854341732e092d22644a6b14b96

chriscarpenter12 commented 2 years ago

Any official comment?

thikade commented 2 years ago

This issue is really quite bad, I am seeing the same error as @chriscarpenter12. Additionally, code using openshift.withCredentials("credentialID") no longer works. The error reported for a simple openshift.raw("whoami") command is this:

ERROR: raw command [whoami] returned an error;
{err=error: You must be logged in to the server (Unauthorized)
, verb=, cmd=oc --server=https://openshift.default.svc.cluster.local/ --insecure-skip-tls-verify --namespace=example --token=XXXXX  whoami , out=, status=1

The bad token is of the following type in credentials.xml, exactly as @chriscarpenter12 described: <io.fabric8.jenkins.openshiftsync.OpenShiftTokenCredentials plugin="openshift-sync@1.0.53">

versions used:

Openshift 4.8 Client Version: 4.8.0-202108061927.p0.git.c6ad8ac.assembly.stream-c6ad8ac Kubernetes Version: v1.21.1+9807387
OCP Jenkins 2.303.3
OpenShift Client Jenkins Plugin: 1.0.35
OpenShift Sync: 1.0.53

(btw: I know I dont need an explicit token to access local cluster - this is just an example of the error.)

thikade commented 2 years ago

the implication of this error is that one can no longer sync secrets into openshift client plugin tokens to access external clusters! ... so a whole automation setup is now broken!

chriscarpenter12 commented 2 years ago

@thikade Thank you! I felt like I was going crazy as the only person with this issue. I resorted to manually create the credentials in Jenkins for now and had to tweak our pipeline code as well. Frustrating

thikade commented 2 years ago

We are using lots of ephemeral Jenkins instances that configure themselves on boot using casc-plugin. Worked like a charm ... until now. Maybe I can find the time to see if downgrading to an older client-plugin will fix it as a workaround...

akram commented 2 years ago

@chriscarpenter12 and @thikade thank you for reporting this. I will open a bz for this to see how we can fix it.

cc @gabemontero https://bugzilla.redhat.com/show_bug.cgi?id=2054816

gabemontero commented 2 years ago

@chriscarpenter12 and @thikade thank you for reporting this. I will open a bz for this to see how we can fix it.

cc @gabemontero https://bugzilla.redhat.com/show_bug.cgi?id=2054816

seems like you need to undo https://github.com/openshift/jenkins-sync-plugin/commit/b8948a933b12ce21382d1e0a2fd64b5d281fc072#diff-431bed0f2f8fd5c62e9e129cc018a6d64e0f7a1f244b15719cc035b8e5e1b150 right @akram ?

On the surface I don't see how that had any bearing on switching the sync plugin to shared informers, right?

akram commented 2 years ago

I was about to blame myself...so yes, that's the culprit. I removed that dependency while solving a jar hell that we had between the kubernetes-client-api , openshift-client and the openshift-sync-plugin edit: It allowed us to upgrade the kubernets-client

I will see then how can I fix that.

gabemontero commented 2 years ago

I was about to blame myself...so yes, that's the culprit. I removed that dependency while solving a jar hell that we had between the kubernetes-client-api , openshift-client and the openshift-sync-plugin edit: It allowed us to upgrade the kubernets-client

Ah ... yeah, I could see how that could have cropped up.

I will see then how can I fix that.

yeah not knowing the particulars that lead to ^^, if it is not easily settled now that the dust has settled on the transition to shared informers, maybe there is a portion of code in either client or sync plugin that will need to be moved to a different repo both can share as a dependency.

heck, openshift/jenkins could even be a landing spot in a pinch for something like that

akram commented 2 years ago

Yes, I was thinking about a move like this, but it is another subject to discuss. Let's book a few minutes tomorrow

akram commented 2 years ago

@chriscarpenter12 and @thikade do you have the ability to test https://github.com/openshift/jenkins-client-plugin/pull/378 ? The fix for this issue would go with an update of the openshift-client plugin .

chriscarpenter12 commented 2 years ago

Looks like the PR was just merged. Is there a snapshot version automatically built or do I need to build the hpi? When can we expect back port of this version to 4.8 branch of Jenkins base-plugins.txt

thikade commented 2 years ago

@chriscarpenter12 and @thikade do you have the ability to test openshift/jenkins-client-plugin#378 ? The fix for this issue would go with an update of the openshift-client plugin

... not that easily, as our Jenkinses are ephemeral and the update requires a restart => I need to get the plugin hpi into the image first.

gabemontero commented 2 years ago

I will be initiating the new release of the client plugin after lunch US Eastern today.

Getting the plugin into the image is a much more laborious process. And we cannot go directly to 4.8. We have to

update master branch (4.11 now) .... update the openshift/jenkins repo, but then work with our internal release team to cut RPMs for that OCP level
update 4.10 (which is currently frozen) in the same way .... we may be able to get a bypass exception for that
update 4.9 in the same way
then update 4.8

gabemontero commented 2 years ago

maybe end of this week but most likely next week to get the 4.8 image updated internally ... then it is a question of when the next 4.8 errata goes out, though I think those are still at a weekly cadence. 4.8.32 appears to have been in the pipeline for release for 6 days now. If so, this would get in 4.8.33 or 4.8.34.

chriscarpenter12 commented 2 years ago

Thanks for the update. I built the plugin locally and tested. The synced credential is now selectable in the global client config.

openshift-client: 1.0.36-SNAPSHOT (private-d352d634-chris) jenkins: 2.303.3

gabemontero commented 2 years ago

thanks for the confirmation @chriscarpenter12

still getting up to speed from when @akram had to step away today

he in fact submitted v1.0.36 of the client plugin a few hours ago with https://github.com/jenkinsci/openshift-client-plugin/commit/9d4f2954737013e5339f531d0ec60118bb578a69 but it is not yet available at https://updates.jenkins-ci.org/download/plugins/openshift-client/

I will be monitoring that page (as well as picking the release commits back to the openshift side @akram )

akram commented 2 years ago

@gabemontero and @chriscarpenter12 ; I have just done the release of 1.0.36 which includes the fix.

akram commented 2 years ago

@gabemontero and @chriscarpenter12 ; I have just done the release of 1.0.36 which includes the fix.

thikade commented 2 years ago

@akram @chriscarpenter12 : I too can confirm that the synced credential is now selectable in the cluster config dropdown, but unfortunately my issue is not resolved by that fix. I still get unauthorized when running openshift.withCredentials("credentialID") with credentialID being my synced Openshift client token ID.

credential used:

<io.fabric8.jenkins.openshiftsync.OpenShiftTokenCredentials plugin="openshift-sync@1.0.53">
  <scope>GLOBAL</scope>
  <id>synced_jenkins_token</id>
  <secret>...</secret>
</io.fabric8.jenkins.openshiftsync.OpenShiftTokenCredentials>

My test code is simply doing a oc whoami raw-cmd to show the id of connected user:

def call (String clusterAddress="default", String namespace = null, String tokenID = null) {
  openshift.withCluster(clusterAddress) {
    openshift.withCredentials(tokenID) {
        openshift.withProject(namespace) {
            def x = openshift.raw("whoami")
            def user = x.out.trim()
            println "running as user: \"${user}\"  in project: \"${openshift.project()}\"  on cluster \"${openshift.cluster()}\""
        }
    }
  }
}

Result:

ERROR: raw command [whoami] returned an error;
2022-02-23 11:18:25 |  {err=error: You must be logged in to the server (Unauthorized)
2022-02-23 11:18:25 |  , verb=, cmd=oc --server=https://openshift.default.svc.cluster.local/ --insecure-skip-tls-verify --namespace=jenkins-test --token=XXXXX  whoami , out=, status=1}

Same code still works when credentialID is a manually created Jenkins credential of (old) type Openshift client token:

<com.openshift.jenkins.plugins.OpenShiftTokenCredentials plugin="openshift-client@1.0.36">
    <scope>GLOBAL</scope>
    <id>jenkins-manually-added-1</id>
    <secret>...</secret>
</com.openshift.jenkins.plugins.OpenShiftTokenCredentials>

Result:

running as user: "system:serviceaccount:jenkins-test:jenkins"  in project: "jenkins-test"  on cluster "https://openshift.default.svc.cluster.local/"

versions used:

OpenShift Client Jenkins Plugin: 1.0.36
Openshift Sync: 1.0.53
Jenkins 2.319.2

akram commented 2 years ago

/reopen

openshift-ci[bot] commented 2 years ago

@akram: Reopened this issue.

In response to [this](https://github.com/openshift/jenkins-sync-plugin/issues/414#issuecomment-1048669235): >/reopen Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

akram commented 2 years ago

@thikade did you notice something special in the jenkins logs related to it? Can you paste here?

chriscarpenter12 commented 2 years ago

@akram I can confirm there's an issue with the openshift DSL too. We previously relied on using openshift.withCluster('cluster-name') to inherently get the correct credentials defined in the global config. With the synced credential set on my global config I too see failure when running the even simpler pipeline like so:

openshift.withCluster('test-cluster') {
    openshift.withProject('ci') {
        def x = openshift.raw("whoami")
        def user = x.out.trim()
        println "running as user: \"${user}\"  in project: \"${openshift.project()}\"  on cluster \"${openshift.cluster()}\""
    }
}

I suspect this needs addressed because it's looking for the old class path still or not able to support io.fabric8.jenkins.openshiftsync.OpenShiftTokenCredentials?

chriscarpenter12 commented 2 years ago

Nothing special in my logs after a failed test of that simple pipeline script

2022-02-23 09:04:26 INFO    io.fabric8.jenkins.openshiftsync.BuildSyncRunListener onStarted Run started: credential-test #4
2022-02-23 09:04:26 INFO    io.fabric8.jenkins.openshiftsync.BuildSyncRunListener onStarted Not polling polling build job/credential-test/4/ as its not a WorkflowJob
2022-02-23 09:04:28 INFO    org.jenkinsci.plugins.workflow.job.WorkflowRun finish credential-test #4 completed: FAILURE

thikade commented 2 years ago

@akram neither in my jenkins logs...

thikade commented 2 years ago

@akram I rebuilt a persistent test Jenkins instance with our little test-case pipeline from above, with the small modification of adding "loglevel=9" to the raw cmd. What is interesting is that it behaves exactly the same when I use a really non-existing credentialID, so it would seem that openshift.withCredentials is not able to get the token value from Jenkins credential correctly. Using a manually created credential still works. Here's the relevant info:

script

                    println "credentialID used is: ${credentialID}"
                    openshift.withCluster("default") {
                        openshift.withProject("jenkinsbuild") {
                            openshift.withCredentials(credentialID) {
                                def x = openshift.raw("whoami", "--loglevel=9")
                                def user = x.out.trim()
                                println "running as user: \"${user}\"  in project: \"${openshift.project()}\"  on cluster \"${openshift.cluster()}\""
                    } } }

versions used:

OpenShift Client Jenkins Plugin: 1.0.36 Openshift Sync: 1.0.53 Jenkins 2.303.3

output

2022-02-24 11:31:43 |  credentialID used is: synced-jenkins-token
2022-02-24 11:31:44 |  [Pipeline] End of Pipeline
2022-02-24 11:31:44 |  ERROR: raw command [whoami, --loglevel=9] returned an error;
2022-02-24 11:31:44 |  {err=I0224 11:31:44.175598    2632 round_trippers.go:435] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: oc/4.8.0 (linux/amd64) kubernetes/88e7eba" -H "Authorization: Bearer <masked>" 'https://openshift.default.svc.cluster.local/apis/user.openshift.io/v1/users/~'
2022-02-24 11:31:44 |  I0224 11:31:44.204888    2632 round_trippers.go:454] GET https://openshift.default.svc.cluster.local/apis/user.openshift.io/v1/users/~ 401 Unauthorized in 29 milliseconds
2022-02-24 11:31:44 |  I0224 11:31:44.204925    2632 round_trippers.go:460] Response Headers:
2022-02-24 11:31:44 |  I0224 11:31:44.204939    2632 round_trippers.go:463]     Audit-Id: 715178ef-c474-4031-94fa-07bdfc93379c
2022-02-24 11:31:44 |  I0224 11:31:44.204951    2632 round_trippers.go:463]     Cache-Control: no-cache, private
2022-02-24 11:31:44 |  I0224 11:31:44.204963    2632 round_trippers.go:463]     Content-Type: application/json
2022-02-24 11:31:44 |  I0224 11:31:44.204972    2632 round_trippers.go:463]     Content-Length: 129
2022-02-24 11:31:44 |  I0224 11:31:44.204981    2632 round_trippers.go:463]     Date: Thu, 24 Feb 2022 10:31:44 GMT
2022-02-24 11:31:44 |  I0224 11:31:44.205020    2632 request.go:1123] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
2022-02-24 11:31:44 |  I0224 11:31:44.205587    2632 helpers.go:216] server response object: [{
2022-02-24 11:31:44 |    "metadata": {},
2022-02-24 11:31:44 |    "status": "Failure",
2022-02-24 11:31:44 |    "message": "Unauthorized",
2022-02-24 11:31:44 |    "reason": "Unauthorized",
2022-02-24 11:31:44 |    "code": 401
2022-02-24 11:31:44 |  }]
2022-02-24 11:31:44 |  F0224 11:31:44.205623    2632 helpers.go:115] error: You must be logged in to the server (Unauthorized)
... (goroutine output removed)
2022-02-24 11:31:44 |  , verb=, cmd=oc --server=https://openshift.default.svc.cluster.local --insecure-skip-tls-verify --namespace=jenkinsbuild --token=XXXXX  whoami --loglevel=9 , out=, status=255}
2022-02-24 11:31:44 |  
2022-02-24 11:31:44 |  Finished: FAILURE

jenkins log

2022-02-24 11:31:41 INFO    io.fabric8.jenkins.openshiftsync.BuildSyncRunListener onStarted Run started: client-plugin-test #2
2022-02-24 11:31:41 INFO    io.fabric8.jenkins.openshiftsync.BuildSyncRunListener onStarted Not polling polling build job/client-plugin-test/2/ as its not a WorkflowJob
2022-02-24 11:31:44 INFO    org.jenkinsci.plugins.workflow.job.WorkflowRun finish client-plugin-test #2 completed: FAILURE

synced token from credentials.xml

        <io.fabric8.jenkins.openshiftsync.OpenShiftTokenCredentials plugin="openshift-sync@1.0.53">
          <scope>GLOBAL</scope>
          <id>synced-jenkins-token</id>
          <description>synced-jenkins-token</description>
          <secret>...</secret>
        </io.fabric8.jenkins.openshiftsync.OpenShiftTokenCredentials>

manually created token from credentials.xml

        <com.openshift.jenkins.plugins.OpenShiftTokenCredentials plugin="openshift-client@1.0.36">
          <scope>GLOBAL</scope>
          <id>manually-created-token</id>
          <description></description>
          <secret>...</secret>
        </com.openshift.jenkins.plugins.OpenShiftTokenCredentials>

proof that secret 'synced-jenkins-token' is valid

$ oc --token $(oc extract secret/synced-jenkins-token --to=-)  whoami
system:serviceaccount:jenkinsbuild:jenkins

thikade commented 2 years ago

@akram I have built a patched snapshot version of openshift-client-plugin that shows the token value without hiding it in the error message. I believe this confirms that it fails to resolve the credentialID to the actual token value, and falls back to using the credential name.
I am pretty sure that findCredentialById fails to get the credential. Update: I see now, it's the same issue as the previous one: code is calling findCredentialById with the wrong, old class com.openshift.jenkins.plugins.OpenShiftTokenCredentials instead of io.fabric8.jenkins.openshiftsync.OpenShiftTokenCredentials.

2022-02-24 13:48:04 |  error: You must be logged in to the server (Unauthorized)
2022-02-24 13:48:04 |  , verb=, cmd=oc --server=https://openshift.default.svc.cluster.local/ --insecure-skip-tls-verify --namespace=jenkinsbuild --token=synced-jenkins-token  whoami --loglevel=5 , out=, status=1}

akram commented 2 years ago

Hi @thikade , I started to work on a fix and you are true that's happening here .

Can you try this branch: https://github.com/akram/jenkins-client-plugin/tree/use-parent-credentials-class I wasn't able to test it yet because lacking of clusters and I couldn't get credentials from the cluster that I have for some reasons.

akram commented 2 years ago

a few quick notes for the reproduced:

oc new-app jenkins-persistent
oc rsh <jenkins-1-mypod>
oc create secret generic openshift-client-token-sync --from-literal=openshift-client-token=$(oc whoami -t)
oc label secret openshift-client-token-sync credential.sync.jenkins.openshift.io=true

create a cluster name my-cluster in openshift client with the following parameters:

name: my-cluster
api-url: https://kubernetes.default
secret: openshift-client-token-sync

Use the freestyle pipeline:

openshift.withCluster(my-cluster) {
            def x = openshift.raw("whoami")
            def user = x.out.trim()
            println "running as user: \"${user}\"  in project: \"${openshift.project()}\"  on cluster \"${openshift.cluster()}\""
}

Should display: the same result as oc whoami run in jenkins pod: system:serviceaccount:my-namespace:jenkins

edit: @thikade I fixe the oc create secret command

thikade commented 2 years ago

Thanks @akram !

Can you try this branch: https://github.com/akram/jenkins-client-plugin/tree/use-parent-credentials-class I wasn't able to test it yet because lacking of clusters and I couldn't get credentials from the cluster that I have for some reasons.

Great! Yes, will do later this afternoon and send feedback.

akram commented 2 years ago

works for me with my latest changes. I am awaiting confirmation from you @thikade and I will cut a new version if all your tests work then.

akram commented 2 years ago

@gabemontero can you PTAL?

thikade commented 2 years ago

a few quick notes for the reproduced:
oc create secret generic openshift-client-token-sync --from-literal=openshift-client-token-sync=$(oc whoami -t)
Probably a typo? IMHO it should read ... --from-literal=openshift-client-token= ...

... else a Jenkins credential of kind: Secret Text will be created (instead of kind: OpenShift Token for OpenShift Client Plugin ) , which does not work and returns this error in pipeline log:

hudson.remoting.ProxyException: groovy.lang.MissingMethodException: No signature of method: org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl.getToken() is applicable for argument types: () values: []

thikade commented 2 years ago

@akram Looks good! If I use ... --from-literal=openshift-client-token= ... to create the secret it works now!

my testcases: run pipeline code with different token types ...

[x] synced token (of type io.fabric8.jenkins.openshiftsync.OpenShiftTokenCredentials plugin="openshift-sync@1.0.53")
[x] manually created token in Jenkins (of type com.openshift.jenkins.plugins.OpenShiftTokenCredentials plugin="openshift-client@1.0.37-SNAPSHOT")
[x] manually created token in Jenkins (of type io.fabric8.jenkins.openshiftsync.OpenShiftToken plugin="openshift-sync@1.0.53")
[x] specifiying an invalid credential id fails as expected with unauthorized

The only (minor) thing that is misguiding is that I can see 3 Openshift token type choices in Add Jenkins credentials view. Mind that this is not an issue for me as I prefer to use synced secrets, but it might be confusing to other users:

the first creates com.openshift.jenkins.plugins.OpenShiftTokenCredentials credentials
the second is unusable
the third creates io.fabric8.jenkins.openshiftsync.OpenShiftTokenCredentials credentials

akram commented 2 years ago

ok, great @thikade , I will take into account the review of @gabemontero and push a new version then regarding one of the option in the drop down list, I will investigate that in a later issue so we can fix the blocker now

openshift / jenkins-sync-plugin