Open sbollwe opened 3 months ago
I would like to work on this bug.
@arsenalzp : This issue requires knowledge of Kubernetes and a Red Hat OpenShift cluster.
Here is the code where we're trying to get logs of OpenShift Build Pod:
Yes, I have enough knowledge of both solutions. Any way, if questions will arise I can ask this friendly community.
Could you please assign it to me? Is there any deadline for this issue?
Is there any deadline for this issue?
@arsenalzp : Not that I know of, the user is able to perform build. He is only concerned about the warning due to additional permissions jkube is requesting.
I've just tried to reproduce this issue on my sandbox, new service account was created, bound to the role below, related BuildConfig was also changed:
kind: Role
metadata:
creationTimestamp: "2024-04-10T19:45:40Z"
name: jkubebuilder
namespace: arsenalzp-dev
resourceVersion: "2102664605"
uid: e0de6100-38e8-4f61-83fc-e6d07c09609c
rules:
- apiGroups:
- build.openshift.io
resources:
- builds
- buildconfigs
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- image.openshift.io
resources:
- imagestreams
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- build.openshift.io
resources:
- builds/details
verbs:
- update
$ oc get buildconfigs.build.openshift.io spring-boot-s2i -o yaml|grep service
serviceAccount: jkubebuilder
$
Try to run mvn oc:build -Popenshift
, here is full log of build:
[INFO] Scanning for projects...
[INFO]
[INFO] ----------< org.eclipse.jkube.quickstarts.maven:spring-boot >-----------
[INFO] Building Eclipse JKube :: Quickstarts :: Maven :: Spring Boot Web 1.16.1
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- openshift-maven-plugin:1.16.1:build (default-cli) @ spring-boot ---
[INFO] oc: Using OpenShift build with strategy S2I
[INFO] oc: Building Docker image
[INFO] oc: Running generator spring-boot
[INFO] oc: spring-boot: Using Docker image quay.io/jkube/jkube-java:0.0.23 as base / builder
[INFO] oc: Spring Boot layered jar detected
[INFO] oc: Final output artifact file was not rebuilt since last build. HINT: try to compile and package your application prior to running the container image build task.
[INFO] oc: [spring-boot:1.16.1] spring-boot: Created docker source tar /home/username/eclipse-workspace/jkube/quickstarts/maven/spring-boot/target/docker/spring-boot/1.16.1/tmp/docker-build.tar
[INFO] oc: Updating BuildServiceConfig spring-boot-s2i for Source strategy
[INFO] oc: Adding to ImageStream spring-boot
[INFO] oc: Starting Build spring-boot-s2i
[INFO] oc: Waiting for build spring-boot-s2i-15 to complete...
[INFO] oc: time=2024-04-10T20:59:10Z level=info msg=Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled
[INFO] oc: I0410 20:59:10.387843 1 defaults.go:112] Defaulting to storage driver overlay with options [mountopt=metacopy=on].
[INFO] oc: Caching blobs under /var/cache/blobs.
[INFO] oc: Trying to pull quay.io/jkube/jkube-java:0.0.23...
[INFO] oc: Getting image source signatures
[INFO] oc: Copying blob sha256:24a3fb335248549803dfdfe5f4ea60fcc9e11d5708cc62bb1fb811f608fd094a
[INFO] oc: Copying blob sha256:94ad07648680df54b9e32724cc2febb7f0a01245dfef48e84fa39ce101b056ca
[INFO] oc: Copying config sha256:8659a9b5de083556380993b0bbb62cb4907fd85e3c49c6952a216b17e67bdbd9
[INFO] oc: Writing manifest to image destination
[INFO] oc: Generating dockerfile with builder image quay.io/jkube/jkube-java:0.0.23
[INFO] oc: Adding transient rw bind mount for /run/secrets/rhsm
[INFO] oc: STEP 1/9: FROM quay.io/jkube/jkube-java:0.0.23
[INFO] oc: STEP 2/9: LABEL io.openshift.s2i.destination=/tmp io.openshift.build.image=quay.io/jkube/jkube-java:0.0.23 io.openshift.build.source-location=/tmp/build/inputs
[INFO] oc: STEP 3/9: ENV JAVA_MAIN_CLASS=org.springframework.boot.loader.JarLauncher JAVA_OPTIONS=-Dspring.output.ansi.enabled=always JAVA_APP_DIR=/deployments OPENSHIFT_BUILD_NAME=spring-boot-s2i-15 OPENSHIFT_BUILD_NAMESPACE=usernamezp-dev
[INFO] oc: STEP 4/9: USER root
[INFO] oc: STEP 5/9: COPY upload/src /tmp/src
[INFO] oc: STEP 6/9: RUN chown -R 1000:0 /tmp/src
[INFO] oc: STEP 7/9: USER 1000
[INFO] oc: STEP 8/9: RUN /usr/local/s2i/assemble
[INFO] oc: INFO S2I source build with plain binaries detected
[INFO] oc: INFO S2I binary build from fabric8-maven-plugin detected
[INFO] oc: INFO Copying binaries from /tmp/src/deployments to /deployments ...
[INFO] oc: BOOT-INF/
[INFO] oc: BOOT-INF/classpath.idx
[INFO] oc: BOOT-INF/layers.idx
[INFO] oc: BOOT-INF/classes/
[INFO] oc: BOOT-INF/classes/application.properties
...
[INFO] oc: org/springframework/boot/loader/util/
[INFO] oc: org/springframework/boot/loader/util/SystemPropertyUtils.class
[INFO] oc: INFO Copying deployments from deployments to /deployments...
[INFO] oc: INFO Cleaning up source directory (/tmp/src)
[INFO] oc: STEP 9/9: CMD /usr/local/s2i/run
[INFO] oc: COMMIT temp.builder.openshift.io/usernamezp-dev/spring-boot-s2i-15:862c6894
[INFO] oc: Getting image source signatures
[INFO] oc: Copying blob sha256:064b40dd2dfb098da5b4d43876da5d481047121e1ed560129383b74d30b5fe99
[INFO] oc: Copying blob sha256:3dfb4347f3e4a1e56a0a43a90d3538b63f9924daeb6f1112df242d7729581f40
[INFO] oc: Copying blob sha256:f9b3fe0936554c9874103714a140a9ef563c0f36bd744df04b9e1bc27b2cad38
[INFO] oc: Copying config sha256:0ead6a6f87d8220535112b1d74174effef24a7716b727569a9790b01f9d88357
[INFO] oc: Writing manifest to image destination
[INFO] oc: --> 0ead6a6f87d8
[INFO] oc: Successfully tagged temp.builder.openshift.io/usernamezp-dev/spring-boot-s2i-15:862c6894
[INFO] oc: 0ead6a6f87d8220535112b1d74174effef24a7716b727569a9790b01f9d88357
[INFO] oc:
[INFO] oc: Pushing image image-registry.openshift-image-registry.svc:5000/usernamezp-dev/spring-boot:1.16.1 ...
[INFO] oc: Getting image source signatures
[INFO] oc: Copying blob sha256:f9b3fe0936554c9874103714a140a9ef563c0f36bd744df04b9e1bc27b2cad38
[INFO] oc: Copying blob sha256:94ad07648680df54b9e32724cc2febb7f0a01245dfef48e84fa39ce101b056ca
[INFO] oc: Copying blob sha256:24a3fb335248549803dfdfe5f4ea60fcc9e11d5708cc62bb1fb811f608fd094a
[INFO] oc: Copying config sha256:0ead6a6f87d8220535112b1d74174effef24a7716b727569a9790b01f9d88357
[INFO] oc: Writing manifest to image destination
[INFO] oc: Successfully pushed image-registry.openshift-image-registry.svc:5000/usernamezp-dev/spring-boot@sha256:893ce0c9e3edf72d5bd3ed66b927ab45b26b9c878a27728429fcc7572dd87a56
[INFO] oc: Push successful
[INFO] oc: Build spring-boot-s2i-15 in status Complete
[INFO] oc: Found tag on ImageStream spring-boot tag: sha256:893ce0c9e3edf72d5bd3ed66b927ab45b26b9c878a27728429fcc7572dd87a56
[INFO] oc: ImageStream spring-boot written to /home/username/eclipse-workspace/jkube/quickstarts/maven/spring-boot/target/spring-boot-is.yml
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:00 min
[INFO] Finished at: 2024-04-10T23:59:40+03:00
[INFO] ------------------------------------------------------------------------
As you may see no errors are observed. Correct me if I'm wrong.
@arsenalzp : The problem @sbollwe is facing is due to using a ServiceAccount with restricted permissions, You would need to create a more strict Role and bind it to some ServiceAccount, then use that ServiceAccount.
@arsenalzp : The problem @sbollwe is facing is due to using a ServiceAccount with restricted permissions, You would need to create a more strict Role and bind it to some ServiceAccount, then use that ServiceAccount.
As you might see in my previous message I created the Role
with the following permission:
rules:
- apiGroups:
- build.openshift.io
resources:
- builds
- buildconfigs
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- image.openshift.io
resources:
- imagestreams
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- build.openshift.io
resources:
- builds/details
verbs:
- update
So it doesn't allow access to pod/
resources.
This role was bound to serviceAccount jkubebuilder
, this serviceAccount is used in appropriate Build
:
$ oc get buildconfigs.build.openshift.io spring-boot-s2i -o yaml|grep service
serviceAccount: jkubebuilder
$
Check whether an action is allowed:
$ oc auth can-i get pod/logs --as system:serviceaccount:arsenalzp-dev:jkubebuilder-n arsenalzp-dev
no
$ oc auth can-i get build/logs --as system:serviceaccount:arsenalzp-dev:jkubebuilder -n arsenalzp-dev
yes
$
@arsenalzp Could you please share your Role, RoleBinding and ServiceAccount YAML snippets that you're trying? The code snippet I had shared earlier is watching pod logs, it's quite strange that you're not getting the warning given that your ServiceAccount doesn't have permission.
@arsenalzp Could you please share your Role, RoleBinding and ServiceAccount YAML snippets that you're trying? The code snippet I had shared earlier is watching pod logs, it's quite strange that you're not getting the warning given that your ServiceAccount doesn't have permission.
Yes, sure:
Role
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
creationTimestamp: "2024-04-10T19:45:40Z"
name: jkubebuilder
namespace: arsenalzp-dev
resourceVersion: "2102664605"
uid: e0de6100-38e8-4f61-83fc-e6d07c09609c
rules:
- apiGroups:
- build.openshift.io
resources:
- builds
- buildconfigs
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- image.openshift.io
resources:
- imagestreams
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- build.openshift.io
resources:
- builds/details
verbs:
- update
RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
creationTimestamp: "2024-04-10T18:52:00Z"
name: jkubebuilder
namespace: arsenalzp-dev
resourceVersion: "2102361421"
uid: 7af5bc1f-883b-47a4-a248-54e09ea52b7b
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: jkubebuilder
subjects:
- kind: ServiceAccount
name: jkubebuilder
namespace: arsenalzp-dev
ServiceAccount:
apiVersion: v1
imagePullSecrets:
- name: jkubebuilder-dockercfg-4sd55
kind: ServiceAccount
metadata:
creationTimestamp: "2024-04-10T18:42:16Z"
name: jkubebuilder
namespace: arsenalzp-dev
resourceVersion: "2102645494"
uid: 3d670c21-1525-4ac2-aca0-7a664d4f0b82
secrets:
- name: internal-registry
- name: jkubebuilder-dockercfg-4sd55
- name: builder-dockercfg-h8m4b
@rohanKanojia I'm checking whether an action is allowed:
$ oc auth can-i get pod/logs --as system:serviceaccount:arsenalzp-dev:jkubebuilder-n arsenalzp-dev
no
$ oc auth can-i watch pods/logs --as=system:serviceaccount:arsenalzp-dev:jkubebuilder
no
$ oc auth can-i get build/logs --as system:serviceaccount:arsenalzp-dev:jkubebuilder -n arsenalzp-dev
yes
As you may see neither get of pods/logs
nor watch of pods/logs
resources are allowed for jkubebuilder
serviceAccount.
Hello colleagues, Did you try to reproduce this bug on your side?
@arsenalzp : Sorry, I didn't get time to look into it. I'll try to spend some time to reproduce it today.
Are you sure that jkube is picking the correct ServiceAccount token while applying these objects onto OpenShift? If you're using this outside the cluster, I think jkube is going to pick up token from your local ~/.kube/config
, not from the ServiceAccount you configured.
You can either try to create a tekton pipeline with configured ServiceAccount to build the application or copy the ServiceAccount token into your kubeconfig (just for the sake of testing). You should be able to see it in the secret created for that particular ServiceAccount:
$ oc get secret
NAME TYPE DATA AGE
jkubebuilder-dockercfg-h4n7k kubernetes.io/dockercfg 1 6m20s
jkubebuilder-token-qxbp4 kubernetes.io/service-account-token 4 6m20s
$ oc describe secret jkubebuilder-token-qxbp4
...
Data
====
ca.crt: 7209 bytes
namespace: 7 bytes
service-ca.crt: 8422 bytes
token: <jkubebuilder-access-token>
You can copy token from there and put this in kube config:
users:
- name: developer/api-crc-testing:6443
user:
token: <jkubebuilder-access-token>
With this, you should be able to test your ServiceAccount permissions from outside the cluster.
Are you sure that jkube is picking the correct ServiceAccount token while applying these objects onto OpenShift? If you're using this outside the cluster, I think jkube is going to pick up token from your local
~/.kube/config
, not from the ServiceAccount you configured.You can either try to create a tekton pipeline with configured ServiceAccount to build the application or copy the ServiceAccount token into your kubeconfig (just for the sake of testing). You should be able to see it in the secret created for that particular ServiceAccount:
$ oc get secret NAME TYPE DATA AGE jkubebuilder-dockercfg-h4n7k kubernetes.io/dockercfg 1 6m20s jkubebuilder-token-qxbp4 kubernetes.io/service-account-token 4 6m20s $ oc describe secret jkubebuilder-token-qxbp4 ... Data ==== ca.crt: 7209 bytes namespace: 7 bytes service-ca.crt: 8422 bytes token: <jkubebuilder-access-token>
You can copy token from there and put this in kube config:
users: - name: developer/api-crc-testing:6443 user: token: <jkubebuilder-access-token>
With this, you should be able to test your ServiceAccount permissions from outside the cluster.
Sorry for delay. I was trying to run crc locally, however my laptop doesn't meet system requirements. I'm playing with Red Hat OCP SandBox, therefore there are some policy limitations which prevented me to reproduce your guide. However I was able to fix the issue by substituting the following line of code:
by the following one:
try (LogWatch logWatch = client.builds().inNamespace(applicableOpenShiftNamespace).withName(buildName).watchLog()) {
It works.
But, that change is causing a failure of OpenshiftBuildServiceIntegrationTest
test with the following exception:
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://localhost:47325/api/v1/namespaces/ns1/pods?labelSelector=openshift.io%2Fbuild.name. Message: Not Found.
I suppose that's because neither Build
nor Pod
resources don't ave labels set.
I tried to set the labels by using the following part of code
however, labels weren't applied to resources.
@rohanKanojia could you please be so kind to review how to properly label resources by using ResourceConfig
? I found it is necessary to pass integration test successfully.
@arsenalzp : Are you talking about OpenShiftBuildServiceIntegrationTest ?
I think you will need to add a new Kubernetes Server Mock(we use this testing library to mock Kubernetes API,see example in this blogpost) expectation for the code change you've done:
@arsenalzp : Are you talking about OpenShiftBuildServiceIntegrationTest ?
I think you will need to add a new Kubernetes Server Mock(we use this testing library to mock Kubernetes API,see example in this blogpost) expectation for the code change you've done:
Sorry I'm disturbing you! Yes, I asked about OpenShiftBuildServiceIntegrationTest
Sorry I'm disturbing you!
Not at all, it's part of our job to provide feedback and help to contributors. You're helping us a lot with your contributions, it doesn't take that much effort to compose a reply. If you want to communicate via chat, please don't hesitate to reach out to us on internal slack channel or Gitter channel.
@arsenalzp : polite ping, are you still working on this issue?
@arsenalzp : polite ping, are you still working on this issue?
Hello, Sorry for delay, I was preparing for AWS Architect certification, so it took some time. Yes, it was done recently, let me push PR!
Hello support,
We running the openshift-maven-plugin against our openshift cluster to build images. The build its self works and pushes to artifactory without any errors.
But we got this message, which shows an error retrieving the logs for that build job.
The message shows that the plugin need access to the resource of pods/log. If I test this with the oc command (oc logs -f build/app-s2i-9), I can retrieve the logs with a service account without access to pods/log.
The service account is defined with the following rules:
In my opinion, the builds/log resource access should be sufficient to retrive the logs of oc:build. Can you adjust this, as it's not a good idea to give more rights than necessary.
openshift-maven-plugin: 1.16.1
Thanks for your work and best regards,
Steffen