Cannot provide jenkins service account with permission to image pull & build on dev namespace

[ajdeziel]

I have been attempting to build & deploy using the jenkins service account from my tools namespace into my dev namespace. However, I keep encountering this error when pulling in base images from my tools namespace into dev:

Pulling image image-registry.openshift-image-registry.svc:5000/9c33a9-tools/aspnet:3.1 ...
Warning: Pull failed, retrying in 5s ...
Warning: Pull failed, retrying in 5s ...
Warning: Pull failed, retrying in 5s ...
error: build error: failed to pull image: After retrying 2 times, Pull image still failed due to error: unauthorized: authentication required
error: the build 9c33a9-dev/dotnet-webapi-develop-temp-ocp4-9 status is "Failed"

To resolve this, I tried to perform the following with no success:

• Provide the service account with the system:image-puller and system:image-builder role binding in my dev namespace.
• Ensuring that the NSPs across dev and tools are able to talk with one another.

[StevenBarre]

1. Shouldn't builds be happening in 9c33a9-tools ?
2. The jenkins SA interacts with the BuildConfig objects to trigger new builds, but the builds themselves run as the builder SA.
3. So the service account 9c33a9-dev/builder needs system:image-puller RoleBinding in 9c33a9-tools

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: jenkins-image-puller
  namespace: 9c33a9-tools
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:image-puller
subjects:
- kind: ServiceAccount
  name: builder
  namespace: 9c33a9-dev
- kind: ServiceAccount
  name: builder
  namespace: 9c33a9-test
- kind: ServiceAccount
  name: builder
  namespace: 9c33a9-prod

/cc @jleach in case I missed anything

[jleach]

@ajdeziel Steven helped explain what you're up to to me. Its pretty strange to have buildconfig in dev. We created a tools space because that's where they're meant to run. Not saying you can't do it, but when you try and do special config like this it does require some NSP and RBAC considerations. Any reason you need to do the build in dev?

[ajdeziel]

@jleach The pipeline was originally constructed in this manner on OCP 3, and I was attempting a lift and shift of it to OCP 4 for simplicity's sake. Is the idea of building it in tools that you can then promote images across environments more easily? Our current pipeline's methodology isolates builds & deploys to their respective space for the purpose of having clear separation between each stage of development.

[jleach]

@ajdeziel I think its easier. What might be the easies build an store the images in tools; then just source them from your deployment. I typically use the tag "latest" for dev, "test" for test and "prod" for prod. Then when the deployment detects that image in tools wit will trigger a deployment. Works great, pretty simple.

EDIT: It's also a common pattern so a bit easier to help with.