configurable buildconfig output

[tbugfinder]

Is your feature request related to a problem? Please describe. It would be beneficial for hybrid workloads to use OpenShift/ODS as the build environment pushing images to an (or even multiple) external docker registry (registries) like Nexus, AWS ECR or Azure Container Registry. I'm using the term hybrid workloads in a sense were a single container image is used within OpenShift, AWS, Azure, HPC-Singularity executor or even plain docker.

Describe the solution you'd like The output target should be configurable so that images could be pushed to a docker registry. I didn't research if output could even be a multi-target item (ImageStream and docker registries).

Describe alternatives you've considered Building images within ODS and use the same repo for external builds which isn't a 1:1 move as Jenkinsfile has to be changed. Not everybody wants to expose the internal registry.

Additional context https://docs.openshift.com/container-platform/3.11/dev_guide/builds/build_output.html https://docs.openshift.com/container-platform/4.2/builds/managing-build-output.html

[michaelsauter]

@tbugfinder Makes sense to me. To my understanding though, the options in OpenShift are very limited - you can't even tag the produced image with two tags (I researched this).

So maybe the image would need to be pulled in Jenkins after building, and then pushed to another registry? Would that work for you?

[tbugfinder]

@michaelsauter Pulling and pushing images in Jenkins is an option. Would I have to use a custom jenkins_slave to add docker/buildah executable?

[michaelsauter]

@michaelsauter Probably. I think pulling might work via oc, put pushing would require some different tool.

I am moving this issue to ods-core for now until we now where exactly to put it.

[tbugfinder]

I spent some time on that without having success.

• oc tooling doesn't download the container image.

h-4.2$ oc get --export isimage test
No resources found.
Error from server (BadRequest): export of "imagestreamimages" is not supported
sh-4.2$ 

• docker pull/push isn't possible within a pod as it requires a running dockerd.

• rootless podman (pull/push) isn't (yet?) possible due to lack of appropriate namespaces within the pod.

• https://github.com/genuinetools/img also relies on namespace support

sh-4.2$ cat /proc/sys/user/max_user_namespaces
0
sh-4.2$ 
sh-4.2$ echo 10000 > /proc/sys/user/max_user_namespaces
sh: /proc/sys/user/max_user_namespaces: Read-only file system
sh-4.2$ 

[michaelsauter]

I think https://github.com/containers/skopeo is the way to go. I have not played with it but that might work within OpenShift.

[clemensutschig]

@tbugfinder - can you try and report on any findings? ...

[tbugfinder]

Yes - I'll provide an update soon.

[tbugfinder]

I've successfully copied an image within OpenShift itself using skopeo. It needs some tweaks for the authentication part and the certificates but overall best tool to do that.

Next test will be copying to an external registry like NEXUS.

https://github.com/nmasse-itix/OpenShift-Examples/blob/master/Using-Skopeo/README.md

[tbugfinder]

Nice read, too. https://blog.openshift.com/promoting-container-images-between-registries-with-skopeo/

[clemensutschig]

@tbugfinder - cool! ... the service account to get access to the registry is already in the pod available ... but looks like you have solved this, eh :)?

[clemensutschig]

I am wondering if we should package scopeo in a slave, but ... :)

[michaelsauter]

@clemensutschig Might be helpful! I would say we give this a test-run now and if we are happy with it and see enough value, we can add it for ODS 3.

[tbugfinder]

Awesome, skopeo copied an image successfully from Openshift registry to NEXUS.

I'd also like to see scopeo in the Jenkinsslave. This should be in ODS2.

[tbugfinder]

I was using:

 yum install -y http://mirror.centos.org/centos/7/extras/x86_64/Packages/skopeo-0.1.37-3.el7.centos.x86_64.rpm http://mirror.centos.org/centos/7/extras/x86_64/Packages/containers-common-0.1.37-3.el7.centos.x86_64.rpm

[clemensutschig]

@tbugfinder anything Else you did?

[tbugfinder]

Well, although my current test code includes podman, buildah, EPEL above line should really be all. I could strip anything else off if you need confirmation.

[clemensutschig]

@tbugfinder see PR .. anything else we miss?

the secrets are injected, and also the certs (bundled into ca.crt)

[tbugfinder]

Nice! I'll test it.

[tbugfinder]

I've picked the RHEL Dockerfile and was able to use skopeo successfully for copying an image. This makes it easy to offload images from the internal registry, even for debugging purposes.

[clemensutschig]

super cool... @michaelsauter - maybe we should add a default stage in the shared lib to use skopeo for uploading the image to nexus?.. just thinking out loud.

[michaelsauter]

@clemensutschig I'd like to understand usage a little better first. Now that skopeo is available, it should be easy enough for people to use code snippets that make use of it. During the next release cycle we could collect use cases and then see which features make sense to provide out-of-the-box. I don't really get the Nexus use-case yet to be honest. Also, people might use this to push an image into AWS or so?

[tbugfinder]

Indeed images could be uploaded to different regions in AWS/Azure, onprem registries or between OpenShift environments (see the issue's top).

The use-case is building the image once and execute it not only within the building OpenShift environment but also on e.g. native docker installations for debugging purposes or in a HPC environment.

A stage in the shared library could have map of registries and tokens as input.

Within Nexus, the docker endpoint has to be configured in addition, AFAIK.

[clemensutschig]

@tbugfinder - what about you providing an example (= code) .. then lets see if we can include ...