Can't run opm cli inside a container running on openshift

[csantanapr]

We are trying to create a CICD Pipeline using OpenShift Pipelines (aka Tekton)

In one of the stages we need to create the bundle index following the instructions https://github.com/operator-framework/operator-registry#building-an-index-of-operators-using-opm

opm index add --bundles quay.io/ibmgaragecloud/memcached-operator-bundle:0.0.2 --tag quay.io/ibmgaragecloud/operator-catalog:0.0.1 --pull-tool podman

We want to generate the Dockerfile such that in a later task or step we can use buildah to build the image and push to a registry

We need to run opm inside a container in tekton task on OpenShift, and the containers runs on top of CRIO with overlay2

opm currently only supports podman or docker to be able to access the merged layer files

When we run opm from inside the container we get the following error

time="2020-11-19T21:52:21Z" level=info msg="building the index" bundles="[quay.io/ibmgaragecloud/memcached-operator-bundle:0.0.2]"
time="2020-11-19T21:52:21Z" level=info msg="running /usr/bin/podman pull quay.io/ibmgaragecloud/memcached-operator-bundle:0.0.2" bundles="[quay.io/ibmgaragecloud/memcached-operator-bundle:0.0.2]"
time="2020-11-19T21:52:22Z" level=info msg="running podman create" bundles="[quay.io/ibmgaragecloud/memcached-operator-bundle:0.0.2]"
time="2020-11-19T21:52:22Z" level=info msg="running podman cp" bundles="[quay.io/ibmgaragecloud/memcached-operator-bundle:0.0.2]"
time="2020-11-19T21:52:22Z" level=error msg="Error: error resolving storage path for container 29baf9b96f58710979ce75a6b9c8225d11880066ff53bb7a8a238bafec38d80f: lstat /var/lib/containers/storage/overlay/5e396bd63d5e10704148e6963b9035753385a6b4323cdc014733ab1c16828393/merged: invalid argument\n" bundles="[quay.io/ibmgaragecloud/memcached-operator-bundle:0.0.2]"
time="2020-11-19T21:52:22Z" level=error msg="permissive mode disabled" bundles="[quay.io/ibmgaragecloud/memcached-operator-bundle:0.0.2]" error="error copying container directory Error: error resolving storage path for container 29baf9b96f58710979ce75a6b9c8225d11880066ff53bb7a8a238bafec38d80f: lstat /var/lib/containers/storage/overlay/5e396bd63d5e10704148e6963b9035753385a6b4323cdc014733ab1c16828393/merged: invalid argument\n: exit status 125"
Error: error copying container directory Error: error resolving storage path for container 29baf9b96f58710979ce75a6b9c8225d11880066ff53bb7a8a238bafec38d80f: lstat /var/lib/containers/storage/overlay/5e396bd63d5e10704148e6963b9035753385a6b4323cdc014733ab1c16828393/merged: invalid argument
: exit status 125

We created the following issue in podman repository as we were not sure if this was a podman config inside the container or a problem with opm tool. It was suggested by @rhatdan https://github.com/containers/podman/issues/8417#issuecomment-731251322 that perhaps it would be better for opm to support buildah when running inside a container.

You can easily give this a try using the reproduce steps here https://github.com/containers/podman/issues/8417

[usrbinkat]

+1 for opm dependency being buildah instead of podman

Per https://github.com/operator-framework/operator-registry/issues/529 this is a sticking point impacting practical devops patterns.

Can opm query public api package lists by default instead, and then support spinning up catalog index images locally only when told explicitly to do so?

[csantanapr]

We found workaround using --generate -d ./index.Dockerfile then the next step in tekton using the Dockerfile with buildah or podman in a tekton task work to build the catalog image and push to external image registry.

We got working using an external registry, but we want to use the internal OpenShift registry image-registry.openshift-image-registry.svc:5000 and using the internal registry there are issues not finding the serviceaccount token same way as buildah and then a problem with the HTTP transport doing the first HTTP HEAD I will open an new issue for this

[mvalarh]

I have same problem in using export vi podman when running in container

[root@ip-172-31-29-117 playbooks]# /tmp/operator-test/bin/opm index export --package hive-operator --index quay.io/openshift-community-operators/catalog:v4.6 -c podman -f /tmp/test/0
INFO[0000] export from the index                         index="quay.io/openshift-community-operators/catalog:v4.6" package="[hive-operator]"
INFO[0000] Pulling previous image quay.io/openshift-community-operators/catalog:v4.6 to get metadata  index="quay.io/openshift-community-operators/catalog:v4.6" package="[hive-operator]"
INFO[0000] running /usr/bin/podman pull quay.io/openshift-community-operators/catalog:v4.6  index="quay.io/openshift-community-operators/catalog:v4.6" package="[hive-operator]"
INFO[0013] running /usr/bin/podman pull quay.io/openshift-community-operators/catalog:v4.6  index="quay.io/openshift-community-operators/catalog:v4.6" package="[hive-operator]"
INFO[0015] Getting label data from previous image        index="quay.io/openshift-community-operators/catalog:v4.6" package="[hive-operator]"
INFO[0015] running podman inspect                        index="quay.io/openshift-community-operators/catalog:v4.6" package="[hive-operator]"
INFO[0015] running podman create                         index="quay.io/openshift-community-operators/catalog:v4.6" package="[hive-operator]"
INFO[0015] running podman cp                             index="quay.io/openshift-community-operators/catalog:v4.6" package="[hive-operator]"
ERRO[0015] Error: invalid arguments time="2021-02-19T10:17:14Z" level=error msg="unable to write pod event: \"write unixgram @0030b->/run/systemd/journal/socket: sendmsg: no such file or directory\""
94ada44cab9322acfc04aa8f0b6e8acb09b6481bdf1fd3818f5ef1c53f16d68b:/., index_tmp_117266040 you must use just one container  index="quay.io/openshift-community-operators/catalog:v4.6" package="[hive-operator]"
Error: error copying container directory Error: invalid arguments time="2021-02-19T10:17:14Z" level=error msg="unable to write pod event: \"write unixgram @0030b->/run/systemd/journal/socket: sendmsg: no such file or directory\""
94ada44cab9322acfc04aa8f0b6e8acb09b6481bdf1fd3818f5ef1c53f16d68b:/., index_tmp_117266040 you must use just one container
: exit status 125
Usage:
  opm index export [flags]
...
...

Will it be supported?

[exdx]

The opm build-tool defaults to podman -- can podman build a container inside a container?

The opm export command runs docker|podman cp which goes and creates a container, which fails inside a container due to the container trying to access a mount namespace and lacking privileges to do so.

I think this is a legitimate bug that should be addressed.