Default to in-cluster container registry

[lkingland]

Currently there are two problems with the registry configuration. First, it is defaulted to an external registry (quay.io) and secondly the repositories created during the creation process are defaulted to private, and there appears no way to update this default behavior. Taken together, these create an unfortunate barrier to usage. Ideally, we would deafault to using a dedicated in-cluster registry for function container artifacts, with the ability to upgrade to using an external, third-party registry if desired. This can be accomplished by manually configuring a v2 Docker Registry mirror in cluster, which we can later encapsulate in the Functions Operator installation process.

It is unclear to me at this time what steps are necessary to expose the registry for direct-push, rather than rely entirely on its mirroring of Docker hub, but I assume it is possible because one can push directly to the registry when running locally. I presume it will require setting a public route, and perhaps placing it behind an Oauth proxy. We could also perhaps rely on opening a tunnel such as with kubectl proxy.

With this registry endpoint available, however, it can become the default option, with the --registry flag defaulted to empty, indicating faas should attempt to use the in-cluster registry accessible in-cluster.

[matejvasek]

@lkingland as for tunneling by kubectl port-forward: It could work, but it is not going to be easy. At least CLI clients (docker,podman) are tightly coupling tag and host where the tag is pushed.

For instance it's not possible to push tag our-image-registry.svc.cluster.local:5000/faas/quarkus-fn to localhost:5000 (kubectl port-forward exposes remote port on localhost).

This can be workaround: add line to /etc/hosts -- 127.0.0.1 our-image-registry.svc.cluster.local

There might be hope. Maybe we could call docker registry API manually (via simple lib), then maybe just maybe we could decouple tag and network host of the registry.

[matejvasek]

I could look into this more.

[lance]

There is a thread on this in @serverless-dev pointing to https://docs.openshift.com/container-platform/4.5/registry/securing-exposing-registry.html

[lance]

In the document linked above seems pretty straightforward to push to the internal registry on OpenShift. But we definitely want to have a solution for upstream as well. I suggest that we aim for three targets.

1. Use docker.io as the default since it doesn't suffer from the private-by-default issue and requires minimal if any configuration for the user
2. Document (and test) using an exposed OpenShift internal registry
3. Document (and test) using a registry exposed in the cluster as mentioned above in the issue description

For Dev Preview, I think they should be prioritized in that order as well.

[matejvasek]

@lance @lkingland Indeed for OpenShift it's not that difficult. It is not even necessary to expose it publicly. Just

kubectl port-forward -n openshift-image-registry service/image-registry 5000:5000

this exposes registry locally (using HTTP).

Then it's possible to push it

podman push docker-daemon:www.example.com:latest localhost:5000/NAMESAPCE/www.example.com --tls-verify=false

Of course you need to podman login, credentials should be in kubectl's configfile.

Subsequently you can deploy ksvc by:

kn service create mvasek-redhat-cz-42 --image image-registry.openshift-image-registry.svc.cluster.local:5000/NAMESAPCE/www.example.com:latest --namespace default --env VERBOSE=true --label faas.domain=redhat.cz --label bosonFunction=true --annotation faas.subdomain=mvasek

Only inconvenience is that this exposes registry via insecure connection. To use it you either must use podman with --tls-verify=false or edit (as a root) docker config file to allow insecure connection for localhost:5000.

I am trying to automate this in Go I really would like to do it rootless and not require additional binary (podman). Also we are not really rootless now since we use docker for image building.

As for vanilla k8s I don't have solution yet.

[matejvasek]

Only inconvenience is that this exposes registry via insecure connection.

NOTE: If you expose it publicly via HTTPS you still may have issues. For instance the cluster that I have uses selfsigned certificate. I have to allow it system wide (as root) and then restart docker daemon.

[matejvasek]

Only inconvenience is that this exposes registry via insecure connection

NOTE: don't be scared by insecure connection the tunnel that is used should encrypt the communication AFAIK.

[matejvasek]

Just to be sure that it works, @lkingland could you please apply the steps above on your OpenShift setup?

[matejvasek]

@lance @nainaz as discussed this should be done by adding documentation, what it the place where I should put the documentation? Should I also add that to template READEs?

[lance]

@matejvasek maybe as a standalone guide in https://github.com/openshift-knative/docs?

[lance]

Hmm... as I feared this fails with port forwarding in a shared cluster. I don't have permissions to access resources in the regsitry's namespace.

❯ kubectl port-forward -n openshift-image-registry service/image-registry 5000:5000
Error from server (Forbidden): services "image-registry" is forbidden: User "lance" cannot get resource "services" in API group "" in the namespace "openshift-image-registry"

❯ oc whoami -c                                                                                                         570ms  Thu 01 Oct 2020 05:35:39 PM EDT
example/api-devint-openshiftknativedemo-org:6443/lance

[matejvasek]

@lance so they would need ask admin for permission?

[lance]

@lance so they would need ask admin for permission?

Yes I believe so. I'm not sure exactly what kubectl is doing, but I assume it's modifying a service resource to expose the port.

[matejvasek]

Too bad, at least it can be useful for CRC.

[matejvasek]

api-devint-openshiftknativedemo-org

I never used that.

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Reopen the issue with /reopen. Mark the issue as fresh by adding the comment /remove-lifecycle stale.

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Reopen the issue with /reopen. Mark the issue as fresh by adding the comment /remove-lifecycle stale.