Private OCI registry authentication not working

[anirudhprasad-sap]

Followup of bug - https://github.com/kyma-project/module-manager/issues/201

We followed the guide https://github.com/kyma-project/lifecycle-manager/blob/main/docs/user/config-private-registry.md to create the secrets and labels to pull the images from a private registry. But on doing so, the lifecycle is crashing with the below error-

Steps-

1. We created a docker-registry secret with label - operator.kyma-project.io/oci-registry-cred: operator-regcred

2. Called kyma alpha create module with flag - --registry-cred-selector=operator.kyma-project.io/oci-registry-cred=operator-regcred which created the moduleTemplate with the given labels.

But on applying the module template to the cluster, we get the above errors in the lifecycle-manager logs. Can you please look into this?

[anirudhprasad-sap]

@ruanxin Can you please confirm if the issue is solved or not? We tried with the latest version of the lifecycle-manager, and there are no errors in the lifecycle-manager logs now. But the manifest resource went into an error state with the following error-

We followed all the steps mentioned in https://github.com/kyma-project/lifecycle-manager/blob/main/docs/user/tutorials/config-private-registry.md. But not sure why it is still not able to find the secrets.

[ruanxin]

Hi @anirudhprasad-sap, after revisiting our document, one important label is missing to mention, you should give the secret another label because the lifecycle manager controller client used to access the cluster is caching the secret for the performance perspective. I'll update the document soon with a PR, but what you can do now is add the following label to your secret, sorry for this confusion.

labels:
   operator.kyma-project.io/managed-by: lifecycle-manager

[anirudhprasad-sap]

@ruanxin We just tried testing the end-to-end flow. By creating the secrets in the kcp-system namespace, the lifecycle-manager was able to install our operator-manager (which itself was failing before this fix).

But within the new operator-manager namespace, the deployment image pull failed because of missing secrets. So here, we had to once again manually create the secrets in the operator-manager namespace and also edit the deployment to add the imagePullSecrets for it to work. Is this the expectation?

We assumed that the secret that we created in the kcp-system namespace would also be copied over to the operater-manager namespace for the pull to work from the private OCI registry. Can you please clarify?

[ruanxin]

Hi @anirudhprasad-sap, that's actually not expected behavior, the secret which we try to fetch is not namespaced, which means you can deploy your secret under any namespace, just need to make sure those label exists.

Can you verify during your tests, only one secret exists? I mean have you removed the secrets previously deployed under kcp-system before you deploy it into your operator namespace?

And sorry overlooked your questions for such long time, next time, feel free reach me over slack or teams, it's more efficient.

[ruanxin]

@anirudhprasad-sap, I'm also wondering if the use case for you to put the secret under operator namespace make sense, The operator namespace will be created during deploying your operator by lifecycle manager, but the yaml file configured those resources (operator namespace, RBAC, operator deployment..) are persistent in your private OCI image layer, you should not manually create your operator namespace, also means the secret should not put under this namespace.

[Pavan-SAP]

Hi @ruanxin, Alright.. we can create the secret in any namespace with the right labels. This will then enable lifecycle-manager to pull the relevant info from OCI registry. However, when the lifecycle-manager creates the operator-manager namespace and deployment - those need to be updated too (maybe based on the oci-registry-cred label?). How would the deployment (for our operator-manager) work without any imagePullSecret and also the corresponding Secret being created in the namespace where the operator-manager would be deployed.

Until this deployment gets up and running we (operator-manager) cannot do much.

Or did I miss something?

[ruanxin]

Hi @pvnn,

tbh, I'm not fully understanding what's your problem here. But let me clarify what's currently we offered, beyond this, is not currently implemented.

What is currently supported in Lifecycle Manager for this secret credential is only for accessing private oci registry (referred to as registry A) which you used to serve for your module operator resource yaml files, which you created by kyma cli alpha create module command, but nothing else. It is not currently supported to access other resources stored in a private registry for different purposes, and we should even question if this should be the lifecycle manager's responsible.

For example, if you need to rely on the imagePullSecret which you defined in your operator-manager's deployment yaml to get access to some private images to pull your operator-manager image, then you need to config that secret separately, please do not mix it together with this oci-registry-cred secret, they serve for completely different purpose.

One possible solution could be to include this secret along with your module resource YAML files stored in registry A, so that Lifecycle Manager can deploy it along with other resources.

[anirudhprasad-sap]

Hi @ruanxin ,

I think it might help to have a quick sync on this over a call. I will try to set up something for next week. Hope that's fine.

[ruanxin]

Hi @anirudhprasad-sap,

Sure, feel free to create a timeslot for me on outlook, any of my available times there is fine.