Add the Ability to Provide a Custom CRD download URL for Karmada Operator

[jabellard]

What would you like to be added:

As a prerequisite for provisioning new Karmada instances, the Karmada operator needs to ensure required CRDs are downloaded. By default, it will attempt to download them from a release artifact on GitHub. This feature request is for having the ability to specify a custom URL that should be used to download the CRDs tarball.

Why is this needed: At Bloomberg, we're currently building a managed Karmada platform and want to use the Karmada operator to manage the entire lifecycle of managed Karmada instances. By default, services running in our internal network can't access resources on the internet. Whenever possible, we have a strong preference for hydrating external resources to locations that are accessible within our internal network, and would like to follow that approach for downloading the CRDs.

[RainbowMango]

It makes 100% sense to me that provide the ability to download the CRDs from a customized location instead of the default GitHub release.

I don't dig into how to implement yet, do you have an idea?

[jabellard]

It makes 100% sense to me that provide the ability to download the CRDs from a customized location instead of the default GitHub release.

I don't dig into how to implement yet, do you have an idea?

Yeah. Here's the initial idea I have: The CRDRemoteURL option is used to specify the location where the init job should download the CRDs. By default, it's set to a well known location. For this feature, I'll just add a new flag somewhere around here, that when set, will be used to override the default value.

[RainbowMango]

That's a doable solution I guess. But what does the flag look like? Is it used to specify storage path for a particular version of Karmada, or to store multiple versions of CRDs?

cc author @calvin0327 here for comments.

Another approach would be introducing a new filed to the Karmada API.

[jabellard]

That's a doable solution I guess. But what does the flag look like? Is it used to specify storage path for a particular version of Karmada, or to store multiple versions of CRDs?

cc author @calvin0327 here for comments.

Another approach would be introducing a new filed to the Karmada API.

Thanks for bringing this up. The initial idea I had can indeed work, but as I started to take a closer look into the code, I realized we might actually need a different approach where the URL would be specified as part of the Karmada resource, as you're suggesting. As of now, the version of the CRDs that get installed for a new Karmada instance is actually based on the version of the operator, and not based on the version of Karmada I want to install. Code path:

• Operator Version
• URL Template
• Remote URL

From local testing, with version v1.9.0 of the operator running:

mc -n karmada-system logs -f pods/karmada-operator-59cc874b6b-2wbj5
I0705 14:37:15.349461       1 defaults.go:53] default Karmada Image Version: v1.9.0
I0705 14:37:15.351320       1 operator.go:101] karmada-operator version: version.Info{GitVersion:"v1.9.0", GitCommit:"a03aa846cfd2c1978b166660d6f592a1c10aeb3d", GitTreeState:"clean", BuildDate:"2024-02-29T08:15:53Z", GoVersion:"go1.20.11", Compiler:"gc", Platform:"linux/arm64"}
I0705 14:37:15.351335       1 operator.go:102] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I0705 14:37:15.351547       1 context.go:80] Starting "karmada"
I0705 14:37:15.351575       1 context.go:90] Started "karmada"
I0705 14:37:15.351634       1 server.go:50] "starting server" kind="health probe" addr="[::]:8443"
I0705 14:37:15.351679       1 leaderelection.go:250] attempting to acquire leader lease karmada-system/karmada-operator...
I0705 14:37:15.356442       1 leaderelection.go:260] successfully acquired lease karmada-system/karmada-operator

When I attempt to provision a Karmada instance with version v1.10.2:

apiVersion: operator.karmada.io/v1alpha1
kind: Karmada
metadata:
  name: t1
  namespace: t1
spec:
  components:
    karmadaAPIServer:
      certSANs:
      - 192.168.1.241
      serviceType: NodePort
    karmadaAggregatedAPIServer:
      imageRepository: docker.io/karmada/karmada-aggregated-apiserver
      imageTag: v1.10.2
    karmadaControllerManager:
      imageRepository: docker.io/karmada/karmada-controller-manager
      imageTag: v1.10.2
    karmadaMetricsAdapter:
      imageRepository: docker.io/karmada/karmada-metrics-adapter
      imageTag: v1.10.2
    karmadaScheduler:
      imageRepository: docker.io/karmada/karmada-scheduler
      imageTag: v1.10.2
    karmadaWebhook:
      imageRepository: docker.io/karmada/karmada-webhook
      imageTag: v1.10.2

The operator would actually install version v1.9.0 of the CRDs:

I0705 14:39:03.268982       1 controller.go:100] "Reconciling karmada" name="t1"
I0705 14:39:03.269040       1 planner.go:108] "Start execute the workflow" workflow="init" karmada="t1/t1"
I0705 14:39:03.278060       1 crd.go:65] "[prepare-crds] Using crd folder" folder="/var/lib/karmada/1.9.0" karmada="t1/t1"
Downloading...[ 23.99% ]    1 util.go:55]
Downloading...[ 47.99% ]    1 util.go:55]
Downloading...[ 71.98% ]    1 util.go:55]
Downloading...[ 100.00% ]   1 util.go:55]
I0705 14:39:03.700140       1 util.go:50]
Download complete.
I0705 14:39:03.700493       1 crd.go:119] "[download-crds] Successfully downloaded crd package for remote url" karmada="t1/t1"
I0705 14:39:03.709211       1 crd.go:145] "[unpack] Successfully unpacked crd tar" karmada="t1/t1"

[jabellard]

@RainbowMango , I just submitted a proposal for this.

[RainbowMango]

As of now, the version of the CRDs that get installed for a new Karmada instance is actually based on the version of the operator, and not based on the version of Karmada I want to install. Code path:

Yeah, that is because there is no place for people to declare which Karmda version they want to install in the Karmada API. So the operator takes the version from the build version. This results in the limitation that the operator can only install Karmada with the same version as the operator. For instance, operator v1.9.0 can only install Karmada v1.9.0, it can't be used to install v1.10.0.

[RainbowMango]

Great thanks for the proposal, I'll look into it, and let's discuss it on that PR.

[RainbowMango]

/assign @jabellard Thanks again to @jabellard for the great job!