Allow importing multiple CRDs in the same namespace

[mcouthon]

Description of the feature or enhancement:

Allow multiple CRDs with the same spec group to be imported using cdk8s import.

Use Case:

I have this cdk8s.yaml:

language: python
imports:
    - k8s
    - https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.3.1/config/crd/bases/elbv2.k8s.aws_ingressclassparams.yaml
    - https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.3.1/config/crd/bases/elbv2.k8s.aws_targetgroupbindings.yaml

Using it as is with cdk8s import results in the second CRD overwriting the first one in my imports folder.

Meaning, it results in the file imports/aws/k8s/elbv2/__init__.py having this at the bottom (resources only from the second CRD YAML):

__all__ = [
    "TargetGroupBinding",
    "TargetGroupBindingProps",
    "TargetGroupBindingSpec",
    "TargetGroupBindingSpecNetworking",
    "TargetGroupBindingSpecNetworkingIngress",
    "TargetGroupBindingSpecNetworkingIngressFrom",
    "TargetGroupBindingSpecNetworkingIngressFromIpBlock",
    "TargetGroupBindingSpecNetworkingIngressFromSecurityGroup",
    "TargetGroupBindingSpecNetworkingIngressPorts",
    "TargetGroupBindingSpecNetworkingIngressPortsPort",
    "TargetGroupBindingSpecNetworkingIngressPortsProtocol",
    "TargetGroupBindingSpecServiceRef",
    "TargetGroupBindingSpecServiceRefPort",
    "TargetGroupBindingSpecTargetType",
]

Obviously, switching the two YAMLs places will result in the opposite effect.

Proposed Solution:

A current workaround is to use aliases, like this:

language: python
imports:
    - k8s
    - ingressclassparams:=https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.3.1/config/crd/bases/elbv2.k8s.aws_ingressclassparams.yaml
    - targetgroupbindings:=https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.3.1/config/crd/bases/elbv2.k8s.aws_targetgroupbindings.yaml

This is ok, but it's inconvenient. A better outcome would be to combine the two. Meaning, imports/aws/k8s/elbv2/__init__.py will have both of the YAMLs' resources in it:

__all__ = [
    "IngressClassParams",
    "IngressClassParamsProps",
    "IngressClassParamsSpec",
    "IngressClassParamsSpecGroup",
    "IngressClassParamsSpecIpAddressType",
    "IngressClassParamsSpecLoadBalancerAttributes",
    "IngressClassParamsSpecNamespaceSelector",
    "IngressClassParamsSpecNamespaceSelectorMatchExpressions",
    "IngressClassParamsSpecScheme",
    "IngressClassParamsSpecTags",
    "TargetGroupBinding",
    "TargetGroupBindingProps",
    "TargetGroupBindingSpec",
    "TargetGroupBindingSpecNetworking",
    "TargetGroupBindingSpecNetworkingIngress",
    "TargetGroupBindingSpecNetworkingIngressFrom",
    "TargetGroupBindingSpecNetworkingIngressFromIpBlock",
    "TargetGroupBindingSpecNetworkingIngressFromSecurityGroup",
    "TargetGroupBindingSpecNetworkingIngressPorts",
    "TargetGroupBindingSpecNetworkingIngressPortsPort",
    "TargetGroupBindingSpecNetworkingIngressPortsProtocol",
    "TargetGroupBindingSpecServiceRef",
    "TargetGroupBindingSpecServiceRefPort",
    "TargetGroupBindingSpecTargetType",
]

Other:

• [ ] :wave: I may be able to implement this feature request
• [ ] :warning: This feature might incur a breaking change

---

This is a :rocket: Feature Request

[jeverett1522]

This is a big issue that is hitting us right now also. Any progress would be much appreciated!

This is what it is importing like mentioned above:

Importing k8s v1.22.0...
Importing resources, this may take a few moments...
k8s
Importing resources, this may take a few moments...
shipa.crossplane.io
  shipa.crossplane.io/app
Importing resources, this may take a few moments...
shipa.crossplane.io
  shipa.crossplane.io/appdeploy

As you can see the same group has different sub names. Would it be hard to just create the file structure like ./imports/io/crossplane/shipa/<crd_name> ?

This would allow us to just do

from imports.io.crossplane.shipa import app, appdeploy

Or just merge them together like above. :)

[shinebayar-g]

I was confused a little bit until I find this post. I'm glad that aliasing exists, but it's not clean. I found out I could merge all of the CRDs into a giant yaml file, but this gives back a giant single file. If CRDs are generated in more organized structure that'd be a super helpful.

[yuriy-yarosh]

Well, faced the same with Crossplane

app: go mod download && go mod tidy && go run .
language: go
imports:
  - https://raw.githubusercontent.com/crossplane/crossplane/master/cluster/crds/apiextensions.crossplane.io_environmentconfigs.yaml
  - https://raw.githubusercontent.com/crossplane/crossplane/master/cluster/crds/apiextensions.crossplane.io_compositions.yaml

Which overwrites envconfigs with compositions :\

[bderrly]

I have written a Bash script to handle this while a better solution is forthcoming. The script pulls down the CRDs from the cluster and merges any with the same kind into a single YAML file. I then import the single file normally: cdk8s import crds/*.yaml.

#!/usr/bin/env bash

PROJECT_ROOT=$(pwd)
CRD_PATH="${PROJECT_ROOT}/crds"

CRDS=(
    vaultauths.secrets.hashicorp.com
    vaultstaticsecrets.secrets.hashicorp.com
    )

if [[ ! -d "${CRD_PATH}" ]]; then
    mkdir "${CRD_PATH}"
fi

for crd in "${CRDS[@]}"; do
    group=${crd#[a-z]*.} # strip kind
    echo "---" >> "${CRD_PATH}/${group}.yaml"
    kubectl get "crd/${crd}" -o yaml >> "${CRD_PATH}/${group}.yaml"
    echo "fetched ${crd}"
done

[LS80]

I do it like this

yq \
    <(wget -qO - https://raw.githubusercontent.com/argoproj/argo-cd/v2.9.0/manifests/crds/application-crd.yaml) \
    <(wget -qO - https://raw.githubusercontent.com/argoproj/argo-cd/v2.9.0/manifests/crds/applicationset-crd.yaml) \
    | cdk8s import --language python /dev/stdin

Hopefully this will get fixed at some point as we won't need the workarounds.