Adapt ytt syntax to leverage K8S CR completion during authoring

[gberche-orange]

Thanks for sharing the great work with ytt for reducing verbosity.

I had worked on similar approach using ytt and kappcontroller for deployment in a gitops maneer without local preprocessing.

My initial goal was not that much reducing verbosity but leveraging IDE built-in code assistance for authoring crossplane XRD and compositions (see https://github.com/crossplane/crossplane/issues/3197#issuecomment-1194624402 for screenshot)

I see that in the ytt resources such as the following, the IDE support can't be leveraged https://github.com/vfarcic/devops-toolkit-crossplane/blob/986cf719d82efb5320960b5bdbf3f4874c736f05/packages/ytt-k8s/_ytt_lib/google/config.yaml#L59-L66

Here is code snippet of the yaml ytt templating logic I've used to leverage IDE coding assistance support to author XRD, and Composition

composition.yaml ``` #@ load("/cloudsql-resource-data.lib.yaml", "resource") #@ load("/cloudsql-others-data.lib.yaml", "others") apiVersion: apiextensions.crossplane.io/v1 kind: Composition metadata: name: xpostgresqlinstances.gcp.database.orange.com labels: provider: gcp spec: writeConnectionSecretsToNamespace: 75-crossplane-gcp-cnx-secrets compositeTypeRef: apiVersion: database.orange.com/v1alpha1 kind: XPostgreSQLInstance #@ resource_yaml_fragment = resource() #@ resource = { "base": dict(**resource_yaml_fragment) } #@ other_elements = dict(**others()["spec"]["resources"][0]) #@ other_elements.pop("base") #@ resource.update(other_elements) #@ resources = [ resource ] resources: #@ resources ``` cloudsql-resource-data.lib.yaml: ``` #! To be excluded from output, this lib have no yaml document --- header #@ load("@ytt:data", "data") #@ def resource(): apiVersion: database.gcp.crossplane.io/v1beta1 kind: CloudSQLInstance metadata: name: cloudsql spec: forProvider: databaseVersion: POSTGRES_14 region: #@ data.values.external_gcp_region settings: tier: db-f1-micro #! tier: db-custom-1-3840 # 1 CPU, 3840 MB RAM. See https://cloud.google.com/sql/docs/postgres/create-instance#machine-types availabilityType: "ZONAL" #! for HA db use REGIONAL backupConfiguration: { #! for HA db ie. REGIONAL set both following to true binaryLogEnabled: false, enabled: false } #! dataDiskType: PD_SSD dataDiskType: PD_HDD #! cheaper ipConfiguration: ipv4Enabled: false privateNetwork: #@ "projects/{}/global/networks/{}".format(data.values.external_gcp_project_id, data.values.external_gcp_poc_openshift_cluster_vpc_network) requireSsl: false databaseFlags: #! Orange security requirement - name: "log_checkpoints" value: "on" - name: "log_connections" value: "on" - name: "log_disconnections" value: "on" - name: "log_lock_waits" value: "on" - name: "log_temp_files" value: "10000" #! 10,000 KB. The security spec isn't providing a value and suggest "0" as default value: If all temporary files are not logged, it may be more difficult to identify potential performance issues that may be due to either poor application coding or deliberate resource starvation attempts. writeConnectionSecretToRef: name: default-name-overriden-by-patch namespace: 75-crossplane-gcp-cnx-secrets #@ end ``` cloudsql-others-data.lib.yaml: ``` #! To be excluded from output, this lib have no yaml document --- header #@ def others(): apiVersion: apiextensions.crossplane.io/v1 kind: Composition spec: resources: - patches: - type: FromCompositeFieldPath fromFieldPath: "metadata.uid" toFieldPath: "spec.writeConnectionSecretToRef.name" transforms: - type: string string: fmt: "%s-postgresql" #! - fromFieldPath: "spec.parameters.storageGB" #! toFieldPath: "spec.forProvider.settings.dataDiskSizeGb" #! Try to implement the service binding spec #! https://github.com/servicebinding/spec#provisioned-service - type: ToCompositeFieldPath fromFieldPath: metadata.labels[crossplane.io/claim-name] toFieldPath: status.binding.name #! To facilitate discoverability, it is RECOMMENDED that a CustomResourceDefinition exposing a Provisioned Service add servicebinding.io/provisioned-service: "true" as a label. - type: ToCompositeFieldPath toFieldPath: metadata.labels[servicebinding.io/provisioned-service] fromFieldPath: status.atProvider.settingsVersion #! try to wait for a field which is only available when the resource is ready transforms: #! first convert int to string - type: convert convert: toType: string #! then format string with zero length - type: string string: fmt: "true%.0s" #! constant string %.0s prints the string with a max zero width. Otherwise gofmt reports #! an error about the argument not being used. #! EXTRA string #! more at https://pkg.go.dev/fmt #! https://github.com/golang/go/issues/8151 #! interactive test https://go.dev/play/ connectionDetails: - type: FromConnectionSecretKey name: host fromConnectionSecretKey: privateIP - type: FromConnectionSecretKey name: username fromConnectionSecretKey: username - type: FromConnectionSecretKey name: password fromConnectionSecretKey: password - type: FromValue name: port value: "5432" - type: FromValue name: type value: "postgresql" - type: FromValue name: provider value: "gcp cloudsql" - type: FromValue name: database #! required by spring-clod-bindings when jdbc-url is missing, see https://github.com/spring-cloud/spring-cloud-bindings#postgresql-rdbms value: "" base: {} compositeTypeRef: apiVersion: dummy kind: dummy #@ end ```

I'm still looking for a solution to reduce proportion of imperative syntax, and reduce starlak editions when adding new resources. The overlays is something I need to explore.

I wonder whether this is something you would be interested in exploring ?

[vfarcic]

I'm not sure I would go deeper myself. For a while now, I switched to cdk8s for managing more complex (tedious) cases like Crossplane packages. The ability to download CRDs and generate libraries that, later on, I can use when constructing packages was a winner for me. It makes working in IDEs like VS Code very pleasant.