Convert all singleton lists in the MR APIs to embedded objects

ulucinar commented 2 months ago

Description of your changes

Depends on: https://github.com/crossplane/upjet/pull/387, https://github.com/crossplane/upjet/pull/400

Terraform configuration blocks, even if they have a MaxItems constraint of 1, are (almost) always generated as lists. We now generate the lists with a MaxItems constraint of 1 as embedded objects in our MR APIs.

This also helps when updating or patching via SSA the (previously list) objects. The merging strategy implemented by SSA requires configuration for associative lists and converting the singleton lists into embedded objects removes the configuration need.

The provider generates the converted embedded objects in the new v1beta2 versions of the CRD APIs and registers upjet's identity converter and the singleton list converters to be invoked, in chain, by the CRD API conversion webhooks. This implies that the v1beta1 versions stay intact, and old clients not aware of the new APIs should continue functioning as before (backwards-compatibility). Any clients willing to use the embedded objects-based APIs should be updated to use the v1beta2 versions of the CRD APIs.

If a resource contains no singleton lists, then the v1beta2 version is not generated for it.

We had to configure field name overrides for HTTP{S}HealthCheck.compute and Version.dialogflowcx so that the correspnding type names for the spec.forProvider, spec.initParameter and state.atProvider fields are not duplicated with the introduction of v1beta2 packages.

We have also removed the SSA merge strategy configurations from the v1beta2 versions of Cluster.container & NodePool.container resources because the related fields are no longer lists and have been converted to embedded objects in their v1beta2 versions.

I have:

[x] Read and followed Crossplane's contribution process.
[x] Run make reviewable to ensure this PR is ready for review.
[x] Added backport release-x.y labels to auto-backport this PR if necessary.
[x] New versions for the converted CRD APIs
[x] Replace config.Resource.References[*].Type with config.Resource.References[*].TerraformName
[x] Adjust the manually authored example manifests: With https://github.com/crossplane-contrib/provider-upjet-gcp/pull/528
[x] API Conversion functions (invoked by the Conversion webhooks) to convert between the new & old API versions

How has this code been tested

A successful uptest run for the 'NodePool.container` resource with the API converters is registered here. Another run with GKE back plans is here.

Also via the following uptest runs:

ulucinar commented 2 months ago

/test-examples="examples/container/v1beta2/nodepool.yaml"

ulucinar commented 2 months ago

/test-examples="examples/container/v1beta2/nodepool.yaml"

ulucinar commented 2 months ago

/test-examples="examples/container/v1beta2/nodepool.yaml"

ulucinar commented 2 months ago

/test-examples="examples/container/v1beta2/nodepool.yaml"

ulucinar commented 2 months ago

/test-examples="examples/container/v1beta2/nodepool.yaml"

ulucinar commented 2 months ago

/test-examples="examples/container/v1beta2/nodepool.yaml"

ulucinar commented 2 months ago

/test-examples="examples/container/v1beta2/nodepool.yaml"

ulucinar commented 2 months ago

/test-examples="examples/gke/v1beta2/backupbackupplan.yaml"

ulucinar commented 2 months ago

/test-examples="examples/gke/v1beta2/backupbackupplan.yaml"

ulucinar commented 2 months ago

/test-examples="examples/gke/v1beta2/backupbackupplan.yaml"

ulucinar commented 2 months ago

/test-examples="examples/gke/v1beta2/backupbackupplan.yaml"

ulucinar commented 2 months ago

/test-examples="examples/container/v1beta2/nodepool.yaml"

ulucinar commented 2 months ago

Hi @JonathanO, This PR converts the singleton lists in the provider API to embedded objects together with the converter functions invoked by the conversion webhooks to convert between the (old) v1beta1 APIs (where the APIs are singleton lists) and the new v1beta2 APIs (where the APIs are embedded objects).

We've done some preliminary tests using the {Cluster,NodePool}.eks, ServiceAccount.cloudplatform and BackupBackupPlan.gke resources using uptest. Also built resource providers using the v1.1.0-db914f4f8174cbf845f9df9895b9826ed26d2e6c tag.

It would be great if you could provide us some feedback using these providers:

Family config provider: xpkg.upbound.io/upbound/provider-family-gcp:v1.1.0-db914f4f8174cbf845f9df9895b9826ed26d2e6c
Monolith: xpkg.upbound.io/upbound/provider-gcp:v1.1.0-db914f4f8174cbf845f9df9895b9826ed26d2e6c

As discussed above, the resource providers (except for the new containerattached provider) have also the same tag v1.1.0-db914f4f8174cbf845f9df9895b9826ed26d2e6c, i.e., xpkg.upbound.io/upbound/provider-gcp-container:v1.1.0-db914f4f8174cbf845f9df9895b9826ed26d2e6c.

This is a relatively large & involved change (both the code generation pipelines and the provider runtime are affected and we also attempt to hide the breaking API changes from the v1beta1 clients). I would really appreciate if you could give these providers a try in your test environment if you can spare some time.

Not all v1beta2 example manifests have been updated to reflect the API changes. However, if you'd like to check here are some manifests updated to facilitate testing:

examples/gke/v1beta2/backupbackupplan.yaml
examples/container/v1beta2/{cluster,nodepool}.yaml
examples/gke/v1beta2/backupbackupplan.yaml

P.S. There's also a counterpart of this PR in crossplane-contrib/provider-upjet-azuread.

turkenf commented 2 months ago

/test-examples="examples/bigquery/v1beta1/routine.yaml"

turkenf commented 2 months ago

/test-examples="examples/compute/v1beta1/nodegroup.yaml"

turkenf commented 2 months ago

/test-examples="examples/dns/v1beta1/policy.yaml"

turkenf commented 2 months ago

/test-examples="examples/sql/v1beta1/user.yaml"

turkenf commented 2 months ago

/test-examples="examples/secretmanager/v1beta1/secret.yaml"

turkenf commented 2 months ago

/test-examples="examples/secretmanager/v1beta1/secret.yaml"

Uptest run: https://github.com/crossplane-contrib/provider-upjet-gcp/actions/runs/8787492987

turkenf commented 2 months ago

/test-examples="examples/sql/v1beta1/user.yaml"

turkenf commented 2 months ago

/test-examples="examples/dns/v1beta1/policy.yaml"

Uptest run: https://github.com/crossplane-contrib/provider-upjet-gcp/actions/runs/8787497642

turkenf commented 2 months ago

/test-examples="examples/bigquery/v1beta1/routine.yaml"

Uptest run: https://github.com/crossplane-contrib/provider-upjet-gcp/actions/runs/8787501144

ulucinar commented 1 month ago

/test-examples="examples/container/v1beta2/nodepool.yaml"

ulucinar commented 1 month ago

/test-examples="examples/gke/v1beta2/backupbackupplan.yaml"