search

pulumi / pulumi-terraform-bridge

A library allowing Terraform providers to be bridged into Pulumi.
Apache License 2.0
199 stars 43 forks source link

Support re-rolling unrolled recursive TF types #1468

Open iwahbe opened 1 year ago

iwahbe commented 1 year ago

Hello!

Issue details

The terraform type system is structural, meaning that subtypes are not named. This prevents terraform from describing recursive types. Terraform providers fake recursion by manually unrolling recursive types to a given depth, generating a huge number of types in the process.

Pulumi needs to generate an in-code representation of these types leading to PRs like this:

Screenshot 2023-10-24 at 2 34 46 PM

Pulumi's type system is nominal (mirroring most programming languages), and we do support recursive types. Recursive typing is worse in Pulumi then it is in TF (for bridged providers) because of the translation, but Pulumi providers can actually offer a better experience than TF providers here.

Solution

The bridge should have native support for re-rolling recursive types back into their recursive descriptions.

A MVP implementation enables designating a resource type on ProviderInfo as "unrolled recursive" on specific fields. The bridge would then handle re-rolling that type.

An ideal solution would automatically detect deep nesting, and then re-roll without manually marking as recursive. This would require inferring that a type is recursive, and on what fields it recurses on, then calling into the MVP solution.

Backwards compatibility

The MVP is fully backwards compatible, since it is off by default. Automatic detection is technically breaking, but we can configure it so that it only breaks providers with an unusable number of resources (wafv2 style).

Affected area/feature

Prior Art

iwahbe commented 1 year ago

This effectively blocks https://github.com/pulumi/pulumi-datadog/pull/359, unless we want to ship a massive performance regression.

iwahbe commented 1 year ago

Looks like this is also blocking https://github.com/pulumi/pulumi-akamai/pull/314.

t0yv0 commented 10 months ago

Curious @iwahbe if you want to hand it over to me or @VenelinMartinov or still planning to finish out. Exciting feature.

iwahbe commented 10 months ago

Curious @iwahbe if you want to hand it over to me or @VenelinMartinov or still planning to finish out. Exciting feature.

I'm going to continue next iteration. I'd love some help finishing, it's a thornier technical issue then I expected.

t0yv0 commented 2 months ago

There's some related work I've been trying to wrap up. There are coupled problems.

  1. detect recursion emulation
  2. detect type replicas
  3. remove type replicas

This PR has reasonably good recursion detector, but it is written against PackageSpec instead of shim.Schema:

https://github.com/pulumi/pulumi-terraform-bridge/pull/2134

The next PR explores non-recursive type replica detection and removal at scale. It turns out scale makes things difficult-ish. The evolving approach is to have tooling that detects replicas and then has maintainer approve of them and pick names. Approval step is helpful to eliminate accidentally shared types that will evolve separately upstream. Naming is necessary to avoid super long tokens. Working on shim.Schema level instead of PackageSpec level seems advantageous for this use case.

https://github.com/pulumi/pulumi-aws/pull/4449

This PR landed to give a mechanism for curtail type explosion at tfgen level:

https://github.com/pulumi/pulumi-terraform-bridge/pull/2409

iwahbe commented 1 month ago

https://github.com/pulumi/pulumi-akamai/issues/366 is an instance of recursive types (and the subsequent compilation resource explosion).

t0yv0 commented 1 month ago

The PR is not quite ready but I learned a lot from prototyping, I'd like to pitch this sometime soon-ish on this to move forward.