search

pulumi / pulumi-terraform-bridge

A library allowing Terraform providers to be bridged into Pulumi.
Apache License 2.0
199 stars 43 forks source link

Dynamically bridged provider replay tests cleverer log matching strategy #2571

Open VenelinMartinov opened 3 weeks ago

VenelinMartinov commented 3 weeks ago

What happened?

The replay tests for dynamically bridged providers use a simple strategy of matching logs to requests by assuming the order is consistent.

We can do something cleverer if we need to like matching the inputs.

See https://github.com/pulumi/pulumi-terraform-bridge/pull/2484#discussion_r1823093165

Example

.

Output of pulumi about

.

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

t0yv0 commented 3 weeks ago

I would push on this. It does not have to be clever, the default strategy can match on request inputs. The current strategy matches requests based on the request type and random ordering position is not acceptable and will lead to lost time failing to reproduce issues and debugging non-deterministic replay request ordering.

For more complicated cases there probably should be a hook, CustomMatchers or some such, for the owner of a concrete test scenario to add some code to customize how the logs are matched to the incoming requests. This seems to be a common refrain in libraries that are build this way, see for example:

https://github.com/dnaeon/go-vcr?tab=readme-ov-file#custom-request-matching

VenelinMartinov commented 3 weeks ago

Yeah, that sound sensible. Do we have an example of when this will be needed? Something around multiple resources created in parallel perhaps?

VenelinMartinov commented 3 weeks ago

I would also like to see the framework get any usage, frankly, before investing more here. It's currently only there as a proof of concept

t0yv0 commented 3 weeks ago

This will be needed any time you have anything non-trivial. The requests aren't coming in-order.

t0yv0 commented 3 weeks ago

ACK.