search

crossplane / terrajet

Generate Crossplane Providers from any Terraform Provider
https://crossplane.io
Apache License 2.0
290 stars 38 forks source link

Consider making Terraform CLI talk to the single provider server #38

Closed muvaf closed 2 years ago

muvaf commented 3 years ago

What problem are you facing?

Right now we have a single provider binary and all terraform invocations use that one but each of them spins up a new provider server to talk to. We haven't observed noticeable issues but it's likely that this will cause performance issues.

How could Terrajet help solve your problem?

@ulucinar discovered a way to make Terraform CLI talk to the existing provider server. He didn't feel great about it for the initial pass but we might need to reconsider it after testing the providers with 100+ custom resources using composition.

muvaf commented 2 years ago

We need more data about how much of a problem this is.

ulucinar commented 2 years ago

The shared gRPC server packages for provider-jet-azure are ready here: ulucinar/provider-jet-azure-arm64:shared-grpc ulucinar/provider-jet-azure-amd64:shared-grpc

We are still collecting performance data using these images.

muvaf commented 2 years ago

@ulucinar Could you share the branch you're working on as a draft PR if it's in a state where others can compile?

turkenh commented 2 years ago

@haarchri mentioned this PR could be related/helpful to/with this issue: https://github.com/weaveworks/tf-controller/pull/67

ulucinar commented 2 years ago

@ulucinar Could you share the branch you're working on as a draft PR if it's in a state where others can compile?

@muvaf, will do.

Now that we have a provider package implementing a shared gRPC server ready, we need to evaluate the performance of it via a series of experiments.

@turkenh, will check the issue you referenced above, thank you.

muvaf commented 2 years ago

With the results in https://github.com/crossplane/terrajet/issues/233 , I think we are good to go with implementation of this if we're all on the same page. @ulucinar @sergenyalcin thoughts on closing this and opening an implementation issue?

sergenyalcin commented 2 years ago

With the results in #233 , I think we are good to go with implementation of this if we're all on the same page. @ulucinar @sergenyalcin thoughts on closing this and opening an implementation issue?

@muvaf As you said that, using gRPC server-based implementation provides improvements in the context of performance metrics. So, in light of the findings in the test issues, it seems reasonable to close this issue and open a new one for implementation.