Terraform stability for release

[lilatomic]

Terraform is nearing being ready for a release from experimental. That will mean that we have all the functionality in place to provide a stable user frontend and decent user experience. There are things left to do:

• [ ] change options scope download-terraform -> terraform. Probably do as we exit experimental since it too will be a breaking change
• [ ] some way of ejecting from pants, so you can run manual steps. For example, sometimes you'll need to upgrade state or migrate backends (which has come up a few times). There are also importing resources and inspecting state. Pants doesn't have to have knobs for all of these, but can help in many ways (such as pants export mapping to terraform init; or pants run trying to pass in vars files)
• [ ] perf: the TF backend is currently really slow because it doesn't cache providers. There are 2 opportunities to speed this up:
  • [ ] caching downloaded providers, so we don't have to download them each time
  • [ ] decoupling sources and required providers by using an intermediate of required providers to enable caching. Currently we just throw all sources at terraform init, and that's technically required and always valid. But it does mean that changes which don't affect the required providers still cause us to reinitialise terraform which takes a wile
• [ ] dependencies that need to be packaged/published first. Maybe we want to make this a default for all experimental_deployable things?
• [ ] pull in resources (not just files)

[tdyas]

some way of ejecting from pants, so you can run manual steps. For example, sometimes you'll need to upgrade state or migrate backends (which has come up a few times). There are also importing resources and inspecting state. Pants doesn't have to have knobs for all of these, but can help in many ways (such as pants export mapping to terraform init; or pants run trying to pass in vars files)

See https://github.com/pantsbuild/pants/pull/13023 for a PR which would have added a tf-run goal to invoke Terraform with specified arguments.

[tdyas]

the TF backend is currently really slow because it doesn't cache providers

Are the provider downloaded to the sandbox currently? If so, can Terraform be configured to cache them elsewhere?

[tdyas]

https://developer.hashicorp.com/terraform/cli/config/config-file#provider-plugin-cache seems relevant for configuring a shared Terraform plugin cache.