Add `-json` option for `terraform providers` sub-command

deepbrook commented 2 years ago

We're currently doing shell acrobatics to programmatically parse provider versions needed by our terraform configurations.

In the current version of terraform, terraform providers renders a neat human-readable output:

🐟 ❯ terraform providers

Providers required by configuration:
.
├── provider[registry.terraform.io/hashicorp/aws]
├── provider[registry.terraform.io/opsgenie/opsgenie] 0.6.10
└── module.teams
    └── provider[registry.terraform.io/opsgenie/opsgenie] ~> 0.6.10

Providers required by state:

    provider[registry.terraform.io/hashicorp/aws]

    provider[registry.terraform.io/opsgenie/opsgenie]

..with an additional warning, in case we use experiments:

╷
│ Warning: Experimental feature "module_variable_optional_attrs" is active
│
│   on version.tf line 9, in terraform:
│    9:   experiments = [module_variable_optional_attrs]
│
│ Experimental features are subject to breaking changes in future minor or patch releases, based on feedback.
│
│ If you have feedback on the design of this feature, please open a GitHub issue to discuss it.
╵

Extracting the provider versions programmatically from this is at the very least cumbersome.

Current Terraform Version

Terraform v1.2.3
on linux_amd64

Use-cases

We cache plugins for our configurations, and would like to correctly deduce whether or not we need to refresh the cache. Using merely version.tf's checksum is too vague, as we also keep documentation in this and other TF files, which has quite a bit of churn.

Attempted Solutions

We've used the usual shell utils to parse the output of the CLI so far. We've also used them to parse lock and version.tf files. The problem remains the same: the format isn't automation friendly.

Proposal

Add a -json option to terraform provider, which generates the output as a JSON-encoded string:

🐟 ❯ terraform providers -json

{
  "required-by-configuration":
    "registry.terraform.io/hashicorp/aws": "*",
    "registry.terraform.io/opsgenie/opsgenie": "0.6.10"
  "required-by-state": [
    "registry.terraform.io/hashicorp/aws",
    "registry.terraform.io/opsgenie/opsgenie"
  ]
  "experiments": [
    "module_variable_optional_attrs"
  ]
}

In case of sub-modules with specific provider requirements: ideally, terraform merges all required providers and tells me exactly which provider versions work with the modules.

apparentlymart commented 2 years ago

Hi @deepbrook! Thanks for sharing this use-case.

Terraform tracks a couple different things that overlap in scope but are not equivalent:

The provider requirements declared in the configuration, both with explicit required_providers blocks and various other backward-compatibility-oriented implicit detection strategies. This tells us a version constraint (not an exact version) associated with each provider source address.
- Terraform can also determine this separately on a per-module basis, as we see in the terraform providers output, but Terraform in practice doesn't use that information itself and only does so to help users debug conflicting version constraints between modules.
The provider source addresses inferred from provider configuration addresses in the state. In this case we only know the provider's source address and not any version information whatsoever. We also don't currently track this on a per-module basis, although the state format is shaped such that we could in principle... it's just not been useful to do so, since the use-case of resolving conflicting version constraints between modules doesn't apply here.
The final exact provider versions selected by the provider installer after considering the above two items and which versions are available from the configuration installation sources. This is what we track in the .terraform.lock.hcl file.

Can you confirm which of these are important to the automation you are trying to build, and which are not? That will help us evaluate what makes sense to export as JSON, even if it doesn't necessarily exactly match what terraform providers does.

One specific reason for this question -- although we do just generally like to understand the full requirements of a problem before trying to solve it -- is that in your proposal it seems that you already discarded the information about which version constraints belong to each module, which I assume means that your automation doesn't actually need that. But communicating that information is the primary reason terraform providers exists as a separate command, so I have a suspicion that your use-case doesn't exactly match what terraform providers is designed to do and so some other solution might be a better fit.

Thanks again!

deepbrook commented 2 years ago

Hey @apparentlymart thanks for getting back to me this fast!

Essentially, I'd like a computer-readable file I can inspect for changes to provider versions. Something like Pip's requirements.txt, if you will.

I understand this is what the lock.hcl does, am I right? Then I "simply" need it in a machine-friendly format (e.g. JSON)

Incidentally, I tried to run terraform provider lock -json first - only to find out this is just for updating the lock file, not creating it (afait).

As for your question regarding what it is we're building: we keep an extensive cache of plugins, providers and modules. We've solved caching modules by simply caching the backend (.terraform dir), and while our caching for providers works somewhat (we're tracking each and all version.tf files, in the repository and its checksum), it's recently got annoying, because we also have extensive documentation right in the tf files, as well as annotations in comments (including version.tf) - changes to these then trigger the cache to be partially rebuild.

I hope that made sense. Thanks again, I appreciate the effort.

hashicorp / terraform