Allow private provider registries to self-host provider packages/checksums/signatures under the same credentials as the registry itself

[BarakHc]

I have a remote Terraform registry that experiencing 401 error on the provider download (download_url, shasums_url and shasums_signature_url). It appears that only for those 3 downloads, the client does not add the authorization header.

I have one domain for the registry API, the modules and the providers files. All the other registry API calls are executed with the relevant domain authorization token, as expected. The modules downloads are executed with the creds in the .netrc file, as expected. But the download of the SHA256SUMS, .sig and provider.zip files are executed with no authorization header at all.

The host of the registry and the download links are the same.
(The token for the domain is updated in the credentials.tfrc.json and the creds are in the .netrc file)

I am working with a reverse-proxy between the client to the registry so I have a record of all the request&responses with their headers.

main.tf file:

terraform {
 required_providers {
   aws = {
     source = "terraform-remote.mydomain.com/hashicorp/aws"
   }
 }
}

module "vpc" {
 source  = "terraform-remote.mydomain.com/terraform-aws-modules/vpc/aws"
 version = "2.21.0"
}

ClientOutTrace.txt

Reverse proxy record + the v1/providers/hashicorp/aws/3.53.0/download/darwin/amd64 response:

Registry call headers (200)

The 401 download request headers:

Terraform version: 1.0.4

[apparentlymart]

Hi @BarakHc! Thanks for reporting this.

I guess this is the provider registry equivalent of #28659, where we were discussing the same behavior for module installation. Again, unfortunately this behavior is intended rather than a bug because from Terraform's perspective the registry is just a pointer to the location for the files, and the files themselves are treated as an entirely separate system.

As in that other issue, I do see the benefit of having a way to treat the package files as if they are "part of the registry" with the same credentials, and I think it would make sense for us to try to solve both of these in the same way so that the opt-in protocol is consistent across both registry types.

However, I'm going to leave both issues open because they do represent to distinct use-cases requiring separate implementation to address, even if the two implementations have some protocol features in common. One difference for provider installation compared to direct-HTTP-based module installation is that, IIRC, provider installation doesn't honor .netrc, and so perhaps there's also an interim solution here of adding NetRC support to the provider installer so that the explicit separate configuration you tried would work, although in the long run I'd still like to have a means for a private provider registry to host packages itself (on its own hostname) without any special configuration on the end-user's part.

Since we're discussing a new feature which needs to be designed I'm going to relabel this as an enhancement just to send it over into our feature request process where we consider changes to Terraform's design, as opposed to Terraform's implementation of the existing design.

As with the other issue, in the meantime the approach for current versions of Terraform will be the same as I suggested in the other issue: make your provider registry generate a URL containing a time-limited, user-specific signature (or some other suitable credential) and then have the server which handles that request verify the credential before returning the package.

Thanks again for sharing this!

[BarakHc]

Hi @apparentlymart , Thank you for the informative answer!

[k0da]

Nice I'm not alone with this use case. My use case is similar but we use github releases as storage backend. Once provider is built GH action triggers Provider registry generator which publishes provider metadata into github pages.

So we're fine to have registry token the same as download token. So far we're on the same page here. I think this could be a good point to start with. Later once another usecase appears with requirement to have different token for download, it could be extended with lets say dowloadToken within host credential structure.

I have implemented and successfully tested my case within https://github.com/hashicorp/terraform/pull/29642 I'd love to see any feedback on taken approach.

[k0da]

Any feedback?

[aaklilu]

@apparentlymart here's the interim solution you suggested