S3 backend should not require region when endpoint given

[OJFord]

Terraform Version

0.14.2

Terraform Configuration Files

backend "s3" {}

Expected Behavior

region should not be required if endpoint is specified.

Actual Behavior

region is required, despite endpoint specified, and is validated by default, as are credentials despite the validation not necessarily being appropriate for the given endpoint.

Steps to Reproduce

1. terraform init -backend-config=endpoint=s3.us-west-002.backblazeb2.com -backend-config=skip_credentials_validation=true -backend-config=skip_region_validation=true -backend-config=region= -backend-config=profile=foobar -backend-config=bucket=foo -backend-config=key=bar

Additional Context

The above reproduction is a minimal init for a backend "s3" {} that's really a B2.

Due to region validation, giving a true but useless us-west-002 is not allowed.

Due to region being Required: true, -backend-config=region= (blank) is still needed with -backend-config=skip_region_validation=true.

Due to AWS STS credential validation, -backend-config=skip_credentials_validation=true will always be needed for non-AWS endpoints.

The remaining configuration (profile, bucket, and key, in my case) is as expected. There's an argument for the validation still making sense with endpoint too - I don't have a good handle on the prevalence of VPC endpoints vs. non-AWS endpoints and how likely they are to be used as Terraform state backends, so this issue is just about the region really.

[apparentlymart]

Thanks for raising this, @OJFord!

Based on the current implementation it isn't clear to me what the design intent was here, and it seems like there are some design decisions to weigh to decide what action to take here, and so that makes this more suited to our feature request process (changing the design to match a new need) rather than our bug report process (changing the implementation to better match the design intent). I'm going to relabel it with that in mind.

The S3 backend is typically maintained primarily by the AWS provider team rather than the Terraform Core team, because generally the S3 backend interface tries to follow the example set in the AWS provider for consistency, and so I'm also going to label this as relating to that backend in the hope that the provider team will have an opportunity to reflect on it when they have some time.

I should note for completeness that we generally consider the support for non-Amazon reimplementations of AWS services as a "best effort" sort of thing and not a high priority for work on the backend. That might mean that this will sit for a while before being considered, but the AWS provider team will be the ones mainly driving the priority here. However, what you've suggested here does seem reasonable to me on the surface, so hopefully there will be a way to make it more convenient to work with unofficial S3 reimplementations in the way you described.

[OJFord]

Just a note to add that since I opened this, an official Backblaze provider has appeared, though unfortunately I notice it will not be able to provide a backend, since:

The built-in backends are the only backends. You cannot load additional backends as plugins.

That's unfortunate, but I just mention it really in case there's any planned work to allow backend plugins; that might be an alternative way for this issue to be resolved.

[TechIsCool]

It should be noted that you can set skip_region_validation = true and then define the regions that are used by Backblaze. It also seems that Terraform doesn't mind it even being a different region defined as the endpoints are specific to each datacenter at Backblaze.

[OJFord]

@TechIsCool That is noted in OP; there I used an obviously fake empty region, it just needs to specified.