Ability to download providers as compressed files for slow connections

OmpahDev commented 2 years ago

Current Terraform Version

...1.2.0

Use-cases

I live in a rural area with low quality internet, where my signal is around 2-5 megabits per second. I've had numerous problems where running "terraform init" takes 20-30 minutes, and sometimes even times out, because of the huge amount of time it takes to download all the providers (azurerm, aws, etc.) from hashicorp on my slow connection.

There are lots of people, both in rural areas of the developed world, and in the developing world, who are stuck with connections like mine or even slower ones. In my case, until Starlink launches in my area, I have literally zero ability to have a faster connection no matter how much I pay.

To make terraform a more usable product for us, create some way to allow compressed/zipped versions of providers to be downloaded and then inflated after downloading is complete.

Attempted Solutions

None

Proposal

I honestly don't know.

References

apparentlymart commented 2 years ago

Hi @tdevopsottawa! I'm sorry to hear that you are having trouble installing Terraform providers.

The provider packages for official providers (the ones we at HashiCorp are able to control directly) are already delivered as .zip packages with compression. I verified that they are indeed compressed by downloading the latest AWS provider release (v4.14.0 at the time I'm trying this) and comparing the size of the .zip package with the size of the plugin executable inside it:

 54M terraform-provider-aws_4.14.0_linux_amd64.zip
254M terraform-provider-aws_v4.14.0_x5

The archive that terraform init is downloading is 20% of the size of the plugin executable packed inside of it, and so my sense is that this is already compressed well and that unfortunately compression alone won't be sufficient to make these packages small enough to download quickly on a slower internet connection, or on an internet connection with high packet loss. I don't have any immediate ideas for any other ways we could make these files smaller while retaining the full functionality of the plugin inside.

However, if you are able to download the providers you need once then there are already some options available to you in today's Terraform, if you create a CLI configuration file and use it to customize some of Terraform's provider installation settings:

You can configure a provider plugin cache directory. If you do so, whenever terraform init needs to install a particular provider it will first check the cache directory to see if there's a package already present there, and it'll reuse the cached package if so. If there is no package present, it'll first download it into the cache directory and then install from the cache.

This is the mechanism most directly focused on situations like yours, since you should be able to set it up just once and let Terraform automatically grow the cache as needed. The first time you use a new package that you haven't previously downloaded you'll still need to wait to download it, but once that download succeeds you should not need to download it again.
You can reconfigure provider installation methods to force Terraform to always look in a local directory only, and not to look at origin registries for packages at all. This is a more extreme answer that will require some active management on your part to decide what to place in the directory, but it comes with the advantage that terraform init will only consider what you already have installed as versions available for installation, and so you can avoid getting into a situation where terraform init learns that a newer version is available than what you had in your cache and therefore insists on downloading that new version rather than using what's cached.

Specifically for the goal I described above where Terraform only looks at a local directory, you'd have a provider_installation block with only a filesystem_mirror block, intentionally omitting the direct {} block that would normally permit automatic installation from an origin registry:
```
provider_installation {
  filesystem_mirror {
    path = "/home/example/.terraform/providers"
  }
}
```
The documentation for the filesystem_mirror installation method describes what directory structures Terraform will expect to find in that directory, so you can download the packages by whatever means is convenient to you and then place them at the expected path yourself. As an extreme example, if you are able to travel temporarily to a place with a faster or more reliable internet connection and download the packages you want to a portable storage device, then you could bring those files to your normal location and create the filesystem layout Terraform is expecting. Of course, that would be necessary only if it were totally impractical to download these packages once over your normal internet connection, and it sounds like you are able to do that except that sometimes you experience errors and need to retry.

I hope that helps! I don't really know what else we would do here in order to make this situation work better by default on unreliable or slow internet connections, but if we are able to identify a practical path to do so then I expect we would pursue it.

crw commented 2 years ago

Thanks for this issue report @tdevopsottawa! Based on the above answer, I am going to close this issue. Please let me know if you believe me to be closing this issue in error. Thanks again for the report!

github-actions[bot] commented 2 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / terraform