Release additional architectures for docker images

[jlosito]

Current Terraform Version

...

Use-cases

Currently, I believe that only amd64 docker images are available. It would be nice if there were some more architectures available especially armv6 and armv7. There's a lot of arm based devices now such as the Raspberry Pi, Pinebook, etc, and it would be nice if I could pull a Terraform docker image for their respective architecture.

Attempted Solutions

Proposal

I think there's a couple different ways that this problem could be approached. For one, I believe TravisCI makes different architectures available, and more recently GitHub runners are available for arm. One could build the images using those methods. I'm not too familiar with the second option, but another option might be to use docker buildx.

References

[apparentlymart]

Hi @jlosito! Thanks for sharing this feature request.

In the very near future we don't plan to change the available docker images, but this could perhaps come eventually. We focus mainly on the normal archive releases on releases.hashicorp.com and the docker images are there as a convenience but we don't have the resources on our team to maintain an increasingly-large surface area of official distribution channels at this time.

In the meantime, you could potentially take the Dockerfile we use to produce the existing "light" images and adapt it to work with a different architecture:

• change the base image from alpine:latest to a similar base image for the architecure you want
• change the archive URL template to say e.g. linux_arm instead of linux_amd64
• change the other later references to the filenames containing linux_amd64 to be e.g. linux_arm instead

With that said, there are currently no official Terraform release packages for linux_arm64, so the above strategy will work only for 32-bit ARM at the moment. 64-bit ARM support is likely to come eventually -- I would expect it to come before docker image support on any other architectures, because the base releases are a prerequisite for the docker images -- but again, we don't have any firm plans to add other architectures at this time.

It's also worth considering that running Terraform in Docker is not as relevant as running some other software, because it's a statically-linked executable with no dependencies other than the Linux kernel system call interface. Therefore if you have an executable built for the architecture in question (either one of our official builds or your own build from source code) then it should be relatively straightforward to run it directly on one of the ARM-based devices you mentioned, without introducing the complexity of Docker.

[kitchen]

FWIW the motivation for my 👍 here is because I'm using drone on a raspberry pi and it would be great to have an official container to use.

[odidev]

@apparentlymart

I am also interested in the official hashicorp/terraform docker image for the Linux/ARM64.

From the master branch, I have successfully built the terraform docker image on local ARM64 machine, executing below command from the root of the ‘hashicorp/terraform’ repo:

docker build -t terraform:test -f Dockerfile .

For the release process, I have checked that this repo earlier had scripts to build and push the docker image to the docker hub. But then those scripts were removed and the release process was moved to the ‘hashicorp/terraform-releases’ repo. But now that the ‘hashicorp/terraform-releases’ repo has been removed, I am not able to figure out the release process for the terraform docker images.

If interested in adding arm64 support, then I am happy to help and contribute.

Can you please point me to where I can find the docker images release process?

[morganchristiansson]

This should be as simple as adding the terraform arm64 binary to a docker image and creating a manifest to point to amd64/arm64 images.. Would be nice!

I also tried to find the build process for the docker images but couldn't see it. circleci doesn't push any images.

[jsleeio]

@apparentlymart wrote:

It's also worth considering that running Terraform in Docker is not as relevant as running some other software, because it's a statically-linked executable with no dependencies other than the Linux kernel system call interface. Therefore if you have an executable built for the architecture in question (either one of our official builds or your own build from source code) then it should be relatively straightforward to run it directly on one of the ARM-based devices you mentioned, without introducing the complexity of Docker.

Yes, but I think this skips over some valid use cases, like Docker-based pipeline steps that use Terraform. This application of Docker is particularly convenient as it makes upgrading particularly trivial. A snippet from my own Buildkite pipeline definition:

steps:
  - label: ":docker: Terraform plan"
    plugins:
      - docker#v3.8.0:
          image: hashicorp/terraform:1.0.3
[...]

Folks using AWS could easily save some cash in their pipeline operations by running their CI agents on ARM instances ... but in examples similar to my own, they'd now need to manage a Terraform binary (whether in a container image or not)

Unrelated to pipelines — I think this issue also deserves re-examination in light of there now being mass-production, consumer-grade ARM64 laptops. My employer issued one to me, for example.

[Razorr1996]

This request has become even more urgent as more and more developers are taking Macs with the Apple M1 chip. Running an amd64 image on such computers is "sometimes" accompanied by a segmentation violation.

[xarbit]

this sucks .. we need arm64 version's

[krissetto]

I'm quite surprised this hasn't been dealt with considering the size and importance of this project.

I would need this for development on M1 based macs and on ARM instances in the cloud for pipelines

[crw]

Hi, please check out my comment on this issue I now realize is a duplicate: https://github.com/hashicorp/terraform/issues/29895#issuecomment-990399114

[coreylane]

I'm looking forward to this. I'd like to use the ARM processors available for AWS Codebuild jobs

[apparentlymart]

Thanks all for your continued interest in this extra Terraform packaging.

For those who wish to use Docker-based systems with ARM workers in the meantime, it might be useful to know that the official Terraform images are not particularly "sophisticated" in comparison to some other software, and are pretty trivial to replicate yourself if you need to. I did hint at this in an earlier comment and linked to an earlier version of our release Dockerfile, which is no longer exactly what we use for the release process but we still follow essentially the same steps:

• Start from alpine:latest.
• Copy the terraform executable from the linux_arm64 package of your desired Terraform CLI release to /bin/terraform.
• Install git and openssh using the Alpine Linux package manager, which is optional but helps with accessing remote services over TLS and with cloning external Terraform modules from Git repository sources.
• Optionally set the entry point to /bin/terraform, if you are intending to use the image in a context where that is useful.

The same thing that makes it relatively easy to run Terraform without Docker (it being a statically-linked executable, as described above) also makes the image build process relatively simple.

---

Naturally it's the automation of such a build and its integration into a larger release pipeline that's the complex part, and that's why we've not been able to do this on our end so far. However, as @crw noted in the other issue the release engineering team at HashiCorp (which provides services for multiple product teams including our team) is working on a next-generation pipeline that's better suited to the problem of building the ever-increasing set of "derived" package types (the .deb/.rpm packages, the homebrew packages, the Docker images, etc) in addition to the primary .zip package releases that have historically been our primary focus. (All of these other ones are really just wrappers around the same executables included in the .zip files, in the interest of consistency no matter which distribution method you use.)

I happened to be working on the parts of that process that are specific to this repository over in #30283 this week, and although there's still a fair amount of non-public-repository work to glue that into the shared services from the release engineering team, we are making gradual progress. What I added there retains our existing practice of only building docker images for linux_amd64, because I'm focused on first replicating what our current release process does, but the docker package build there is ready to have other architectures added to it once there's a downstream process in place to consume them. I'm pausing work on that for the moment while waiting for the rest of the pipeline to be ready, but we'll pick it up again when the supporting parts are in place.

[keitap]

For those who want to run terraform of the latest version on arm64 like Apple M1, here is the minimum Dockerfile (of course you need to build it as arm64 image).

FROM alpine:edge
RUN apk add --no-cache terraform
ENTRYPOINT ["terraform"]

Like @apparentlymart said, add git and openssh package as you need.

If you'd like to cross build docker image, use buildx:

docker buildx build --push --platform linux/amd64,linux/arm64 -t yourtag

[crw]

Similar issue: https://github.com/hashicorp/terraform/issues/27378

[boostrack]

FROM alpine:edge
RUN apk add --no-cache terraform git
ENTRYPOINT ["terraform"]

[ZI-brad-rigby]

This is something we'd also like to see now that all new Apple laptops are ARM.

[bengaywins]

@apparentlymart Any way we can get the above linked PR looked at? This is making things a massive pain for me at my org.

[apparentlymart]

Hi @bengaywins,

Although that GitHub Actions workflow is building docker images, it isn't building the ones that get consumed by our release pipeline and so merging that PR won't do anything to advance the request that this issue represents.

We are intending to pivot to releasing from the artifacts generated by that GitHub Action eventually, which is part of the work I mentioned in my comment above to adopt a new release pipeline. However, we're intentionally keeping the build.yml workflow quite similar to the official release process right now because significant deviations will make it harder to transition from one pipeline to the other when the time is right. We'll add the arm64 build, probably in just the way you proposed, only once the new pipeline is in place and is correctly replicating exactly the results of the current one.

In the meantime, I can only repeat my previous note that wrapping Terraform in a Docker container on macOS is typically just unnecessary complexity compared to running the darwin_arm64 build directly on your main OS, outside of any virtual machines. I do understand that Docker images are often useful for running Terraform in automation pipelines, but for interactive use in a shell I would always recommend using the main releases directly rather than the Docker images, regardless of whether you are on an arm64 system or an amd64 system: Docker tends to add far more complexity than value in that context.

[bengaywins]

In the meantime, I can only repeat my previous note that wrapping Terraform in a Docker container on macOS is typically just unnecessary complexity compared to running the darwin_arm64 build directly on your main OS, outside of any virtual machines.

This response ultimately results in a sucks to be you in a less than ideal situation feeling from the community. And it sucks.

I want to run it on macOS in a container via Docker Desktop because I need to test a pipeline that is failing to run without needing to make 98723489723490572398742390842354789 commits. I usually use the macOS binaries, but this is a case in which I am trying to repro an issue not on my local development machine. I would hazard a guess a lot of others are in a similar situation in this thread.

[Razorr1996]

but for interactive use in a shell I would always recommend using the main releases directly rather than the Docker images

Our case: We have a lot of independent projects with very different providers, modules, and Terraform versions. And we cannot just push all developers to update their local staff. Instead, we're using separate Dockerfile for each project with all-in-box packages, binaries, and libraries. And yes, we have a big percentage of Apple M1/M2 laptops now. It would be nice to have an official linux/arm64 Docker image of Terraform.

[jsleeio]

but for interactive use in a shell I would always recommend using the main releases directly rather than the Docker images

Our case: We have a lot of independent projects with very different providers, modules, and Terraform versions.

Same. Many Terraform stored states of various versions, many developers and many projects. It's getting easier now with 1.x, but until we've actually migrated every last stored state to 1.x we will continue to need older versions — and for that, we need something more flexible than brew install terraform. Docker fills that need nicely.

Also, Linux/arm64 in production.

[clowa]

Hi everyone, I have build a docker image for arm64 containing the official terraform binaries from hashicorp to work around the current limitation to amd64. Every night a pipeline checks if there is a new latest release at the hashicorp/terraform repo, so new versions should be available as docker images quickly. Maybe you want to take a look till the official image supports arm64.

GitHub Repo: clowa/docker-terraform Docker Hub: clowa/terraform

[jbloomer]

This is a blocker for those of us that run the new M1 macbooks as they are arm64 architecture. As of now the current images fail to run on newer macbooks. This definitely needs to be addressed.

[TheRealWaldo]

It's getting harder to get Intel-based Macs. Some of us are forced to use organization-provided laptops, which in my case, is an M1. While I can hack together a pipeline to build these images, it would be nice not to have to!

[keitap]

I hadn't noticed, but arm64 images have been added since version 1.4.3. https://hub.docker.com/r/hashicorp/terraform/tags

[crw]

@keitap correct, as of 1.4.3 we are using a new build system internally that packages arm64 binaries in a variety of packages. Per the original request, I do not know if these builds support armv6 or armv7, so for now I'll leave this ticket open.

[bengaywins]

@crw The tags in the repo seem to explicitly say armv6 and arm64.

[crw]

@bengaywins thanks for pointing that out! I'll close out this issue.

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.