Feature Request: Specify SSH key for module source so can pull the key from Vault

[jgeorgeson]

Current Terraform Version

Terraform v0.11.13

Use-cases

I'm setting up atlantis in containers and my modules are stored in private Git repos.

Attempted Solutions

Seems there are three options currently:

1. Put private key unencrypted inside container along with an SSH config file
2. Put private key in container and run ssh-agent in container (key could be encrypted but would need to expose the passphrase to the container, which could be done with Vault)
3. Open key in ssh-agent on host, and expost agent to container by mounting the socket file as a volume

Proposal

Add private_key_content parameter, so we can fetch the key from Vault

data "vault_generic_secret" "module_source_key" {
  path = "secret/module_source_key"
}

module "vpc" {
  source = "git::https://example.com/vpc.git"
  private_key_content = "${data.vault_generic_secret.module_source_key[content]}"
}

References

• https://www.terraform.io/docs/modules/sources.html#generic-git-repository

[apparentlymart]

Hi @jgeorgeson! Thanks for this feature request.

We recommend that all credentials needed to run Terraform be provided "ambiently" in the environment, rather than via Terraform configuration itself. This has a number of advantages:

• The Terraform configuration is completely agnostic to what context it is being run in: the automation around Terraform makes sure there are SSH credentials available somehow, but Terraform doesn't need to know how exactly that is acheived. The same configuration can then be applied locally on someone's workstation (e.g. during testing, or in an emergency when automation is unavailable), in your current automation, and in potential future automation that may use a different strategy.

• Once you've configured credentials for git/SSH, those credentials are available for any software that runs git or ssh on that host, rather than having to configure everything separately. (This isn't always important, but nice to have that flexibility if e.g. you need to run ssh commands directly in local-exec sometimes, or similar.)

In your case, I'd be inclined to write a small script that fetches the SSH key from Vault and generates ~/.ssh/id_rsa (or similar, if you are using a different key algorithm or if you want to customize this in the .ssh/config file) before running Terraform. Then when terraform init runs git to install modules, those credentials should be picked up automatically.

Note that terraform init is literally just running git here, so it doesn't directly control where that process looks to find credentials or how it runs SSH. We don't really want to get into the business of configuring git and SSH through Terraform, because git and SSH both already have flexible solutions for dealing with configuration.

[mkielar]

Then when terraform init runs git to install modules

However it seems that git does not use the local .git/config when terraform runs it.

My case:

• I have several rsa keys in my ~/.ssh. For the sake of this example, let's assume I have work.id_rsa and private.id_rsa
• I don't have the default id_rsa at all.
• When I initialize new GIT repository, i do:

  git config core.sshcommand 'ssh -i ~/.ssh/work.id_rsa -F /dev/null'

  This way git knows to use the key when I run it from this repository.

I hoped doing this will do the same for terraform, but no. Terraform is running git in a way that it does not use the "local" git config, and only uses the "global" one, and that fails pulling module sources on non-public git repositories.

I understand why putting the payload of the key into terraform files would ba a bad idea, but perhaps at least being able to specify path path to KeyFile would be an option (e.g. using Env Variable, or otherwise).

Or: make git run with "working directory" pointing inside the terraform module directory, so it could use the core.sshcommand setting in local config.

Currently, there seems to be no way to do this other than having ~/.ssh/id_rsa.

[sakalys]

Like @mkielar, I also am facing a similar issue. I have multiple private keys in ~/.ssh. Furthermore I don't have the luxury to edit the source code of the "module" directive – I'm stuck with one line defining the source:

module "staging" {
    source = "git::https://bitbucket.org/...

Running terraform get tries to fetch the modules using the default id_rsa, but I must use a different one. I also tried prefixing the command with an ENV override GIT_SSH="ssh -i <path to rsa>" terraform apply, but that did have any effect. Anyone else knows how to fix that?

[wayneworkman]

Wanted to add my use case to this feature request. At my place of work, we are required to use individual SSH deployer keys per private GitHub repo that needs accessed. We've put together a duct-tape and band-aids solution to accommodate this for Terraform modules using private GitHub repos, but would prefer to just specify the SSH key to use for accessing the repo on a per-module basis.

[apparentlymart]

Git itself has a very flexible (but also complex) set of features for dealing with various different authentication-related requirements. In the most general case, it supports running an arbitrary external program to obtain credentials, and (according to the input/output format documentation) provides that program with both the host and the repository path that credentials are being requested for, so it seems like it can support returning different credentials per repository even if all repositories have host=github.com.

I understand that this isn't the same as providing an SSH key -- Git's credentials mechanism is for username/password auth rather than SSH auth, because SSH auth is the SSH client's responsibility rather than Git's -- but GitHub does support using use-case-specific OAuth tokens with basic auth over HTTPS and doing that would simplify the problem by taking SSH out of the mix and therefore only needing to deal with Git's mechanisms.

As I mentioned before, we are unlikely to try to replicate parts of Git or SSH's authentication story within Terraform itself. Both of those systems have elaborate and complex configuration mechanisms of their own, and any subset we might try to reimplement is likely to only make it harder to use the features we don't support.

The note above about using non-global Git settings is something we could potentially pursue as a middle ground, but I feel a little unsure as to how it would work, since Terraform is running git clone with the intention of creating an entirely new repository on local disk, and so before it does that there isn't yet any local repository for you to create a .git/config directory inside. However, I'd like to hear more about how that approach works with the package managers from other programming language ecosystems and then perhaps we can adopt a similar compromise in Terraform.