Proposal: port forwarding via SSH tunnel

[nicolai86]

proposal: port forwarding via SSH tunnel

I'd like to start adding port forwarding via SSH tunnels to terraform.

This is useful when you want to use terraform with systems which are only accessible via a jump host, ie. company internal systems.

Right now terraform already ships with a bunch of providers which might need to talk to internal systems (e.g. postgres/ mysql/ influxdb/…).

The status quo is to create a SSH tunnel beforehand, or, in cases where the entire infrastructure is created from scratch, to be split terraform scripts into multiple stages with glue code outside. E.g. one might setup a private cluster with a jump host, open an SSH tunnel via bash, and then run a differen terraform script using the newly created tunnel to access private systems, all wrapped in a single setup.sh script.

Assuming that the SSH tunnel is required for all resources of a given provider, I suggest adding connection settings to the terraform providers as well, like this:

provider "consul" {
    address = "localhost:80"
    datacenter = "nyc1"

    # run "ssh -L localhost:80:demo.consul.io:80" for any resources of this provider
    connection {
        user = "private-user"
        host = "private.jump-host.io"

        forward {
            remote_host = "demo.consul.io"
            remote_port = 80
            local_port = 80
        }
    }
}

# Access a key in Consul; consul is only available via SSH tunnel
resource "consul_keys" "app" {
    key {
        name = "ami"
        path = "service/app/launch_ami"
        default = "ami-1234"
    }
}

Looking forward to any feedback before I head of adding something like this to terraform… ;)

Related: #4442, #4775

[shake76]

Hey Guys, I just ran with the same issue when running tf plan in Terraform Cloud with my private EKS endpoint, just wondering if is there any workaround for SSH tunnel with HTTPS_PROXY env

[pjanuario]

@seanamos do you have any example of the AWS SSM usage?

[UXabre]

ssh tunneling doesn't work for many azurerm resources since it doesn't allow to use the URI anymore but instead demands and supports only the resource id to be provided. The only "work-around" is setting up the entire infrastructure on a jump host...which is very involved. I also second a way to "jump" via ssh tunnels or by any other means automatically running terraform on a jump host

[littlejo]

@pjanuario You can find my version: https://github.com/littlejo/terraform-ssm-tunnel

[burizz]

my grand kids will love this feature when it is released (we also use a wrapper shell script to handle tunneling through a bastion host as a workaround - 7 and a half years after this was proposed)

[corkupine]

Any news on the prioritization of this? Glad it's staying open and remaining under consideration at least.

[omarismail]

Hey @corkupine , the Terraform team is researching this problem, and would love to chat with you (and other folks) about it.

Mind reaching out to me (oismail@hashicorp.com) and we can schedule a time to chat live?

[idelsink]

I'm also interested in this! I'm currently trying to manage a Private Kubernetes cluster inside GCP and I want to use an IAP proxy to connect to that cluster.

So far I'm able to install the gcloud SDK in the terraform Cloud runner (using a custom data external script because the terraform-google0gcloud cannot be use in Terraform Cloud; see https://github.com/terraform-google-modules/terraform-google-gcloud/issues/94 ) and start a Socks5 tunnel (using gcloud iap command) to my private kubernetes cluster using the flaupretre/terraform-ssh-tunnel module.

I am able to connect to my cluster during the plan phase, but the background process is I think stopped during the execution phase. On my local machine it works as expected, so for now I'm assuming that the plan and run phase on TF Cloud spawn new instances? (I'm still very much in an experimentation phase)

[sdemjanenko]

@idelsink I was able to do something like with in Terraform Cloud for AWS SSM. The Terraform provider lifecycle makes it so we have to do use a keepalive data source as well as several depends_on statements: https://github.com/ComplyCo/terraform-provider-aws-ssm-tunnels. The benefit of AWS SSM is that their websocket server that runs is built in Go so including it in the provider code was fairly straightforward.

@omarismail I would love providers to have lifecycle hooks so that my provider could be triggered to start before the provider that uses a tunnel and stays up until the provider is done with the tunnel? I currently have to use a keepalive data source to prevent Terraform from terminating my provider which maintains the tunnel.

[apparentlymart]

Thanks for sharing that SSM Tunnels example, @sdemjanenko.

I've been researching and prototyping in this area as part of my work in https://github.com/hashicorp/terraform/pull/35078, and although I was using traditional SSH tunnels as my main motivating example it seems like SSM tunnels are similar enough in concept that they could also be exposed using the "ephemeral resources" concept I discussed over there.

Either your provider or the official hashicorp/aws provider could offer an ephemeral resource type that negotiates an SSM tunnel using a randomly-selected local port number, and then exposes the address for the local listen socket as an attribute so it could be used in a provider configuration, provisioner configuration, or any other context where ephemeral values are accepted:

resource "aws_instance" "example" {
  # ...
}

# INVALID: This is a hypothetical new concept of "ephemeral resources"
# that we're currently considering as part of a solution to this issue.
ephemeral "aws_ssm_tunnel" "mysql_server" {
  instance_id = aws_instance.example.id
  remote_port = 3306
}

provider "mysql" {
  # The local_endpoint attribute would be something like 127.0.0.1:33453,
  # where 33453 is a randomly-selected TCP port number that serves
  # as the local end of the tunnel.
  endpoint = ephemeral.aws_ssm_tunnel.mysql_server.local_endpoint

  # ...
}

In the prototype Terraform would notice that the MySQL provider instance refers to ephemeral.aws_ssm_tunnel.mysql_server and so would "open" an instance of that ephemeral resource before instantiating the provider, and would wait until the MySQL provider's work is all done before "closing" that instance. That directly addresses the problem described in the documentation for the "aws-ssm-tunnels" provider of Terraform not understanding that other parts of the configuration are directly using the tunnel.

The MySQL provider would really be connecting to a listen socket belonging to the hashicorp/aws plugin process, which would then in turn forward to the remote MySQL server, in a similar way as would be true for the stubby prototype SSH tunnels ephemeral resource type I implemented in the aforementioned draft PR.

[seanamos]

@apparentlymart I like the idea. My initial thinking was that having a first class resource/concept for a "connection" in Terraform would be the way to go, however, a more generic "ephemeral" resource might neatly solve an even broader set of cases.

Would an ephemeral resource run during both plan and apply phases? Currently, all the community workarounds to this issue have all sorts of plan/apply shortcomings. For example, with the data source approach (which we currently use):

terraform plan -out=plan # works, connection will open
terraform apply plan # will fail, connection only opened during plan

terraform apply # works

[apparentlymart]

Yes, the general concept of "ephemeral" is to describe anything that exists only temporarily in memory during one Terraform phase, and ephemeral resources in particular represent "objects" (in Terraform's usual fuzzy sense of that word) that ought to be opened when needed and closed soon after they are no longer needed, including repeating that process separately for the plan and apply phases.

As a provider developer one analogy that might resonate is the provider instance itself: those are also "ephemeral" despite us not having used that term to describe them before now. Terraform re-instantiates each provider again in each phase (plan vs. apply) and the provider instance has no "memory" preserved between phases. Ephemeral resources are a similar concept except that the "open" and "close" behaviors are decided by the provider itself, rather than by Terraform Core.

An ephemeral resource type representing a tunnel would therefore implement "open" by opening the connection to the remote service and the local listen socket, and "close" by closing both of those, and those two actions would be repeated for both the plan and apply phases.

[seanamos]

Thanks for the in-depth explanation. After taking a better look at your PR, I can say I'm quite excited for this to land in TF 👍. Can finally get rid our various flaky hacks and workarounds.

[tomalok]

ephemeral "aws_ssm_tunnel" "mysql_server" {
  # This tunnel is used only for provisioning, so is only needed
  # during the apply phase.
  count = terraform.applying ? 1 : 0

  instance_id = aws_instance.example.id
  remote_port = 3306
}```

Note that you might need to have ephemeral resources existing in order to successfully plan what's going to happen during apply...

That is, the tunnel needs to be set up so that you can reach your MySQL instance to pull the current state (refresh, iirc) to determine the deltas for potentially making modifications to the MySQL schema, etc.

[sdemjanenko]

@apparentlymart I like this idea. It would simplify my provider (and make the dev experience way better) + increase reliability dramatically.

[apparentlymart]

@tomalok indeed sorry I was adapting a few different examples I'd already written for other purposes and I forgot to remove the count = terraform.applying ? 1 : 0 part that wasn't relevant here.

Making the ephemeral resource only get opened and closed during the apply phase would be valid for a tunnel to an SSH server used for provisioners, but would not be valid for a provider configuration since (as you say) a provider needs to be instantiated during the plan phase too.

I've edited the earlier comment to remove the declaration that didn't belong. If you're curious you can see in @tomalok's comment what I originally shared, which was incorrect.