search

hashicorp / terraform

Terraform enables you to safely and predictably create, change, and improve infrastructure. It is a source-available tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.
https://www.terraform.io/
Other
42.71k stars 9.55k forks source link

Socks5 proxy support #17754

Closed 0just0 closed 8 months ago

0just0 commented 6 years ago

Hi, I'm wondering whether you're planning to add support for socks5 proxy in terraform like Packer has?

It would be very useful to have this feature cause right now I'm struggling with running terraform behind our company's proxy and have to use it through ssh tunnel and bastion host.

I was trying to add socks5 support by myself but faced several issues:

  1. I added new connection properties for proxy to connectionInfo struct in "communicator/ssh/provisioner.go"(I suppose it is the right place for modification) 
    type connectionInfo struct {
/*
some other properties here
*/
SSHProxyHost      string `mapstructure:"proxy_host"`
SSHProxyPort      int    `mapstructure:"proxy_port"`
SSHProxyUser      string `mapstructure:"proxy_user"`
SSHProxyPassword  string `mapstructure:"proxy_password"`
}

    but when I built a binary I got this error -

Error: aws_instance.example: unknown 'connection' argument "proxy_host"

So it looks like terraform can't get new properties from configuration.

I also added almost identical code from packer source to "communicator/ssh/communicator.go"

// ProxyConnectFunc func is a new method for returning a function
// that connects to the host using SOCKS5 proxy like Packer does
func ProxyConnectFunc(socksProxy string, socksAuth *proxy.Auth, network, addr string) func() (net.Conn, error) {
    return func() (net.Conn, error) {
        dialer, err := proxy.SOCKS5("tcp", socksProxy, socksAuth, proxy.Direct)
        if err != nil {
            return nil, fmt.Errorf("Can't connect to the proxy: %s", err)
        }

        c, err := dialer.Dial(network, addr)
        if err != nil {
            return nil, err
        }

        return c, nil
    }
}
trinitronx commented 6 years ago

👍 +1 to this enhancement request!

Just wanted to add some findings here and a use-case to clarify an example networking setup.

My team has found that running Terraform inside a docker container is best in order to ensure every DevOps engineer is using the right version of Terraform for any given infrastructure as code repo. This has the benefits of removing hard dependency on Terraform by just relying on docker, and insulating against Terraform template syntax deprecations and future API breakage (Terraform is still 0.x, so this is likely to happen). However, running anything in a container presents some networking challenges when a process inside the container needs to connect through tunnels to a Bastion Host and finally to a service within an Amazon VPC. Recently, I came across a way to connect from a docker container to a tunnel running on a host. When combined with a SOCKS5h SSH proxy (Note the h, it's important!), this allows common utilities such as curl to access internal VPC services through this proxy + tunnel!

Here is a diagram of the setup:

Docker SOCKS5h Proxy Diagram

The way to set this up is to do the following:

# First, set up docker networking
docker network create -d bridge --subnet 10.1.123.0/22 --gateway 10.1.123.1 bridgenet

# On Mac OSX, this is required to actually access host services via alias IP
sudo ifconfig lo0 alias 172.16.222.111

# Next, set up your SSH tunnel via DynamicForward or -D
# Note that your SSH tunnel tool needs to provide SOCKS5h proxy capability (OpenSSH should, SSH Tunnel.app on Mac also does)
# For example, let's use port 4711 as in the SSH tunneling blog post example
# Note we are using the alias IP for interface lo0 on Mac OSX
ssh -f -N -v -D 172.16.222.111:4711 ssh-bastion-host.example.com
# You should see this line in output:
#    debug1: Local forwarding listening on 172.16.222.111 port 4711

# Next, run a Terraform docker container on bridgenet
docker run -it --rm -u $(id -u):$(id -g) \
    -v $HOME/.aws:$HOME/.aws:ro \
    --net=bridgenet \
    --add-host proxy.local:172.16.222.111 \
    -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -v $HOME:$HOME -e HOME \
    -e TF_DATA_DIR \
    -v $(pwd):/wd -w /wd \
    --entrypoint=/bin/sh hashicorp/terraform:0.11.3

# Check that proxy.local is now set as hostname in the container for the alias IP: 172.16.222.111
cat /etc/hosts
# You should see this line:
#     172.16.222.111    proxy.local

# Now, try accessing an internal VPC service or host via socks5h://
export ALL_PROXY=socks5h://proxy.local:4711; export HTTPS_PROXY=$ALL_PROXY; export HTTP_PROXY=$ALL_PROXY;
curl -v http://your-service.vpc.local
# Optional: Check your WAN Egress IP matches either Public IP of bastion host, or NAT Gateway IP (for private subnets)
curl -v ifconfig.co

Simple proof of concept, but very useful! This can be used or abused so far in Terraform via null_resource & local-exec to call curl commands! I have tested this, and it works!

However, if you need output from a local-exec (e.g. curl -o /tmp/file, then using local_file data provider to read this file... you are blocked by this issue)

Current Terraform Problem

However, Terraform itself has problems when combined with the standard proxy environment variables: ALL_PROXY, HTTP_PROXY, HTTPS_PROXY. So if you were trying to run for example:

docker run -it --rm -u $(id -u):$(id -g) \
    -v $HOME/.aws:$HOME/.aws:ro \
    --net=bridgenet \
    --add-host proxy.local:172.16.222.111 \
    -e ALL_PROXY=socks5h://172.16.222.111:4711 \
    -e HTTPS_PROXY=socks5h://172.16.222.111:4711 \
    -e HTTP_PROXY=socks5h://172.16.222.111:4711 \
    -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -v $HOME:$HOME -e HOME \
    -e TF_DATA_DIR \
    -v $(pwd):/wd -w /wd \
    hashicorp/terraform:0.11.3 plan -var foo=bar -var env=dev  -out dev.plan

Then, Terraform appears to hang indefinitely! Note: This behavior is with aws provider like:

provider "aws" {
  region  = "${var.aws_region}"
  version = "~> 1.10"
}

If you were curious to run with debug output, by setting: -e TF_LOG=TRACE...

Then, you would run into errors such as:

2018-04-19T18:18:00.974Z [DEBUG] plugin.terraform-provider-aws_v1.14.1_x4: 2018/04/19 18:18:00 [DEBUG] [aws-sdk-go] DEBUG: Retrying Request sts/GetCallerIdentity, attempt 8
2018-04-19T18:18:00.976Z [DEBUG] plugin.terraform-provider-aws_v1.14.1_x4: 2018/04/19 18:18:00 [DEBUG] [aws-sdk-go] DEBUG: Request sts/GetCallerIdentity Details:
2018-04-19T18:18:00.976Z [DEBUG] plugin.terraform-provider-aws_v1.14.1_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2018-04-19T18:18:00.976Z [DEBUG] plugin.terraform-provider-aws_v1.14.1_x4: POST / HTTP/1.1
2018-04-19T18:18:00.976Z [DEBUG] plugin.terraform-provider-aws_v1.14.1_x4: Host: sts.amazonaws.com
2018-04-19T18:18:00.976Z [DEBUG] plugin.terraform-provider-aws_v1.14.1_x4: User-Agent: aws-sdk-go/1.13.28 (go1.9.2; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.4
2018-04-19T18:18:00.976Z [DEBUG] plugin.terraform-provider-aws_v1.14.1_x4: Content-Length: 43
2018-04-19T18:18:00.976Z [DEBUG] plugin.terraform-provider-aws_v1.14.1_x4: Authorization: AWS4-HMAC-SHA256 Credential=AKIABZJE2NVQDE93FA6D/20180419/us-east-1/sts/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date, Signature=77b9f317ed7477a2ee84b0b7fae197241744c2177013dbbfe8c778944acb7d9d
2018-04-19T18:18:00.976Z [DEBUG] plugin.terraform-provider-aws_v1.14.1_x4: Content-Type: application/x-www-form-urlencoded; charset=utf-8
2018-04-19T18:18:00.976Z [DEBUG] plugin.terraform-provider-aws_v1.14.1_x4: X-Amz-Date: 20180419T181800Z
2018-04-19T18:18:00.976Z [DEBUG] plugin.terraform-provider-aws_v1.14.1_x4: Accept-Encoding: gzip
2018-04-19T18:18:00.976Z [DEBUG] plugin.terraform-provider-aws_v1.14.1_x4:
2018-04-19T18:18:00.976Z [DEBUG] plugin.terraform-provider-aws_v1.14.1_x4: Action=GetCallerIdentity&Version=2011-06-15
2018-04-19T18:18:00.976Z [DEBUG] plugin.terraform-provider-aws_v1.14.1_x4: -----------------------------------------------------
2018-04-19T18:18:16.316Z [DEBUG] plugin.terraform-provider-aws_v1.14.1_x4: 2018/04/19 18:18:16 [DEBUG] [aws-sdk-go] DEBUG: Send Request sts/GetCallerIdentity failed, will retry, error RequestError: send request failed
2018-04-19T18:18:16.316Z [DEBUG] plugin.terraform-provider-aws_v1.14.1_x4: caused by: Post https://sts.amazonaws.com/: proxyconnect tcp: dial tcp: lookup socks6h on 127.0.0.11:63: no such host
2018-04-19T18:18:16.316Z [DEBUG] plugin.terraform-provider-aws_v1.14.1_x4: 2018/04/19 18:18:16 [DEBUG] [aws-sdk-go] DEBUG: Retrying Request sts/GetCallerIdentity, attempt 9
2018-04-19T18:18:16.317Z [DEBUG] plugin.terraform-provider-aws_v1.14.1_x4: 2018/04/19 18:18:16 [DEBUG] [aws-sdk-go] DEBUG: Request sts/GetCallerIdentity Details:
2018-04-19T18:18:16.317Z [DEBUG] plugin.terraform-provider-aws_v1.14.1_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2018-04-19T18:18:16.317Z [DEBUG] plugin.terraform-provider-aws_v1.14.1_x4: POST / HTTP/1.1
2018-04-19T18:18:16.317Z [DEBUG] plugin.terraform-provider-aws_v1.14.1_x4: Host: sts.amazonaws.com
2018-04-19T18:18:16.317Z [DEBUG] plugin.terraform-provider-aws_v1.14.1_x4: User-Agent: aws-sdk-go/1.13.28 (go1.9.2; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.4
2018-04-19T18:18:16.317Z [DEBUG] plugin.terraform-provider-aws_v1.14.1_x4: Content-Length: 43
2018-04-19T18:18:16.317Z [DEBUG] plugin.terraform-provider-aws_v1.14.1_x4: Authorization: AWS4-HMAC-SHA256 Credential=AKIABZJE2NVQDE93FA6D/20180419/us-east-1/sts/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date, Signature=442ff47df7fe4482797c44c2e777899db41bcf7f4a978b2774224948ae2ff7aa
2018-04-19T18:18:16.317Z [DEBUG] plugin.terraform-provider-aws_v1.14.1_x4: Content-Type: application/x-www-form-urlencoded; charset=utf-8
2018-04-19T18:18:16.317Z [DEBUG] plugin.terraform-provider-aws_v1.14.1_x4: X-Amz-Date: 20180419T181816Z
2018-04-19T18:18:16.317Z [DEBUG] plugin.terraform-provider-aws_v1.14.1_x4: Accept-Encoding: gzip
2018-04-19T18:18:16.317Z [DEBUG] plugin.terraform-provider-aws_v1.14.1_x4:
2018-04-19T18:18:16.317Z [DEBUG] plugin.terraform-provider-aws_v1.14.1_x4: Action=GetCallerIdentity&Version=2011-06-15
2018-04-19T18:18:16.317Z [DEBUG] plugin.terraform-provider-aws_v1.14.1_x4: -----------------------------------------------------

This error repeated many times, as Terraform's aws provider was trying to reconnect many times to https://sts.amazonaws.com/ and failing with a DNS lookup error. However, the DNS name it was trying to look up was socks5h. This indicated a problem in the parsing of the HTTPS_PROXY or ALL_PROXY URLs. The requests were to use the GetCallerIdentity method "Action" against AWS STS. This was during the AWS provider plugin's data gathering steps. Basically, it was just trying to determine who the owner of AWS_ACCESS_KEY_ID was in IAM by using STS on AWS API. These requests were being signed by AWS v4 signing protocol via Authorization: AWS4-HMAC-SHA256 headers for secure authentication and protection against replay attacks. This is why it's ok that I post the exact request logs... you cannot replay these because timestamps are signed and hashed along with the exact request info (however, I have randomized them further for fun).

I tested one last connection using curl to the failing URL:

curl -kv https://sts.amazonaws.com/
*   Trying 205.251.243.54...
* TCP_NODELAY set
* Connected to sts.amazonaws.com (205.251.243.54) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-SHA
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=sts.amazonaws.com
*  start date: Nov 10 00:00:00 2017 GMT
*  expire date: Nov 10 12:00:00 2018 GMT
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: sts.amazonaws.com
> User-Agent: curl/7.57.0
> Accept: */*
>
< HTTP/1.1 302 Found
< x-amzn-RequestId: 9ed462f6-440e-11e8-9da7-516d7e5cf072
< Location: https://aws.amazon.com/iam
< Content-Length: 0
< Date: Thu, 19 Apr 2018 20:17:01 GMT
<
* Connection #0 to host sts.amazonaws.com left intact

This indicated success in that curl was able to connect via *_PROXY settings, doing DNS lookup after the proxy (on Bastion Host), and finally connect to sts.amazonaws.com to get a 302 Found response!

So, this test seemed to indicate a problem with the GoLang Terraform binary itself! It could also indicate some issue in GoLang, or the specific library or connection method that it Terraform is using.

So, although I feel like we have made 99% progress towards solving the Docker container to VPC tunneling issue, the problem now is just with the Terraform tool itself. This is most likely due to the known issue in Golang's x/net/proxy or "Dialer" libraries. There is no support yet for these standard *_PROXY variables using socks5h://. Also, without the h form of socks5h:// protocol, terraform cannot resolve internal VPC DNS names through the proxy such as internal VPC Route53 private zone records.

The good news is that whenever this issue is solved upstream in Golang, we should be able to use a future version of terraform built with this SOCKS5h support to tunnel correctly!

Potential Path Forward

There is light at the end of the tunnel! (Pun intended)

There is an upstream bug in Golang to ask for socks5h:// support in x/net/proxy (golang/go#13454). If this is ever fixed, perhaps Terraform providers and code that uses standard x/net/proxy library will just work!

0just0 commented 6 years ago

@jbardin, I would like to know if there any progress on this issue? :)

jbardin commented 6 years ago

Hi @0just0, Any updates on the issue would show up referenced here.

Thanks for detail @trinitronx. There's really 2 separate problems here. The original issue is to add socks5 support to the provisioner, which is possible for ssh and only blocked for now because of other core working being done at the moment. This should be fairly straightforward since that tcp connection is controlled by terraform itself, and limited to a single process.

The second http proxy issue which you highlighted, which is the more difficult issue, is that terraform can't control all http connections from every sdk. While the core of terraform can control its http clients for various internal usage (downloading modules, fetching provider binaries, provisioner communication), there's no way to add that support to all possible dependencies, nor is there a way to force providers to use the proxy if it's possible at all. The only complete solution for socks5h would need to come from upstream.

trinitronx commented 6 years ago

@jbardin : Thanks for the info! Good to know it's possible for provisioner to gain support via SSH! That's one of the main use cases for the need to proxy: running terraform outside a VPC, then SSH proxy through Bastion Host into the VPC.

Regarding the Golang socks5h:// support, I just wanted to add another interesting data point:

So it's possible to do this in a way similar to the kubectl binary, but obviously the better option would be to have full socks5h:// support in Golang core libraries.

jbardin commented 6 years ago

@trinitronx,

Though I haven't verified the code, k8s is most likely able to do it with kubectl because it communicates only with kubernetes from the cli process. If they have control over all the clients and their name resolution, then situation is far easier.

As mentioned earlier, terraform can only directly control the clients used in the main binary. The providers would each have to independently setup their own clients to use the custom proxy (which may not be possible with all SDKs, though the overall situation is improving). It's not unsurmountable but it is a large task to cover over many independent codebases.

Because larger organizations tend to run terraform in automation, the infrastructure can usually be built to accommodate terraform, rather than trying to work around limitations in the infrastructure, so this isn't something that is frequently encountered.

I understand you may have a use case for this type of proxy, but since adding socks5 proxy support to the provisioner and adding socks5h support globally are quite different, this particular feature request is going to strictly be for provisioners. Feel free to open a separate feature request for full socks5h support though, with the info you've provided here.

trinitronx commented 6 years ago

Just wanted to circle back on this because I ran into another use case today for using terraform-provider-http as data source to access a URL behind a SOCKS5 proxy (through SSH Tunnel).

I did some research today and found that some upstream GoLang changes regarding SOCKS5 have happened since this original issue was posted. So far it looks like this is intended to replace the old implementation, but still supports only socks5:// scheme. The first question is whether this implementation supports DNS lookups through the SOCKS5 proxy, so I did some digging.

At first glance, it looks like as long as the proxy URL begins with socks5:// scheme then it should trigger the code above in transport.go to use socksNewDialer() and DialWithConn() functions that are implemented by the new SOCKS5 code. Additionally, the call to DialWithConn() passes the target address as the fourth address string argument cm.targetAddr. Inside that function, we can see it then passes this through to connect().

Next, this gets split via sockssplitHostPort() into host, and port. Finally, it checks if this host portion is an IPv4, IPv6, or FQDN type address. If it's not an IP, then it adds socksAddrTypeFQDN (Socks type = 0x03) to the byte array it sends for address type.

So it appears to be aware of FQDN / DNS names now... just without socks5h:// scheme on the protocol specifier.

Testing data.http provider with environment variables set to use socks5:// still doesn't seem to work as of hashicorp/terraform:0.11.8 version and * provider.http: version = "~> 1.0". So the question remains what changes are necessary to enable terraform-provider-http to use SOCKS5 via *_PROXY environment variables?

maxsono commented 4 years ago

This feature would be incredibly helpful when configuring services that run in VPC and are only reachable from within the VPC (e.g. k8s services).

I tried to socksify terraform but this did not work. Probably because the configuration uses a 3rd-arty plugin which is run using a clone() syscall:

clone(child_stack=0x7fccbbe9ef70, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tid=[572621], tls=0x7fccbbe9f700, child_tidptr=0x7fccbbe9f9d0) = 572621

I believe this is were socksify stops working.

EDIT:

The command that I used:

SOCKS5_SERVER=127.0.0.1:9999 strace socksify terraform plan
marsteel commented 4 years ago

I use proxychains-ng

cd /usr/local/etc vi proxychains.conf

proxychains4 terraform init proxychains4 terraform plan

castironclay commented 3 years ago

I was able to start proxying via socks5h by starting a sock5 proxy (tor in my case) then updating my /etc/hosts file as well as setting some ENV variables. I've verified Terraform is in fact using the proxy by monitoring iptables rules and tcpdump. This consistently works for me. Hope this helps. Tor listener on 9050

tor --HTTPTunnelPort 127.0.0.11:80
export ALL_PROXY=socks5h://127.0.0.1:9050
export HTTPS_PROXY=socks5h://127.0.0.1:9050
export HTTP_PROXY=socks5h://127.0.0.1:9050

Add to /etc/hosts I know it says 127.0.0.11. I found that 127.0.0.11 is what Terraform is looking for. Maybe this is a native Linux thing? I'll admit after I got this working I still didn't fully understand why it worked. 

127.0.0.11 socks5h
simoncpu commented 2 years ago

Greetings from the future! I'm a time traveler from the year 2022. SOCKS5 support would be great, and I'm unable to compile proxychains-ng on an M1 CPU.

sorenisanerd commented 1 year ago

Contrary to what the documentation may lead us to believe, this already works. https://developer.hashicorp.com/terraform/language/resources/provisioners/connection#connection-through-a-http-proxy-with-ssh shows how to use an HTTP proxy. Just set the scheme to "socks5".

trinitronx commented 1 year ago

Contrary to what the documentation may lead us to believe, this already works

Hmm... maybe something has changed since years ago? It would be interesting to check if also socks5h:// (e.g. DNS lookups through the proxy) is working properly now too? (note that the protocol scheme socks5:// may not do this automatically depending on the implementation).

crw commented 1 year ago

If you check out the PR generously provided by @sorenisanerd, it links to the code in the go library that seems to indicate that socks5h is also supported at some level. I am not an expert on this so it would still need to be verified.

github-actions[bot] commented 7 months ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.