Open schmichael opened 4 years ago
port-forward but for tooling purposes, we can just deploy util job then exec and play around there, but if you want port-forward for connections to different cluster better deploy envoy or another proxy
I really wanted this feature this past Sunday after deploying a service with an HTTP API I didn't want to expose to my SSH bastion, or any other networked services within my cluster, or fiddle with additional load balancers, or edge router configs. I found myself using nomad alloc exec
to get a shell within the running task. But, the container didn't include a lot of tooling I needed to interact with the service, and often times those tools are already installed or available in a local container on my host.
I just wanted to forward the traffic from a local listener on my host to an upstream service available to the task.
laptop-browser → nomad-cli-listener → load-balancer → nomad-server → nomad-client → nomad-driver → nomad-task → nomad-alloc → upstream-service-listener
So, I went down a fairly deep Nomad-shaped rabbit hole to understand how to make this a reality. Along the way, I learned a lot -- like, how it actually sort'of already exists, and how it could work better in the future. 🕳🐇
nomad alloc exec
It's already possible to wrap nomad alloc exec
to facilitate what effectively gives you a port-forward command, but with three requirements:
nomad
CLI has been configured with an ACL token (with alloc-exec
permissions) if ACLs are enabled, with any other environment variables set to be able to connect, authenticate, and authorize access to the cluster.socat
(or equivalent byte-facilitator) so that you can run that command within the execution context of our target task in order to pass bytes through STDIN to whatever upstream service available available for the container and recv more back from STDOUT.nomad
command wrapper (like the one below) that can start a listener which sends those bytes to the STDIN of the nomad alloc exec
command, and then read the response out of its STDOUT back to the client.A shell's STDIN/STDOUT can just be a TCP proxy.
package main
import (
"flag"
"fmt"
"log"
"net"
"os"
"os/exec"
"strings"
)
func main() {
task := flag.String("task", "", "task name if alloc contains multiple")
socatPath := flag.String("socat-path", "/usr/bin/socat", "path to socat binary in task")
portMap := flag.String("p", "", "port mapping local_port:remote_port")
flag.Parse()
args := flag.Args()
if len(args) != 1 {
log.Fatalf("expected 1 alloc argument given %d", len(args))
}
portMapParts := strings.Split(*portMap, ":")
if len(portMapParts) != 2 {
log.Fatalf("expected 2 parts (local_port:remote_port) for -p flag, given %d", len(portMapParts))
}
ln, err := net.Listen("tcp", fmt.Sprintf("localhost:%s", portMapParts[0]))
if err != nil {
log.Fatalf("failed to create local listener: %v", err)
}
defer ln.Close()
log.Printf("started local server: %v", ln.Addr())
for {
conn, err := ln.Accept()
if err != nil {
log.Fatalf("failed to accept new connection: %v", err)
}
log.Printf("accepted new connection: %v", conn.RemoteAddr())
go func(conn net.Conn) {
defer conn.Close()
defer log.Printf("closed connection: %v", conn.RemoteAddr())
argsStr := fmt.Sprintf("alloc exec -i -t=false -task=%s %s %s - TCP4:localhost:%s", *task, args[0], *socatPath, portMapParts[1])
log.Printf("running command: nomad %s", argsStr)
cmd := exec.Command("nomad", strings.Split(argsStr, " ")...)
cmd.Stdin = conn
cmd.Stdout = conn
cmd.Stderr = os.Stderr
err = cmd.Run()
if err != nil {
log.Printf("nomad exec command error: %v", err)
return
}
}(conn)
}
}
$ go run main.go -p 3100:3100 -task=$TASK_NAME $ALLOC_ID
2021/04/09 16:55:35 started local server: 127.0.0.1:3100
2021/04/09 16:55:37 accepted new connection: 127.0.0.1:60777
2021/04/09 16:55:37 running command: nomad alloc exec -i -t=false -task=promtail 0d253bda /usr/bin/socat - TCP4:localhost:3100
2021/04/09 16:55:39 closed connection: 127.0.0.1:60777
$ curl http://localhost:3100/config
...
☝️ From my laptop I can now curl
the upstream Loki service running in a different Nomad task available to the promtail
task exposed through an Envoy sidecar managed by Consul on localhost:3100
within the container running promtail
. It's localhost turtles all the way down.
From an ACL security perspective, this means that alloc-exec
is essentially alloc-net
(or whatever it will be named) unless you explicitly prevent applications like socat
from running in your cluster, which could be done in various ways.
nomad alloc port-forward
While an exec port-forward is one solution, I don't think it's ideal. For one, it would require having socat
installed on any container/vm/host running your alloc/task. It also doesn't allow you to cleanly separate TCP port-forward access from exec access, even if those lines can be still be blurry for various reasons.
I started chipping away at figuring out all the pieces to support a "native" port forwarding experience following @notnoop's work in https://github.com/hashicorp/nomad/pull/5632, since it's so incredibly similar. I have some working tests, RPC endpoints, and task driver changes.
I plan to make a PR in the near future, and look forward to feedback!
Just wanted to chip on @picatz awesome existing workaround.
If someone doesn't want to use Go, it's possible to use socat
on your local machine too:
socat tcp4-listen:1234,reuseaddr,fork system:'(nomad alloc exec <job/task params> -i socat - TCP4:yourhost:1234)'
Kubernetes has a port-forward command that allows operators to dynamically and ephemerally forward a local port to a remote pod for debugging and other operations.
This feature seems particularly useful when using Consul Connect as Connect's mTLS requirements make it difficult for operators to peek at Connectified services.
Implementation
Port forwarding should use Nomad's existing region-aware RPC infrastructure to allow forwarding ports across regions.
Implementation on the client-side (CNI? driver specific?) is TBD.
Security
A new ACL capability would be added:
namespace:alloc-net
(name TBD). While port forwarding offers a similarly high level of container access asnamespace:alloc-exec
, this feature should have a distinct ACL to avoid having to give operators remote execution privileges.