Question ch10 - port-forwarding is not working

[nicewook]

Hello, This is HS Jeong from South Korea. SW engineer.

I went through the book and got stucked on ch 10. I found your repo and tried to test your code for ch 10 but failed Could you give me any guide?

My situation

1. Windows WSL2 - Ubunto 20.04
2. Changed git head to 690314c - your ch10 complete commit
3. Did all the way to helm install proglog deploy/proglog and checked all three pods running okay using $ kubectl get pods
4. port-forwarding and run cmd/getservers/main.go - but failed.

Terminal output

$ go run cmd/getservers/main.go 
2022/06/02 17:49:28 rpc error: code = Unavailable desc = connection closed before server preface received
exit status 1

$ kubectl port-forward pod/proglog-0 8400
Forwarding from 127.0.0.1:8400 -> 8400
Forwarding from [::1]:8400 -> 8400
Handling connection for 8400
E0602 17:49:28.379357   10869 portforward.go:406] an error occurred forwarding 8400 -> 8400: error forwarding port 8400 to pod d871ac34568f6a8d9a87107d4ecca4fb5fd1c4988110266b83a70e42cd1645a6, uid : failed to execute portforward in network namespace "/var/run/netns/cni-00ea3bcd-7544-2ddc-97d0-4414947404b1": failed to connect to localhost:8400 inside namespace "d871ac34568f6a8d9a87107d4ecca4fb5fd1c4988110266b83a70e42cd1645a6", IPv4: dial tcp4 127.0.0.1:8400: connect: connection refused IPv6 dial tcp6 [::1]:8400: connect: connection refused 
E0602 17:49:28.379731   10869 portforward.go:234] lost connection to pod

[evdzhurov]

Hello, as far as I remember port forwarding also didn't work for me. I think it might be related to the fact that the pods are using fully qualified domain names for binding the services and you might actually need to also listen on the localhost to perform the forwarding (I'm speculating on that). I remember trying to modify the code so I can bind the grpc server on 0.0.0.0 (any) port but that broke the example code. So I moved on. This didn't cause any issues when deploying to Google Cloud later.

[nicewook]

What a precious information. Thanks a lot. actually I'm on translation job of this book to Korean. Many thanks to you.

[nicewook]

FYI, I found this link, though it's not working neither. If I found any clue I will share here. : https://itnext.io/exposing-statefulsets-in-kubernetes-698730fb92a1#6fc0

[nicewook]

Solved!

My friend got the issue point. The issue is that net.Listen() is listening cluster IP and port. If we want the servet to listen the port forwarded request to connect, then we shouldn't specify IP. Let me share my fixed code

func (a *Agent) setupMux() error {
    // addr, err := net.ResolveTCPAddr("tcp", a.Config.BindAddr)
    // if err != nil {
    //  return err
    // }
    // rpcAddr := fmt.Sprintf(
    //  "%s:%d",
    //  addr.IP.String(),
    //  a.Config.RPCPort,
    // )
    rpcAddr := fmt.Sprintf(":%d", a.Config.RPCPort)
    ln, err := net.Listen("tcp", rpcAddr)
    if err != nil {
        return err
    }
    a.mux = cmux.New(ln)
    return nil
}

[evdzhurov]

Ok, thanks.

[khun84]

@evdzhurov your speculation https://github.com/evdzhurov/dist-services-with-go/issues/1#issuecomment-1144644223 is correct. The pod (or container) is not listening localhost on that port hence the network traffic from port forwarding is rejected.

> kubectl exec -it proglog-0 -- sh                                                                                                                            ✔  2.5.1   k3d-k3s-default ⎈  09:38:56
Defaulted container "proglog" out of: proglog, proglog-config-init (init)
/go # netstat -nltp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 10.42.0.114:8400        0.0.0.0:*               LISTEN      1/proglog
tcp        0      0 10.42.0.114:8401        0.0.0.0:*               LISTEN      1/proglog

I think kubectl destination ip for port-forward is always localhost and there isn't any way to specify remote host in kubectl port-forward command (origin host is supported though). There is a discussion issue on this in kubernetes github https://github.com/kubernetes/kubernetes/issues/72597

In case someone is not inclined to change setMux to bind to port only (without ip), you can still achieve port forwarding with the method described in the following link.

https://github.com/kubernetes/kubernetes/issues/72597#issuecomment-693149447

Basically you need to spin up another pod as a network relay (socat) to achieve the same. You will need to provide the pod ip kubectl get pod pod-name --template '{{ .status.IP }}' as socat tcp-connect target ip. IMO this can be abit far-fetched for some ppl but it stands out to be a more generic solution in some cases when its not convenient/possible/safe for you to change source code/config.

cc @nicewook

[bxffour]

Hey, I had the same problem but I used a plugin to solve this. I wanted to add this to the list of possible solutions. https://github.com/knight42/krelay

[cschar]

@bxffour what was the command you used? kubectl relay service/proglog 8400:rpc ?

[cschar]

@nicewook @evdzhurov were you guys able to write a simple go program similar to getservers/main.go that would instead of listing out consumers, actually write data to the program?

I have a simple program here , but running it locally against my kind instance crashes

package main

import (
    "context"
    "flag"
    "fmt"
    "log"
    api "github.com/cschar/proglog/api/v1"
    "google.golang.org/grpc"
)

func main() {
    addr := flag.String("addr", ":8400", "service address")
    flag.Parse()
    conn, err := grpc.Dial(*addr, grpc.WithInsecure())
    if err != nil {
        log.Fatal(err)
    }
    client := api.NewLogClient(conn)
    ctx := context.Background()
    req := &api.ProduceRequest{
        Record: &api.Record{
            Value: []byte("foo"),
        },
    }
    fmt.Println("Attempting to send produce request...")
        # crashes proglog-0 pod here
    produceResponse, err := client.Produce(ctx, req)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println("got produce response with")
    fmt.Printf("%d \n", produceResponse.Offset)

}