Problem with port-forward

[sstubbs]

I have installed yugabyte with the helm chart and it works well. I can access it when I'm on the same network with the loadbalancer service. I would rather not expose this service publicly for security reasons.

When I try port forward to the tserver pod or to the loadbalancer service I am getting the following error:

E0604 11:51:13.445374   15898 portforward.go:400] an error occurred forwarding 5433 -> 5433: error forwarding port 5433 to pod 3aeddedb837eba9946ed42dfed7f74784ea5458c391dbff48e63afed02b2c99c, uid : failed to execute portforward in network namespace "/var/run/netns/cni-5f2b2eec-0835-14c4-1475-0a0755326dd4": socat command returns error: exit status 1, stderr: "2020/06/04 10:51:13 socat[2108153] E connect(5, AF=2 127.0.0.1:5433, 16): Connection refused\n"

Port forwarding is working to other pods and services so I think it is probably something to do with this chart.

[iSignal]

Hello @sstubbs, thanks for reporting this issue. This is a known limitation due to the way kubectl port-forward works. port-forward setups a socat proxy that binds to 127.0.0.1:remote_port inside the pod. In the case of our chart, the client interface is not bound to 127.0.0.1 inside the pod (it is bound the to pod IP instead).

To work around this, one option would be to change the helm chart to bind to 0.0.0.0:5433 at https://github.com/yugabyte/charts/blob/master/stable/yugabyte/templates/service.yaml#L290. If you make that change in a local helm chart, that should get your port-forward workflow unblocked.

We are going to consider making this change in our helm chart by default too.

Edit: relevant issue on k8s: https://github.com/kubernetes/kubernetes/issues/72597

[sstubbs]

Hi,

I've had some time to test this again. That works thank you. Is there anybody else who has had this issue? It would be great if this could be the default. I would think a lot of people work with port-forward but I could be wrong.

[sstubbs]

Works great for cqlsh too if I change that to 0.0.0.0. Thanks. A value in the chart to alter this would be great if the default can't be changed.

[sstubbs]

If anyone else has this issue and want all 3 connection types: alter these to: "--cql_proxy_bind_address=0.0.0.0" "--pgsql_proxy_bind_address=0.0.0.0:5433" and add this "--redis_proxy_bind_address=0.0.0.0"

[iSignal]

This should work for SQL port 5433 after https://github.com/yugabyte/charts/commit/ea342e49de39732bffcce338ac0acfb9a9e60b42

[sstubbs]

@iSignal great thanks. I understand yedis is not being developed anymore but would it be possible to also enable this for cql by default? I would rather not have to alter this chart everytime I reinstall/upgrade.

[sstubbs]

Neverimnd I see istioCompatibility which I need anyway enables this.

[hongbo-miao]

Alan helped me answer the solution regarding how to port-forward YugabyteDB Web UI and YSQL API (5433) and YugabyteDB Web UI (7000) when installed by operator at https://forum.yugabyte.com/t/how-to-port-forward-correctly/1181/3

Hopefully save some time for further people! 😃

[cesarqdt]

will this be part of any release of the chart? I'm also having issues when port-forwarding to the tserver, can't connect to the database.

kubectl port-forward service/yb-tservers 5433 -n yb-production
Forwarding from 127.0.0.1:5433 -> 5433
Forwarding from [::1]:5433 -> 5433
Handling connection for 5433

docker run -it --rm -v $(pwd)/certs/:/root/.yugabytedb/:ro yugabytedb/yugabyte-client:latest ysqlsh -h localhost "sslmode=require"
ysqlsh: could not connect to server: Connection refused
    Is the server running on host "localhost" (127.0.0.1) and accepting
    TCP/IP connections on port 5433?
could not connect to server: Cannot assign requested address
    Is the server running on host "localhost" (::1) and accepting
    TCP/IP connections on port 5433?