--to-pod doesn't work for Cloud SQL auth proxy sidecar

[manterfield]

I'm running a service with a Cloud SQL auth proxy sidecar on GKE.

When I intercept using telepresence intercept app_name --port 3000:80 --to-pod 5432 --env-file .env my local app instance isn't able to connect to the DB. Logs say connection failed and also that it was dropped unexpectedly (though it never seems to have been made in the first place).

If I instead connect without --to-pod 5432 and separately run a vanilla port-forward kubectl port-forward deployment/app_name 5432:5432 the local app starts up and connects to the DB successfully.

Looking at the logs for traffic-agent I can see repeated lines like: 2022-12-02 17:06:12.0836 error sidecar/dialWait : !! CONN tcp 10.128.0.0:50086 -> 10.128.0.75:5432, failed to establish connection: dial tcp4 10.128.0.75:5432: connect: connection refused

To Reproduce I'm not sure I can sensibly give a minimal reproduction here as it requires quite a few pieces (GCP project with Cloud SQL, Auth proxy container etc etc)

Telepresence version 2.9.3 (installed via brew install datawire/blackbird/telepresence-arm64)

Expected behavior

I'd expect --to-pod to effectively do the same thing as port-forward for that port on that pod. I'm not sure what it is actually doing instead.

[omidb]

I have the same problem as @manterfield with different sidecar.

[manterfield]

@omidb Are you on the same version as me? I'm on 2.9.3.

I'm trying some older versions now to see if they work, but it's a bit awkward since I can't change the cluster side on a whim.

[manterfield]

Looking into this, I think it's because the traffic-agent isn't connecting via localhost/127.0.0.1

My sidecar only accepts connections from 127.0.0.1 but it looks like traffic-agent is connecting from 10.128.0.75.

Changing Cloud SQL proxy argument to this: -instances=$(INSTANCE_CONNECTION_NAME)=tcp:0.0.0.0:5432 has allowed it to connect.

@omidb I'd check if your sidecar also explicitly allows specific hosts.

---

To the Telepresence team, is there any chance of having this changed so this isn't required and traffic to sidecars appears as local?

This workaround is tolerable, but we are changing our network config purely to accommodate Telepresence. I can see scenarios where that's not possible and it's a bit of a gotcha.

If it can't be changed easily, then some additional docs on this would be really handy.

[omidb]

Thanks @manterfield , in my case I'm using dapr and I don't have fine-grained access to change the -instances as Cloud SQL.

[thallgren]

@manterfield , this is a bug for sure. The traffic-agent must let the connection originate from 127.0.0.1 when it originates from a --to-pod. We'll have to fix that.

[omidb]

I also could figure out how to change dapr sidecar to accept traffic from 0.0.0.0 and it started to work

[alexeys-poshtech]

Same experience as manterfield for some time now - is this issue still on-going?

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment, or this will be closed in 7 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 7 days with no activity.