ark3
commented
6 years ago Your Mac ends up in some sort of broken state such that DNS resolution of cluster resources with vpn-tcp fails, and you have to restart the machine to fix things. Is that right? The next time this happens, could you please link to a gist of telepresence.log? Please try curl, dig, etc., so the DNS lookups are visible in the log.
If you have to redact the logfile contents, it may be easier to extract the sshuttle and pod logs. See the trace below for an example.
Thank you for your help!
$ telepresence
Starting proxy with method 'vpn-tcp', which has the following limitations: All processes are affected, only one telepresence can run per machine, and you can't use other VPNs. You may need to add cloud hosts with --also-proxy. For a full list of method limitations see https://telepresence.io/reference/methods.html
Volumes are rooted at $TELEPRESENCE_ROOT. See https://telepresence.io/howto/volumes.html for details.
No traffic is being forwarded from the remote Deployment to your local machine. You can use the --expose option to specify which ports you want to forward.
Guessing that Services IP range is 10.3.240.0/20. Services started after this point will be inaccessible if are outside this range; restart telepresence if you can't access a new Service.
@gke_datawireio_us-central1-a_telepresence-testing|bash-4.4$ curl -sk https://kubernetes/api/
{
"kind": "APIVersions",
"versions": [
"v1"
],
"serverAddressByClientCIDRs": [
{
"clientCIDR": "0.0.0.0/0",
"serverAddress": "35.184.xx.xx"
}
]
}@gke_datawireio_us-central1-a_telepresence-testing|bash-4.4$
@gke_datawireio_us-central1-a_telepresence-testing|bash-4.4$ dig kubernetes
; <<>> DiG 9.9.7-P3 <<>> kubernetes
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1260
;; flags: qr ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;kubernetes. IN A
;; ANSWER SECTION:
kubernetes. 0 IN A 10.3.240.1
;; Query time: 109 msec
;; SERVER: 2001:558:feed::1#53(2001:558:feed::1)
;; WHEN: Fri Apr 06 09:14:20 EDT 2018
;; MSG SIZE rcvd: 44
@gke_datawireio_us-central1-a_telepresence-testing|bash-4.4$ host kubernetes
kubernetes has address 10.3.240.1
kubernetes has address 10.3.240.1
Host kubernetes not found: 3(NXDOMAIN)
@gke_datawireio_us-central1-a_telepresence-testing|bash-4.4$
@gke_datawireio_us-central1-a_telepresence-testing|bash-4.4$ exit
exit
$ fgrep "Launching: sshuttle" telepresence.log
20.5 TEL | [35] Launching: sshuttle-telepresence -v --dns --method auto -e 'ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -F /dev/null' --to-ns 127.0.0.1:9053 -r telepresence@localhost:58044 10.0.125.0/24 10.0.126.0/24 10.3.240.0/20 10.0.127.0/24 10.0.19.0/24
$ fgrep "35 |" telepresence.log
20.8 35 | Starting sshuttle proxy.
21.0 35 | firewall manager: Starting firewall with Python version 3.6.5
[...]
$ fgrep logs telepresence.log
12.7 TEL | [25] Launching: kubectl --context gke_datawireio_us-central1-a_telepresence-testing --namespace default logs -f telepresence-1523020367-938933-52094-1060178697-lcx4v --container telepresence-1523020367-938933-52094
$ fgrep "25 |" telepresence.log
19.4 25 | Listening...
19.4 25 | 2018-04-06T13:13:09+0000 [-] Loading ./forwarder.py...
19.4 25 | 2018-04-06T13:13:10+0000 [-] /etc/resolv.conf changed, reparsing
19.4 25 | 2018-04-06T13:13:10+0000 [-] Resolver added ('10.3.240.10', 53) to server list
[...]
40.6 25 | 2018-04-06T13:13:31+0000 [stdout#info] A query: b'kubernetes'
40.6 25 | 2018-04-06T13:13:31+0000 [stdout#info] AAAA query, sending back A instead: b'kubernetes'
40.6 25 | 2018-04-06T13:13:31+0000 [stdout#info] A query: b'kubernetes'
40.6 25 | 2018-04-06T13:13:31+0000 [stdout#info] Result for b'kubernetes' is ['10.3.240.1']
40.6 25 | 2018-04-06T13:13:31+0000 [stdout#info] Result for b'kubernetes' is ['10.3.240.1']
esorey
alexisvincent
el-davo
amarchen
I'm opening a telepresence session to our K8s cluster using method
vpn-tcpwithout swapping out any deployments just to access K8s resources. Roughly half of the time I do this, it works perfectly. The other times, I get errors complaining that DNS resolution of the K8s addresses failed. However, when I rundig <K8s-IP>, it reports status NOERROR, so I know that there is no real issue with the addresses themselves. The only workaround I've found thus far is to restart my machine entirely. I've also tried the same setup on Linux and have not seen the issue after many runs. Specifically, this is on macOS Sierra 10.12.6. This seems like a particularly hairy issue that may be solved by the plans to run DNS locally, but I still wanted to document it.Thank you!