Ingress Controller is Intercepting Traffic

SamMorrowDrums commented 7 years ago

Hi, I can connect to the VPN, and then I cannot access internal services by DNS or IP address.

I did hit some IP addresses of pods and get the 'default backend' from Ingress Controller. This leads me to believe the traffic is being treated by kubernetes as external. I'm not sure what to do, but I'm assuming some configuration might be necessary to either allow VPN IP addresses, or make VPN IP addresses use internal range...

I'm using Weave for CNI networking and their default CIDR, which I used as the pod network.

pieterlange commented 7 years ago

Please provide some examples of what you're doing (terminal output). You should be able to connect to pods or services directly using the cluster DNS / cluster IP's. Example (for a service thats running a HTTP service on port 80):

curl -vv $service.$namespace.svc.cluster.local

I haven't tested this with weave but it should behave just like any other CNI.

SamMorrowDrums commented 7 years ago

VPN connected client

curl weave-scope-app.default                  
curl: (6) Could not resolve host: weave-scope-app.default
➜  ~ cat /etc/resolv.conf        
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.1.1
search svc.cluster.local wireless.private.cam.ac.uk
➜  ~ nslookup weave-scope-app.default
Server:     127.0.1.1
Address:    127.0.1.1#53

** server can't find weave-scope-app.default: NXDOMAIN

Inside the VPN container

root@kube-master:/home/sam/kube-openvpn# kubectl -n testing exec -it openvpn-620719487-b43n8 /bin/bash
bash-4.3# curl weave-scope-app.default
bash: curl: command not found
bash-4.3# wget weave-scope-app.default
Connecting to weave-scope-app.default (10.97.5.188:80)
index.html           100% |*******************************|   815   0:00:00 ETA
bash-4.3# cat index.html 
<!doctype html>
<html class="no-js">
  <head>
    <meta charset="utf-8">
    <title>Weave Scope</title>
    <meta name="description" content="">
    <meta name="viewport" content="width=device-width, initial-scale=1">
  <link href="style-app-973dbbbf1a30796b0126.css?dd3864be5377376667e7" rel="stylesheet"></head>
  <body>
    <!--[if lt IE 10]>
      <p class="browsehappy">You are using an <strong>outdated</strong> browser. Please <a href="http://browsehappy.com/">upgrade your browser</a> to improve your experience.</p>
    <![endif]-->
    <div class="wrap">
      <div id="app"></div>
    </div>
  <script type="text/javascript" src="vendors-b1d34bbf6c6ff737d535.js?dd3864be5377376667e7"></script><script type="text/javascript" src="app-973dbbbf1a30796b0126.js?dd3864be5377376667e7"></script></body>
</html>

SamMorrowDrums commented 7 years ago

On VPN Connected client

curl -v weave-scope-app.default
* Rebuilt URL to: weave-scope-app.default/
* Could not resolve host: weave-scope-app.default
* Closing connection 0
curl: (6) Could not resolve host: weave-scope-app.default

On the cluster machine using service IP


root@kube-master:/home/sam/kube-openvpn# curl weave-scope-app.svc.cluster.local
curl: (6) Could not resolve host: weave-scope-app.svc.cluster.local
root@kube-master:/home/sam/kube-openvpn# kubectl get svc
NAME              CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
heketi            10.96.44.171   <none>        8080/TCP   3d
kubernetes        10.96.0.1      <none>        443/TCP    8d
weave-scope-app   10.97.5.188    <none>        80/TCP     8d
root@kube-master:/home/sam/kube-openvpn# curl 10.97.5.188
<!doctype html>
<html class="no-js">
  <head>
    <meta charset="utf-8">
    <title>Weave Scope</title>
    <meta name="description" content="">
    <meta name="viewport" content="width=device-width, initial-scale=1">
  <link href="style-app-973dbbbf1a30796b0126.css?dd3864be5377376667e7" rel="stylesheet"></head>
  <body>
    <!--[if lt IE 10]>
      <p class="browsehappy">You are using an <strong>outdated</strong> browser. Please <a href="http://browsehappy.com/">upgrade your browser</a> to improve your experience.</p>
    <![endif]-->
    <div class="wrap">
      <div id="app"></div>
    </div>
  <script type="text/javascript" src="vendors-b1d34bbf6c6ff737d535.js?dd3864be5377376667e7"></script><script type="text/javascript" src="app-973dbbbf1a30796b0126.js?dd3864be5377376667e7"></script></body>
</html>

``

pieterlange commented 7 years ago

Something is wrong with your (local) DNS configuration.

Can you access the kube-dns service directly? EG dig weave-scope-app.default.svc.cluster.local 10.96.0.10 (assuming 10.96.0.10 is your kube-dns service IP)

Hit me on the kubernetes slack tomorrow so we don't have to slowchat this through github :)

SamMorrowDrums commented 7 years ago

dig weave-scope-app.default.svc.cluster.local 10.96.0.10

; <<>> DiG 9.10.3-P4-Ubuntu <<>> weave-scope-app.default.svc.cluster.local 10.96.0.10
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 1572
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;weave-scope-app.default.svc.cluster.local. IN A

;; Query time: 0 msec
;; SERVER: 127.0.1.1#53(127.0.1.1)
;; WHEN: Wed Mar 15 17:50:06 GMT 2017
;; MSG SIZE  rcvd: 59

;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 511
;; flags: qr aa rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1280
;; QUESTION SECTION:
;10.96.0.10.            IN  A

;; ANSWER SECTION:
10.96.0.10.     0   IN  A   10.96.0.10

;; Query time: 0 msec
;; SERVER: 127.0.1.1#53(127.0.1.1)
;; WHEN: Wed Mar 15 17:50:06 GMT 2017
;; MSG SIZE  rcvd: 55

pieterlange commented 7 years ago

That's not resolving the service through kube-dns but is in fact using your local resolver.

;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 1572

It seems your kube-dns is using a different serviceip.

SamMorrowDrums commented 7 years ago

From inside VPN pod

echo $(cat /etc/resolv.conf | grep -i nameserver | head -n1 | cut -d ' ' -f2)                                                                                                               
10.96.0.10

SamMorrowDrums commented 7 years ago

OK, so perhaps I need to change configuration to manual DNS, I could try adding kubernetes NS to /etc/resolvconf/resolv.conf.d/base and see what happens

SamMorrowDrums commented 7 years ago

There does seem to be some issues here, with the networking. Surely I should be able to access service IPs if I am local in the network.

nslookup weave-scope-app.default.svc.cluster.local 10.96.0.10
;; connection timed out; no servers could be reached

SamMorrowDrums commented 7 years ago

I think somehow my traffic is being firewalled / blocked somewhere in the network when accessing internal IPs... I can for example curl a mongoDB running natively inside the LAN of the kubernetes host machine, on another host. So able to get external traffic to web and on LAN of the kubernetes cluster, but just not IPs of anything internal.

curl 10.13.194.12:27017
It looks like you are trying to access MongoDB over HTTP on the native driver port.

SamMorrowDrums commented 7 years ago

UPDATE

So I've tried this on my old mac, as well as my current Linux machine, and the mac is able to resolve DNS with nslookup weave-scope-app.default.svc.cluster.local 10.96.0.10 and I can then access the given IP address in web browser...

And actually, I was wondering why no additional device was showing in ifconfig, but after some success on the Mac, I used the openvpn CLI, and that has allowed me to at least access IP addresses, and do nslookup with provided IP address, but it still doesn't set the active DNS servers correctly, so both machines don't actually resolve the DNS still.

SamMorrowDrums commented 7 years ago

OK - so many openvpn clients seem not to work fully OOTB! Tunnelblick worked perfectly on OSX. Will need to investigate further, but I am closing this as actually it works!

pieterlange commented 7 years ago

It looks like you're using some linux NetworkManager client that's not working? This works for me, so let me know if you need any more help / if i can add some FAQ entry regarding linux.

pieterlange / kube-openvpn

Ingress Controller is Intercepting Traffic #40

UPDATE