todaygood / openshift-lab

lab on openshift
0 stars 0 forks source link

kb_How_to_trouble_shoot_route #7

Open todaygood opened 6 years ago

todaygood commented 6 years ago

Issue

按照文档,部署了一个ruby-ex业务程序,发现

image

image

Investigation

在Fedora28这个oc客户端上测试,发现不行。

[root@fedora28 ~]# curl -kv ruby-ex-hello.apps.test.cloud.genomics.cn:8080
* Rebuilt URL to: ruby-ex-hello.apps.test.cloud.genomics.cn:8080/
*   Trying 192.168.122.103...
* TCP_NODELAY set
* connect to 192.168.122.103 port 8080 failed: No route to host
* Failed to connect to ruby-ex-hello.apps.test.cloud.genomics.cn port 8080: No route to host
* Closing connection 0
curl: (7) Failed to connect to ruby-ex-hello.apps.test.cloud.genomics.cn port 8080: No route to host
[root@fedora28 ~]# ping  ruby-ex-hello.apps.test.cloud.genomics.cn
PING ruby-ex-hello.apps.test.cloud.genomics.cn (192.168.122.103) 56(84) bytes of data.
64 bytes from ose7.cloud.genomics.cn (192.168.122.103): icmp_seq=1 ttl=64 time=0.324 ms
64 bytes from ose7.cloud.genomics.cn (192.168.122.103): icmp_seq=2 ttl=64 time=0.272 ms
^C
--- ruby-ex-hello.apps.test.cloud.genomics.cn ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1029ms
rtt min/avg/max/mdev = 0.272/0.298/0.324/0.026 ms

注意我访问的是8080端口,实际上是错误的,出现这个错误是因为没有理解route仅支持http, https协议这个知识点。

调试方法:

在普通用户margin下面

[root@ose0 ~]# oc get route 
NAME      HOST/PORT                                   PATH      SERVICES   PORT       TERMINATION   WILDCARD
ruby-ex   ruby-ex-hello.apps.test.cloud.genomics.cn             ruby-ex    8080-tcp                 None
[root@ose0 ~]# oc get service 
NAME      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
ruby-ex   ClusterIP   172.30.150.29   <none>        8080/TCP   4h

没有看到router pod , 需要在admin下面去看。

[root@ose2 ~]# oc get pod --selector=router  -o wide 
NAME             READY     STATUS    RESTARTS   AGE       IP                NODE
router-1-5vr8x   1/1       Running   0          18h       192.168.122.103   ose7.cloud.genomics.cn
router-1-jvztx   1/1       Running   0          18h       192.168.122.44    ose6.cloud.genomics.cn

发现在ose7或者ose6上, 于是dig ruby-ex-hello.apps.test.cloud.genomics.cn 解析为192.168.122.103 就是正确的。

在ose7节点上面发现一切正常。

[root@ose7 ~]# dig ruby-ex-hello.apps.test.cloud.genomics.cn

; <<>> DiG 9.9.4-RedHat-9.9.4-61.el7 <<>> ruby-ex-hello.apps.test.cloud.genomics.cn
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11597
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;ruby-ex-hello.apps.test.cloud.genomics.cn. IN A

;; ANSWER SECTION:
ruby-ex-hello.apps.test.cloud.genomics.cn. 0 IN A 192.168.122.103

;; Query time: 1 msec
;; SERVER: 192.168.122.103#53(192.168.122.103)
;; WHEN: Mon Jul 30 14:32:07 CST 2018
;; MSG SIZE  rcvd: 75
[root@ose7 ~]# curl http://10.131.0.7:8080
成功。
[root@ose7 ~]# curl http://ruby-ex-hello.apps.test.cloud.genomics.cn/
成功。

回到Fedora28上, curl http://ruby-ex-hello.apps.test.cloud.genomics.cn/ 也是成功的。

知识点

router 默认http,https使用的是80,443端口,端口对应http,https协议,这也是文档中强调route只适用于http,https这类协议。

[root@ose0 ~]# oc get route 
NAME      HOST/PORT                                   PATH      SERVICES   PORT       TERMINATION   WILDCARD
ruby-ex   ruby-ex-hello.apps.test.cloud.genomics.cn             ruby-ex    8080-tcp                 None

[root@ose0 ~]# oc get svc
NAME      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
ruby-ex   ClusterIP   172.30.150.29   <none>        8080/TCP   4h

这个是实现了 ruby-ex-hello.apps.test.cloud.genomics.cn:80或者443 <==> service ruby-ex
映射关系。

serivce的访问方式是: 172.30.150.29:8080

[root@ose0 ~]# oc get endpoints NAME ENDPOINTS AGE ruby-ex 10.131.0.7:8080 4h

route hostname (ruby-ex-hello.apps.test.cloud.genomics.cn:80) service (name:ruby-ex)

ClusterIP ( 172.30.150.29:8080 ) | Endpoints ( 10.131.0.7:8080)

参考: https://docs.openshift.org/3.6/admin_guide/sdn_troubleshooting.html#debugging-the-router

todaygood commented 6 years ago

上面web访问失败的原因,是我的环境的问题, broswer所在的机器,所在的环境DNS无法解析ruby-ex-hello.apps.test.cloud.genomics.cn,即使解析到192.168.122.103 了, 网络也是无法访问 192.168.122.103 的, 所以我要搞一个新的windows客户端环境,解决DNS解析和网络互通这两个问题,客户端就可以使用openshift web console了。

todaygood commented 6 years ago

记录,查看router pod 中的配置

  1. 文件1 /var/lib/haproxy/router/routes.json

    "hello:ruby-ex": {
    "Name": "ruby-ex",
    "Namespace": "hello",
    "Host": "ruby-ex-hello.apps.test.cloud.genomics.cn",
    "Path": "",
    "TLSTermination": "",
    "Certificates": null,
    "VerifyServiceHostname": false,
    "Status": "saved",
    "PreferPort": "8080-tcp",
    "InsecureEdgeTerminationPolicy": "",
    "RoutingKeyName": "98b003e2430c51b69f06876c15db0b99",
    "IsWildcard": false,
    "Annotations": {
      "openshift.io/generated-by": "OpenShiftWebConsole",
      "openshift.io/host.generated": "true"
    },
    "ServiceUnits": {
      "hello/ruby-ex": 100
    },
    "ServiceUnitNames": {},
    "ActiveServiceUnits": 1,
    "ActiveEndpoints": 0
    },
    
    "default:docker-registry": {
    "Name": "docker-registry",
    "Namespace": "default",
    "Host": "docker-registry-default.apps.test.cloud.genomics.cn",
    "Path": "",
    "TLSTermination": "passthrough",
    "Certificates": null,
    "VerifyServiceHostname": false,
    "Status": "saved",
    "PreferPort": "",
    "InsecureEdgeTerminationPolicy": "",
    "RoutingKeyName": "172555eec50a0d95563a405b15a8a45f",
    "IsWildcard": false,
    "Annotations": {
      "openshift.io/host.generated": "true"
    },
    "ServiceUnits": {
      "default/docker-registry": 100
    },
    "ServiceUnitNames": {
      "default/docker-registry": 256
    },
    "ActiveServiceUnits": 1,
    "ActiveEndpoints": 1
    },
  2. 文件2 /var/lib/haproxy/conf/haproxy.config 中有:
# Secure backend, pass through
backend be_tcp:default:docker-registry
  balance source

  hash-type consistent
  timeout check 5000ms
  server pod:docker-registry-2-5zpz2:docker-registry:10.130.2.21:5000 10.130.2.21:5000 weight 256

# Plain http backend
backend be_http:hello:ruby-ex
  mode http
  option redispatch
  option forwardfor
  balance leastconn

  timeout check 5000ms
  http-request set-header X-Forwarded-Host %[req.hdr(host)]
  http-request set-header X-Forwarded-Port %[dst_port]
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header Forwarded for=%[src];host=%[req.hdr(host)];proto=%[req.hdr(X-Forwarded-Proto)]
  cookie 98b003e2430c51b69f06876c15db0b99 insert indirect nocache httponly
  server pod:ruby-ex-1-qzzcd:ruby-ex:10.131.0.7:8080 10.131.0.7:8080 cookie 8669b06544a01e34745146110cfa6133 weight 256