search

ucbrise / clipper

A low-latency prediction-serving system
http://clipper.ai
Apache License 2.0
1.4k stars 280 forks source link

Hang several hours:[clipper_admin.py:124] Clipper still initializing. #537

Open xuguangzhao opened 6 years ago

xuguangzhao commented 6 years ago

before clipper is correctly running. because access to private registry is need configed for each node, so when i have done for registry ,I restart docker at each node. so i deploy the clipper in k8s cluster again. but when i exec clipper_admin.start_clipper()， log is always printing Clipper still initializing（the state has several hours）：

In [1]: from clipper_admin import ClipperConnection, KubernetesContainerManager

In [2]: clipper_conn = ClipperConnection(KubernetesContainerManager(useInternalIP=True))

In [3]: clipper_conn.start_clipper() 18-07-11:08:00:42 WARNING [kubernetes_container_manager.py:231] No external node addresses found. Using Internal IP address 18-07-11:08:00:42 INFO [kubernetes_container_manager.py:244] Found 1 nodes: 192.168.13.79 18-07-11:08:00:42 INFO [kubernetes_container_manager.py:253] Setting Clipper mgmt port to 31238 18-07-11:08:00:42 INFO [kubernetes_container_manager.py:261] Setting Clipper query port to 31048 18-07-11:08:00:42 INFO [kubernetes_container_manager.py:276] Setting Clipper metric port to 30965 18-07-11:08:00:42 INFO [clipper_admin.py:124] Clipper still initializing. 18-07-11:08:00:43 INFO [clipper_admin.py:124] Clipper still initializing. 18-07-11:08:00:44 INFO [clipper_admin.py:124] Clipper still initializing. 18-07-11:08:00:45 INFO [clipper_admin.py:124] Clipper still initializing. 18-07-11:08:00:46 INFO [clipper_admin.py:124] Clipper still initializing. 18-07-11:08:00:47 INFO [clipper_admin.py:124] Clipper still initializing. 18-07-11:08:00:48 INFO [clipper_admin.py:124] Clipper still initializing. 18-07-11:08:00:49 INFO [clipper_admin.py:124] Clipper still initializing. 18-07-11:08:00:50 INFO [clipper_admin.py:124] Clipper still initializing. 18-07-11:08:00:51 INFO [clipper_admin.py:124] Clipper still initializing. 18-07-11:08:00:52 INFO [clipper_admin.py:124] Clipper still initializing. 18-07-11:08:00:53 INFO [clipper_admin.py:124] Clipper still initializing. 18-07-11:08:00:54 INFO [clipper_admin.py:124] Clipper still initializing. 18-07-11:08:00:55 INFO [clipper_admin.py:124] Clipper still initializing. 18-07-11:08:00:56 INFO [clipper_admin.py:124] Clipper still initializing. ..........

current environment: [root@zhiyuan1 ~]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE load-generator-7bbbb4fdd4-zsswb 1/1 Running 5 6d 10.244.3.137 zhiyuan2.novalocal metrics-7d577dbc99-lmwwb 1/1 Running 0 6h 10.244.2.161 zhiyuan-3 mgmt-frontend-7858f5c57c-vmhvz 1/1 Running 0 6h 10.244.2.160 zhiyuan-3 php-apache-c55756996-rjkjs 1/1 Running 4 1d 10.244.2.146 zhiyuan-3 query-frontend-0-b4df59c5-tgmbr 2/2 Running 0 6h 10.244.4.60 zhiyuan-4 redis-56d457969d-gkl56 1/1 Running 0 6h 10.244.4.59 zhiyuan-4 webapp-7qkxx 1/1 Running 3 8d 10.244.3.136 zhiyuan2.novalocal webapp-nm9c2 1/1 Running 3 8d 10.244.0.41 zhiyuan1.novalocal [root@zhiyuan1 ~]#

[root@zhiyuan1 ~]# kubectl get service -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR kubernetes ClusterIP 10.96.0.1 443/TCP 14d metrics NodePort 10.96.26.77 9090:30965/TCP 6h ai.clipper.name=metrics mgmt-frontend NodePort 10.96.39.108 1338:31238/TCP 6h ai.clipper.name=mgmt-frontend php-apache ClusterIP 10.106.230.133 192.168.13.82 80/TCP 6d run=php-apache query-frontend NodePort 10.108.199.22 1337:31048/TCP 6h ai.clipper.name=query-frontend query-frontend-0 NodePort 10.100.205.246 7000:32480/TCP 6h ai.clipper.name=query-frontend,ai.clipper.query_frontend.id=0 redis NodePort 10.97.136.40 6379:31481/TCP 6h ai.clipper.name=redis webapp NodePort 10.102.224.207 8081:30003/TCP 9d app=webapp [root@zhiyuan1 ~]#

agneeshdg commented 6 years ago

Hi @xuguangzhao , try using clipper_admin.connect() instead of start_clipper(), since you already have a Clipper cluster as part of the k8s cluster.

YogeshSomawar commented 6 years ago

Hi, I am also facing same issue. I am using k8s from native google cloud.

I have used connect() for connecting to clipper, but on registering application it is getting timeout,

ksoucy commented 6 years ago

I also experience the same issue but using k8s locally with minikube.

gangtao commented 6 years ago

I have similar issue, the initializing cannot stop. the problem is that my k8s is running on AWS, the service is not exposed, so in this case, the clipper cannot access the service by nodename:port. ConnectionError: HTTPConnectionPool(host='ec2-54-68-64-106.us-west-2.compute.amazonaws.com', port=30204): Max retries exceeded with url: /admin/add_app (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1129fae90>: Failed to establish a new connection: [Errno 60] Operation timed out',))

so need to use kubectl proxy instead, I think.

dipayan90 commented 5 years ago

@gangtao I too have the same problem. Were you able to get it to work ? I am using UseInternalIP=true.

Relevant code seems to be: 

            for addr in node.status.addresses:
                if addr.type == "ExternalDNS":
                    external_node_hosts.append(addr.address)

        if len(external_node_hosts) == 0 and (self.useInternalIP):
            msg = "No external node addresses found. Using Internal IP address"
            self.logger.warn(msg)
            for addr in node.status.addresses:
                if addr.type == "InternalIP":
                    external_node_hosts.append(addr.address)

However the node ips are not accesible from outside, so we get connection time out. I created an ingress controller to talk to the service IP, but not sure how to add it to the clipper_admin

withsmilo commented 5 years ago

@gangtao @dipayan90 How about use kubectl proxy?

$ kubectl proxy --port=8080

And then, add kubernetes_proxy_addr parameter when creating a ClipperConnection object.


import urllib3
from clipper_admin import ClipperConnection, KubernetesContainerManager
from clipper_admin.deployers import python as python_deployer
urllib3.disable_warnings(category=urllib3.exceptions.InsecureRequestWarning)

clipper_conn = ClipperConnection(KubernetesContainerManager(useInternalIP=True, kubernetes_proxy_addr="127.0.0.1:8080", create_namespace_if_not_exists=True))
clipper_conn.start_clipper()
avinregmi commented 5 years ago

I created load balancer service on Google Kubernetes engine. It will forward outside traffic from port 80 to mgmt-frontend deployment at port 1338. I'm able to ping port 80 from outside using netcat. Unfortunately, I'm not able to get a respond from "http://{host}/admin/ping" as a result my code is stuck at "Clipper still initializing". How can I solve this?

wcwang07 commented 5 years ago

@withsmilo @xuguangzhao trying to start clipper on aws k8s still facing the Clipper still initilizing issue setting local proxy port 8080

followed steps to start k8s cluster: https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html#vpc-create

k8s_1

HarshRangwala commented 4 years ago

Guys, please help. I cant figure whats causing this error. I am doing a project on self driving car.

ConnectionError: HTTPConnectionPool(host='192.168.0.31', port=5000): Max retries exceeded with url: /messages?data=x (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x000001D456A07E08>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond'))

Following link contains the code that throws the above error https://codeshare.io/aVgIMP