ucbrise / clipper

A low-latency prediction-serving system
http://clipper.ai
Apache License 2.0
1.4k stars 280 forks source link

ConnectionError: HTTPConnectionPool(host='192.168.13.79', port=30112): Max retries exceeded with url: /admin/get_all_models (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0708172e10>: Failed to establish a new connection: [Errno 110] Connection timed out',)) #536

Open xuguangzhao opened 6 years ago

xuguangzhao commented 6 years ago

When I finish deployed in k8s cluster, and the clipper is running. but when i clipper_admin.get_all_models():

ConnectionError Traceback (most recent call last) in () ----> 1 clipper_conn.get_all_models()

/root/anaconda2/lib/python2.7/site-packages/clipper_admin/clipper_admin.pyc in get_all_models(self, verbose) 889 req_json = json.dumps({"verbose": verbose}) 890 headers = {'Content-type': 'application/json'} --> 891 r = requests.post(url, headers=headers, data=req_json) 892 logger.debug(r.text) 893

/root/anaconda2/lib/python2.7/site-packages/requests/api.pyc in post(url, data, json, kwargs) 110 """ 111 --> 112 return request('post', url, data=data, json=json, kwargs) 113 114

/root/anaconda2/lib/python2.7/site-packages/requests/api.pyc in request(method, url, kwargs) 56 # cases, and look like a memory leak in others. 57 with sessions.Session() as session: ---> 58 return session.request(method=method, url=url, kwargs) 59 60

/root/anaconda2/lib/python2.7/site-packages/requests/sessions.pyc in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json) 506 } 507 send_kwargs.update(settings) --> 508 resp = self.send(prep, **send_kwargs) 509 510 return resp

/root/anaconda2/lib/python2.7/site-packages/requests/sessions.pyc in send(self, request, kwargs) 616 617 # Send the request --> 618 r = adapter.send(request, kwargs) 619 620 # Total elapsed time of the request (approximately)

/root/anaconda2/lib/python2.7/site-packages/requests/adapters.pyc in send(self, request, stream, timeout, verify, cert, proxies) 506 raise SSLError(e, request=request) 507 --> 508 raise ConnectionError(e, request=request) 509 510 except ClosedPoolError as e:

ConnectionError: HTTPConnectionPool(host='192.168.13.79', port=30112): Max retries exceeded with url: /admin/get_all_models (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0708172e10>: Failed to establish a new connection: [Errno 110] Connection timed out',))

clipper use all the ports is lisening: 44442486850-aff1a010-8430-11e8-9a2b-679088fd800e

simon-mo commented 6 years ago

Are you using kubernetes proxy? If so, you can pass in kubernetes_proxy_addr as an argument in constructing KubernetesContainerManager as described here: http://docs.clipper.ai/en/v0.3.0/container_managers.html#clipper_admin.KubernetesContainerManager.__init__

xuguangzhao commented 6 years ago

@simon-mo I use proxy kubernetes_proxy_addr but also is Clipper still initializing:

18-07-12:01:18:12 WARNING [kubernetes_container_manager.py:241] No external node addresses found. Using Internal IP address 18-07-12:01:18:12 INFO [kubernetes_container_manager.py:254] Found 1 nodes: 192.168.13.79 18-07-12:01:18:12 INFO [kubernetes_container_manager.py:263] Setting Clipper mgmt port to 31238 18-07-12:01:18:12 INFO [kubernetes_container_manager.py:271] Setting Clipper query port to 31048 18-07-12:01:18:12 INFO [kubernetes_container_manager.py:286] Setting Clipper metric port to 30965 print url address http://192.168.13.82/api/v1/namespaces/default/services/query-frontend:1337/proxy/metrics print url address finish 18-07-12:01:18:17 INFO [clipper_admin.py:127] Clipper still initializing. print url address http://192.168.13.82/api/v1/namespaces/default/services/query-frontend:1337/proxy/metrics print url address finish 18-07-12:01:18:23 INFO [clipper_admin.py:127] Clipper still initializing. print url address http://192.168.13.82/api/v1/namespaces/default/services/query-frontend:1337/proxy/metrics print url address finish 18-07-12:01:18:29 INFO [clipper_admin.py:127] Clipper still initializing. print url address http://192.168.13.82/api/v1/namespaces/default/services/query-frontend:1337/proxy/metrics print url address finish 18-07-12:01:18:35 INFO [clipper_admin.py:127] Clipper still initializing. print url address http://192.168.13.82/api/v1/namespaces/default/services/query-frontend:1337/proxy/metrics print url address finish 18-07-12:01:18:41 INFO [clipper_admin.py:127] Clipper still initializing. print url address http://192.168.13.82/api/v1/namespaces/default/services/query-frontend:1337/proxy/metrics print url address finish 18-07-12:01:18:47 INFO [clipper_admin.py:127] Clipper still initializing. print url address http://192.168.13.82/api/v1/namespaces/default/services/query-frontend:1337/proxy/metrics print url address finish

YogeshSomawar commented 6 years ago

Please have a look at #537 , might help you.