milo157
commented
2 months ago Hi @Gitcqs
I just updated the example to prevent this error from happening. Its the vLLM server that is timing out. I just updated the check_vllm_server function to this:
def check_vllm_model_status(): url = "http://127.0.0.1:5000/v1/models" headers = { "Authorization": f"Bearer {get_secret('HF_TOKEN')}" } max_retries = 8 for _ in range(max_retries): print('Trying vllm server') try: response = requests.get(url, headers=headers) if response.status_code == 200: return True except requests.ConnectionError: print("Connection refused, retrying...") time.sleep(15) return False
Send Result: {'run_id': '71d6c236-fe7a-961f-9548-120375ac8377', 'result': {'error': "HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: /v1/models (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc7730c5490>: Failed to establish a new connection: [Errno 111] Connection refused'))"}, 'run_time_ms': 7026.292324066162}