search

RTXteam / RTX

Software repo for Team Expander Agent (Oregon State U., Institute for Systems Biology, and Penn State U.)
https://arax.ncats.io/
MIT License
33 stars 21 forks source link

Crash in the logs #1993

Open edeutsch opened 1 year ago

edeutsch commented 1 year ago

Hi everyone, here's a crash from the logs when Plover is an in an error state:

2023-03-29T12:32:18.002941 DEBUG: [] Sending query to https://kg2cploverdb.ci.transltr.io
2023-03-29T12:32:18.006977 ERROR: [UncaughtError] An uncaught error was thrown while trying to Expand using infores:rtx-kg2 (local). Error was: Traceback (most recent call last):
  File "/mnt/data/python/Python-3.9.13/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/mnt/data/python/Python-3.9.13/lib/python3.9/site-packages/urllib3/util/connection.py", line 72, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/mnt/data/python/Python-3.9.13/lib/python3.9/socket.py", line 954, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/data/python/Python-3.9.13/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/mnt/data/python/Python-3.9.13/lib/python3.9/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/mnt/data/python/Python-3.9.13/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/mnt/data/python/Python-3.9.13/lib/python3.9/site-packages/urllib3/connection.py", line 358, in connect
    self.sock = conn = self._new_conn()
  File "/mnt/data/python/Python-3.9.13/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f8c9ba17790>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/data/python/Python-3.9.13/lib/python3.9/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/mnt/data/python/Python-3.9.13/lib/python3.9/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/mnt/data/python/Python-3.9.13/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='kg2cploverdb.ci.transltr.io', port=443): Max retries exceeded with url: /query (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f8c9ba17790>: Failed to establish a new connection: [Errno -2] Name or service not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/data/orangeboard/kg2beta/RTX/code/UI/OpenAPI/python-flask-server/KG2/openapi_server/controllers/../../../../../../ARAX/ARAXQuery/ARAX_expander.py", line 789, in _expand_edge_kg2_local
    answer_kg = kg2_querier.answer_one_hop_query(one_hop_qg)
  File "/mnt/data/orangeboard/kg2beta/RTX/code/UI/OpenAPI/python-flask-server/KG2/openapi_server/controllers/../../../../../../ARAX/ARAXQuery/Expand/kg2_querier.py", line 79, in answer_one_hop_query
    plover_answer, response_status = self._answer_query_using_plover(query_graph, log)
  File "/mnt/data/orangeboard/kg2beta/RTX/code/UI/OpenAPI/python-flask-server/KG2/openapi_server/controllers/../../../../../../ARAX/ARAXQuery/Expand/kg2_querier.py", line 184, in _answer_query_using_plover
    response = requests.post(f"{rtxc.plover_url}/query", json=dict_qg, timeout=60,
  File "/mnt/data/python/Python-3.9.13/lib/python3.9/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
  File "/mnt/data/python/Python-3.9.13/lib/python3.9/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/mnt/data/python/Python-3.9.13/lib/python3.9/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/mnt/data/python/Python-3.9.13/lib/python3.9/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/mnt/data/python/Python-3.9.13/lib/python3.9/site-packages/requests/adapters.py", line 565, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='kg2cploverdb.ci.transltr.io', port=443): Max retries exceeded with url: /query (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f8c9ba17790>: Failed to establish a new connection: [Errno -2] Name or service not known'))

2023-03-29T12:32:18.007037 DEBUG: [] Got answers from all KPs; merging them into one KG
2023-03-29T12:32:18.007065 DEBUG: [] Merging answer into Message.KnowledgeGraph
2023-03-29T12:32:18.264410 DEBUG: [] Sending query to https://kg2cploverdb.ci.transltr.io
2023-03-29T12:32:18.268524 ERROR: [UncaughtError] An uncaught error was thrown while trying to Expand using infores:rtx-kg2 (local). Error was: Traceback (most recent call last):
  File "/mnt/data/python/Python-3.9.13/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/mnt/data/python/Python-3.9.13/lib/python3.9/site-packages/urllib3/util/connection.py", line 72, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/mnt/data/python/Python-3.9.13/lib/python3.9/socket.py", line 954, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/data/python/Python-3.9.13/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/mnt/data/python/Python-3.9.13/lib/python3.9/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/mnt/data/python/Python-3.9.13/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/mnt/data/python/Python-3.9.13/lib/python3.9/site-packages/urllib3/connection.py", line 358, in connect
    self.sock = conn = self._new_conn()
  File "/mnt/data/python/Python-3.9.13/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f8d2216d790>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/data/python/Python-3.9.13/lib/python3.9/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/mnt/data/python/Python-3.9.13/lib/python3.9/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/mnt/data/python/Python-3.9.13/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='kg2cploverdb.ci.transltr.io', port=443): Max retries exceeded with url: /query (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f8d2216d790>: Failed to establish a new connection: [Errno -2] Name or service not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/data/orangeboard/kg2beta/RTX/code/UI/OpenAPI/python-flask-server/KG2/openapi_server/controllers/../../../../../../ARAX/ARAXQuery/ARAX_expander.py", line 789, in _expand_edge_kg2_local
    answer_kg = kg2_querier.answer_one_hop_query(one_hop_qg)
  File "/mnt/data/orangeboard/kg2beta/RTX/code/UI/OpenAPI/python-flask-server/KG2/openapi_server/controllers/../../../../../../ARAX/ARAXQuery/Expand/kg2_querier.py", line 79, in answer_one_hop_query
    plover_answer, response_status = self._answer_query_using_plover(query_graph, log)
  File "/mnt/data/orangeboard/kg2beta/RTX/code/UI/OpenAPI/python-flask-server/KG2/openapi_server/controllers/../../../../../../ARAX/ARAXQuery/Expand/kg2_querier.py", line 184, in _answer_query_using_plover
    response = requests.post(f"{rtxc.plover_url}/query", json=dict_qg, timeout=60,
  File "/mnt/data/python/Python-3.9.13/lib/python3.9/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
  File "/mnt/data/python/Python-3.9.13/lib/python3.9/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/mnt/data/python/Python-3.9.13/lib/python3.9/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/mnt/data/python/Python-3.9.13/lib/python3.9/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/mnt/data/python/Python-3.9.13/lib/python3.9/site-packages/requests/adapters.py", line 565, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='kg2cploverdb.ci.transltr.io', port=443): Max retries exceeded with url: /query (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f8d2216d790>: Failed to establish a new connection: [Errno -2] Name or service not known'))

2023-03-29T12:32:18.268581 DEBUG: [] Got answers from all KPs; merging them into one KG
2023-03-29T12:32:18.268610 DEBUG: [] Merging answer into Message.KnowledgeGraph
2023-03-29T12:32:18.687867 DEBUG: [] Sending NodeSynonymizer.get_canonical_curies() a list of 2 curies
2023-03-29T12:32:18.688327 DEBUG: [] Got response back from NodeSynonymizer
2023-03-29T12:32:18.952376 DEBUG: [] Sending NodeSynonymizer.get_canonical_curies() a list of 2 curies
2023-03-29T12:32:18.952827 DEBUG: [] Got response back from NodeSynonymizer
2023-03-29T12:32:19.209123 DEBUG: [] Sending NodeSynonymizer.get_canonical_curies() a list of 736 curies
2023-03-29T12:32:19.221288 DEBUG: [] Got response back from NodeSynonymizer
2023-03-29T12:32:19.221688 WARNING: [] NodeSynonymizer did not recognize: {'UMLS:C1320716', 'UMLS:C0573209', 'UMLS:C1997351', 'UMLS:C0403527', 'UMLS:C0745138', 'UMLS:C3698411', 'UMLS:C0573229', 'UMLS:C2711653'}
2023-03-29T12:32:19.481816 DEBUG: [] Sending NodeSynonymizer.get_canonical_curies() a list of 723 curies
2023-03-29T12:32:19.493331 DEBUG: [] Got response back from NodeSynonymizer
2023-03-29T12:32:19.493736 WARNING: [] NodeSynonymizer did not recognize: {'UMLS:C1320716', 'UMLS:C0573209', 'UMLS:C1997351', 'UMLS:C0403527', 'UMLS:C0745138', 'UMLS:C3698411', 'UMLS:C0573229', 'UMLS:C2711653'}
INFO:werkzeug:127.0.0.1 - - [29/Mar/2023 12:32:19] "POST /beta/api/rtxkg2/v1.3/query HTTP/1.1" 400 -
INFO:werkzeug:127.0.0.1 - - [29/Mar/2023 12:32:20] "POST /beta/api/rtxkg2/v1.3/query HTTP/1.1" 400 -

Might be good to catch this more gracefully.

amykglen commented 1 year ago

yes, we should definitely catch this more gracefully...