iris-hep / idap-200gbps-atlas

benchmarking throughput with PHYSLITE
6 stars 1 forks source link

Fix up timeout issue in servicex_frontend's `get_transform_status` #97

Closed gordonwatts closed 3 months ago

gordonwatts commented 3 months ago

get_transform_status is timeing out when servicex is under very heavy load. The call sometimes take 5 seconds and a bit, which causes a timeout. Fix this by increasing the timeout to 10 seconds.

This is tracking bug and will be linked to an actual bug in the frontend github.

Traceback (most recent call last):
  File "/usr/AnalysisBaseExternals/25.2.2/InstallArea/x86_64-el9-gcc13-opt/lib/python3.9/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/venv/lib/python3.9/site-packages/servicex/query.py", line 194, in transform_complete
    raise task.exception()
  File "/venv/lib/python3.9/site-packages/servicex/query.py", line 333, in transform_status_listener
    await self.retrieve_current_transform_status()
  File "/venv/lib/python3.9/site-packages/servicex/query.py", line 365, in retrieve_current_transform_status
    s = await self.servicex.get_transform_status(self.request_id)
  File "/venv/lib/python3.9/site-packages/servicex/servicex_adapter.py", line 129, in get_transform_status
    r = await client.get(
  File "/venv/lib/python3.9/site-packages/httpx/_client.py", line 1801, in get
    return await self.request(
  File "/venv/lib/python3.9/site-packages/httpx/_client.py", line 1574, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/venv/lib/python3.9/site-packages/httpx/_client.py", line 1661, in send
    response = await self._send_handling_auth(
  File "/venv/lib/python3.9/site-packages/httpx/_client.py", line 1689, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/venv/lib/python3.9/site-packages/httpx/_client.py", line 1726, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/venv/lib/python3.9/site-packages/httpx/_client.py", line 1763, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/venv/lib/python3.9/site-packages/httpx/_transports/default.py", line 373, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/AnalysisBaseExternals/25.2.2/InstallArea/x86_64-el9-gcc13-opt/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/venv/lib/python3.9/site-packages/httpx/_transports/default.py", line 86, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectTimeout
gordonwatts commented 3 months ago

Tracked in PR here: https://github.com/ssl-hep/ServiceX_frontend/pull/374

gordonwatts commented 3 months ago

Fixed!