search

spectacles-ci / spectacles

A continuous integration tool for Looker and LookML.
https://spectacles.dev
MIT License
216 stars 34 forks source link

Make data test timeout configurable #630

Open DylanBaker opened 2 years ago

DylanBaker commented 2 years ago

Currently there is a 5 minute timeout for all client API methods, including running data tests. We have found that a few customers have data tests that take more than 5 minutes. Currently, without forking the code, there is no way around this.

We should aim to make this configurable so that the 5 minute default still applies but that customers can overide it. This should be configurable from the CLI for open-source users and the validate_data_tests method for hosted application users.

Kobold commented 1 year ago

Running into this now. Just for google's sake, here's how this manifested for me.

Our CI setup had spectacles assert time out shortly after 5 minutes with this error:

Encountered unexpected ReadError: ""
Full error traceback logged to file.

The actual traceback emitted:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/asyncio/selector_events.py", line 854, in _read_ready__data_received
    data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 104] Connection reset by peer

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions
    yield
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 33, in read
    return await self._stream.receive(max_bytes=max_bytes)
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/anyio/streams/tls.py", line 195, in receive
    data = await self._call_sslobject_method(self._ssl_object.read, max_bytes)
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/anyio/streams/tls.py", line 137, in _call_sslobject_method
    data = await self.transport_stream.receive()
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1274, in receive
    raise self._protocol.exception
anyio.BrokenResourceError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/httpcore/_async/http11.py", line 105, in handle_async_request
    raise exc
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/httpcore/_async/http11.py", line 84, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/httpcore/_async/http11.py", line 148, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/httpcore/_async/http11.py", line 177, in _receive_event
    data = await self._network_stream.read(
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 35, in read
    return b""
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
    raise to_exc(exc)
httpcore.ReadError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/spectacles/cli.py", line 155, in wrapper
    return function(*args, **kwargs)
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/spectacles/cli.py", line 300, in main
    asyncio.run(
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/spectacles/utils.py", line 60, in timed_function
    result = await fn(*args, **kwargs)
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/spectacles/cli.py", line 876, in run_assert
    results = await runner.validate_data_tests(ref, filters)
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/spectacles/runner.py", line 465, in validate_data_tests
    await validator.validate(tests)
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/spectacles/validators/data_test.py", line 135, in validate
    await asyncio.gather(*(run_test(test, query_slot) for test in tests))
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/spectacles/validators/data_test.py", line [102](https://github.com/TenantBase/analytics/actions/runs/5537197816/jobs/10105763761#step:9:103), in run_test
    results = await self.client.run_lookml_test(
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/backoff/_async.py", line 133, in retry
    ret = await target(*args, **kwargs)
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/spectacles/client.py", line 582, in run_lookml_test
    response = await self.get(
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/spectacles/client.py", line 151, in get
    return await self.request("GET", url, *args, **kwargs)
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/spectacles/client.py", line 148, in request
    return await self.async_client.request(method, url, *args, **kwargs)
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/httpx/_client.py", line 1533, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/httpx/_client.py", line 1620, in send
    response = await self._send_handling_auth(
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/httpx/_client.py", line 1648, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/httpx/_client.py", line 1685, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/httpx/_client.py", line 1722, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/contextlib.py", line [137](https://github.com/TenantBase/analytics/actions/runs/5537197816/jobs/10105763761#step:9:138), in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/runner/.cache/pypoetry/virtualenvs/tb-analytics-8DL94pCV-py3.9/lib/python3.9/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadError

Thanks for the tool. Spectacles is great!