LookML Validation facing error

[danitp97]

Hi, I'm facing this error. I didn't have any issue in previous PRs. I'm not sure what does the error mean. Could somebody help? Thx!

[DylanBaker]

Hi @danitp97. This looks like it is as a result of an HTTP request error. Typically these are transient. Are you getting this repeatedly on this PR?

[matt-lam-thinkific]

I'll add that I'm getting this as well, running Spectacles CLI from GitHub actions. This also looks similar to a previous issue, #725 ...

Seems like the same setup as above: Running something like

spectacles lookml --base-url [redacted] --project [redacted] --client-id $CLIENT_ID --client-secret $CLIENT_SECRET --branch $GITHUB_HEAD_REF --remote-reset

This generally resolves on retry for me, though this seems to be happening with increasing frequency.

[danitp97]

Hey! Yes, exactly I tried several times and finally worked out. Thanks!!

[camtr0n]

Hi this is happening intermittently on our Spectacles CLI github actions. Just started popping off about 3 weeks ago and was totally fine before that. This does not resolve on retry. spectacles lookml --base-url ${LOOKER_BASE_URL} --client-id ${LOOKER_CLIENT_ID} --client-secret ${LOOKER_CLIENT_SECRET} --project replicated --severity error

[matt-lam-thinkific]

One thing I noticed is that every time (at least, every time I've looked at) we get this particular error, the GitHub action run duration is just over 5 minutes. Seems to me like there's some timeout somewhere -- and this also doesn't explain why it runs quickly (~10 seconds) without issue other times.

[Hoanglinh1201]

Hey, I also got this error and now it is becoming more and more unstable. Maybe 1/10 could be finished and when success, it is very quick, however the rest runs until timeout.

Anyone has any hacks or workarounds that I can use?

[camtr0n]

@Hoanglinh1201 one workaround that seems to work consistently for us is, if I go to the Spectacles user in Looker that has api credentials and if I sudo as that user, and go into the dev branch that is struggling, and pull from remote -- it starts working again for that CI run. I've also done the same by pulling from Production to get our nightly job to pass on rerun.

[Hoanglinh1201]

thank @camtr0n for the suggestion. I tried but it doesn't seem to solve our problem. I feel we cannot perform lookml validation at all now with 100/100 jobs timed out.

[matt-lam-thinkific]

Out of curiosity, how complex are everyone's LookML projects who're experiencing this? We've got a pretty large and messy project (lots of include: *.view situations 🙈).

Based on the behaviour @camtr0n is describing, and our own experience, my hypothesis is that LookML validation is just taking longer than 5 minutes, and there's a rogue timeout somewhere in the client code for that long (even though the timeout is set to 7200 seconds...). LookML validation eventually finishes, though, so at some point when we retry the Spectacles run, we get a cache hit and so a retry eventually succeeds and returns results in seconds.

I think this problem is worse when there are lots of CI runs being initiated, all on different PRs on different change sets. In that case, you never get a cache hit because you're always validating a different LookML configuration. We certainly find that retries are likelier to succeed when there are fewer people working on the repo simultaneously.

Any ideas on how we can validate this?

[fredriv]

We're seeing the same issue on validations taking > 5 minutes. Here's a full stack trace from Spectacles (v2.4.8):

Completed validation in 5 minutes and 23 seconds.

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpx/_transports/default.py", line 67, in map_httpcore_exceptions
    yield
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpx/_transports/default.py", line 371, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 216, in handle_async_request
    raise exc from None
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 196, in handle_async_request
    response = await connection.handle_async_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpcore/_async/connection.py", line 101, in handle_async_request
    return await self._connection.handle_async_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpcore/_async/http11.py", line 143, in handle_async_request
    raise exc
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpcore/_async/http11.py", line 113, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpcore/_async/http11.py", line 186, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpcore/_async/http11.py", line 224, in _receive_event
    data = await self._network_stream.read(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 32, in read
    with map_exceptions(exc_map):
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ReadError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/spectacles/cli.py", line 192, in wrapper
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/spectacles/cli.py", line 375, in main
    asyncio.run(
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/spectacles/utils.py", line 65, in timed_function
    result = await fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/spectacles/cli.py", line 826, in run_lookml
    results = await runner.validate_lookml(ref, severity, timeout)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/spectacles/runner.py", line 523, in validate_lookml
    results = await validator.validate(self.project, severity, timeout)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/spectacles/validators/lookml.py", line 43, in validate
    validation_results = await self.client.lookml_validation(project, timeout)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/backoff/_async.py", line 151, in retry
    ret = await target(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/spectacles/client.py", line 873, in lookml_validation
    response = await self.post(url=url, timeout=timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/spectacles/client.py", line 1[62](https://github.com/kolonialno/lookml-models/actions/runs/8780414757/job/24094684720?pr=2752#step:5:63), in post
    return await self.request("POST", url, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x[64](https://github.com/kolonialno/lookml-models/actions/runs/8780414757/job/24094684720?pr=2752#step:5:65)/lib/python3.11/site-packages/spectacles/client.py", line 156, in request
    return await self.async_client.request(method, url, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpx/_client.py", line 1559, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpx/_client.py", line 1646, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpx/_client.py", line 1[67](https://github.com/kolonialno/lookml-models/actions/runs/8780414757/job/24094684720?pr=2752#step:5:68)4, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpx/_client.py", line 1711, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpx/_client.py", line 1748, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpx/_transports/default.py", line 3[70](https://github.com/kolonialno/lookml-models/actions/runs/8780414757/job/24094684720?pr=2752#step:5:71), in handle_async_request
    with map_httpcore_exceptions():
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/httpx/_transports/default.py", line 84, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadError

Encountered unexpected ReadError: ""
Full error traceback logged to file.

For support, please create an issue at https://github.com/spectacles-ci/spectacles/issues

Error: Child_process exited with error code 1

[DylanBaker]

Hey everyone! Just an update that we should be pushing some update retry and httpx code shortly that should hopefully tackle some of these read errors.

We don't yet have a solution to the long running LookML validation via the API, but continue to work with the team at Google/Looker on this.