dpguthrie / dbtc

37 stars 12 forks source link

More verbose logging + add retry if the request returns non-200 #107

Open jeremyyeo opened 5 days ago

jeremyyeo commented 5 days ago

Some customers are running into errors when triggering jobs:

  File "***********lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "*****.py", line 63, in <module>
    run(*sys.argv[1:])
  File "********.py", line 17, in run
    job_trigger = dbt_cloud_client.cloud.trigger_job(account_id = dbt_cloud_account_id,

I don't see a way to get any verbose logging so it's hard to say what exactly is in the returned response that isn't confirming here.

jeremyyeo commented 5 days ago

I think the problem is - when polling for jobs, sometimes the dbt Cloud API may return non-JSON or empty JSON - this causes dbtc to crash because it doesn't handle this case.

https://stackoverflow.com/questions/16573332/jsondecodeerror-expecting-value-line-1-column-1-char-0

jeremyyeo commented 1 day ago

I believe the issue is that the request methods aren't resilient to network errors. Have traced this to the fact that the dbt Cloud API's load balancer can return a 502 error so there is no actual json during response.json(): https://github.com/dpguthrie/dbtc/blob/main/dbtc/client/admin.py#L100

We'll probably want to have some sort of retry for such errors I think.