search

databricks / databricks-sql-python

Databricks SQL Connector for Python
Apache License 2.0
153 stars 87 forks source link

Retry connection on 'Remote end closed connection without response' #433

Open septponf opened 2 weeks ago

septponf commented 2 weeks ago

I initially reported this issue against dbt-databricks but was asked by a collaborator to file it here.

So basically, when running dbt request against a serverless sql warehouse, we get intermittent errors as below and the dag execution is aborted. Runtime Error ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

I believe it should be safe to retry connection and retry execution.

databricks-sql-connector==2.9.5

kravets-levko commented 2 weeks ago

Hi @septponf! Thank you for reporting this issue. Yes, retry logic currently is not perfect, and each report like this helps to make it better. I need to explore your issue more, but generally retrying query execution is not safe, even on errors like terminated connection. See, if you submitted query, but didn't get the response from the server (for any reason) - you don't know on which stage query execution failed. Maybe server didn't even start processing the request, as well as it may be possible that query was successfully executed but server failed when sending the response to the library. And it's relatively safe to re-execute read queries (e.g. SELECT or SHOW TABLES, etc.), but it's definitely not safe to re-execute queries that update data.

We do have a list of errors that a safe to retry, and this library still doesn't fully implement it. Right now I'm doing another iteration to improve the retry logic of this library, and I will check what I can do in your case. But keep in mind what I explained above. In some cases, only user can decide what is safe to retry and what's not

septponf commented 2 weeks ago

Thank you @kravets-levko for swift response. I understand the general predicament. But general retries for i.e. select x, show x, alter x, create or replace x, would be nice.

I appreciate you looking into this