search

crflynn / databricks-dbapi

DBAPI and SQLAlchemy dialect for Databricks Workspace and SQL Analytics clusters
MIT License
22 stars 8 forks source link

Connecting Databricks Delta Lake using SQL Alchemy (Pyhive) #5

Closed swathiAMD closed 4 years ago

swathiAMD commented 4 years ago
  1. I have used the Flask and SQL Alchemy ORM concept for the REST API.
  2. The API simply connects to one of the Databricks Delta lake database , queries one particular table and returns the result.
  3. I have used the following sample code to create the SQL Alchemy engine : URL: _https://pypi.org/project/databricks-dbapi/ Azure Databricks with token provide token, region for url, database name, http_path (with cluster name) engine = create_engine( "databricks+pyhive://token:@.azuredatabricks.net:443/", connect_args={"http_path": ""} )_
  4. I have written the REST API , created a docker image and deployed the image on the AKS cluster.
  5. For starting few days the API was fetching the response. From yesterday I see the attached whole error received DatabricksI_Error.txt) Single line from the error : Error Line: sqlState=None, errorCode=0, errorMessage='Invalid SessionHandle: SessionHandle [4d00c5b7-2ee4-448a-92de-cb04deb773b6]'), operationHandle=None)
  6. I had restarted the docker container yesterday which has this POD using the agent-pool-node serial console after which the API was working. But from today morning again I get this issue. I m not sure why the issue is occurring could you please help me ?

Databricks_Error.txt

crflynn commented 4 years ago

After some research this can happen if the connection is destroyed, and then attempted to be used again. After disconnecting the session is no longer valid and thus reconnection is required.

Has the cluster been restarted during this time? This will destroy existing connections and hence previous sessions would no longer be valid.

You might be able to mitigate this error by creating a new connection when it is thrown in your REST API. Be sure to implement a backoff retry in that case.