Open Uranium2 opened 2 months ago
Hi @Uranium2! Thank you for reporting this, and also for adding all the logs. The exception is thrown by http
library which cannot parse server response. I've found this issue which looks related. So what I would suggest to start from is to somehow dump the raw server response when such error happens next time. Then, analyze the dump - including empty lines and presense of correct newline sequences. If everything looks correct there - please let me know, then we'll think what to do next. But if you'll notice anything wrong with the reponse itself - reach a Databricks support and show them this dump
Hello,
My team and I have been using the
databricks-sql-python
library for a year now. It works great for the majority of our use cases, most of which involve linear algorithms that fetch data one query at a time. Occasionally, we run multiple jobs with parallel queries, and the Databricks Warehouse manages the workload using the queue without any issues.Recently, we encountered a new use case where a developer needed to execute a large number of queries in parallel. I advised him that the Databricks connector is not thread-safe, so he implemented a process pool, creating a new connector for each process. For context, we are using an Azure machine with 96 CPUs and 672 GB RAM. But we limit at 20 process, so 20 queries at the same time possible. I understand that the query queue limit in a warehouse is 1,000 queries. Our Warehouse is set to scale up to 5 nodes and is in Pro + Large. So it should not be an issue.
The issue we're facing is that one query frequently fails near the end of the process, despite all queries being similar (mainly differing by a filter). We're unsure why this happens. We tried increasing the number of nodes and the warehouse size, but without success.
Do you have any insights into what might be causing this issue? We're using the latest version of the
databricks-sql-connector==3.4.0