Uranium2
commented
10 months ago I think I'm having a similar issue, but only on a certain type of environment. In WSL 1 and Compute Instances on Azure Machine Learning, it's working. But on Compute Clusters in Azure Machine Learning I have an issue.
@vcakdwivedi is it possible to share some logs with more logging enabled? Here is how I enable more logs.
import logging
import contextlib
from http.client import HTTPConnection
def debug_requests_on():
'''Switches on logging of the requests module.'''
HTTPConnection.debuglevel = 1
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True
def debug_requests_off():
'''Switches off logging of the requests module, might be some side-effects'''
HTTPConnection.debuglevel = 0
root_logger = logging.getLogger()
root_logger.setLevel(logging.WARNING)
root_logger.handlers = []
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.WARNING)
requests_log.propagate = False
@contextlib.contextmanager
def debug_requests():
'''Use with 'with'!'''
debug_requests_on()
yield
debug_requests_off()
debug_requests_on()
# Insert code to connect to databricks SQL WarehouseBe extra carefull, it shows every single tokens from Azure or PAT from Databricks, you might want to remove them if you post the log.
Here is what I without the logger:
2024-01-23 08:46:12,704 - thrift_backend - Error during request to server: {"method": "OpenSession", "session-id": null, "query-id": null, "http-code": null, "error-message": "", "original-exception": "Retry request would exceed Retry policy max retry duration of 900.0 seconds", "no-retry-reason": "non-retryable error", "bounded-retry-delay": null, "attempt": "1/30", "elapsed-seconds": "846.7760233879089/900.0"} Traceback (most recent call last): File "/opt/miniconda/lib/python3.10/site-packages/databricks/sql/init.py", line 51, in connect return Connection(server_hostname, http_path, access_token, **kwargs) File "/opt/miniconda/lib/python3.10/site-packages/databricks/sql/client.py", line 235, in init self._open_session_resp = self.thrift_backend.open_session( File "/opt/miniconda/lib/python3.10/site-packages/databricks/sql/thrift_backend.py", line 576, in open_session response = self.make_request(self._client.OpenSession, open_session_req) File "/opt/miniconda/lib/python3.10/site-packages/databricks/sql/thrift_backend.py", line 505, in make_request self._handle_request_error(error_info, attempt, elapsed) File "/opt/miniconda/lib/python3.10/site-packages/databricks/sql/thrift_backend.py", line 335, in _handle_request_error raise network_request_error databricks.sql.exc.RequestError: Error during request to server
And with the extra logging:
- Getting the Oauth2 Token from Azure:
INFO:azure.identity._credentials.environment:No environment configuration found. 2024-01-26 09:14:22,751 - environment - No environment configuration found. INFO:azure.identity._credentials.managed_identity:ManagedIdentityCredential will use Azure ML managed identity 2024-01-26 09:14:22,751 - managed_identity - ManagedIdentityCredential will use Azure ML managed identity INFO:azure.identity._credentials.chained:DefaultAzureCredential acquired a token from ManagedIdentityCredential 2024-01-26 09:14:22,894 - chained - DefaultAzureCredential acquired a token from ManagedIdentityCredential send: b'GET /MSI/token?api-version=2017-09-01&resource=xxxxxxxxxx&clientid=xxxxxxxxxxxx HTTP/1.1\r\nHost: localhost:46809\r\nUser-Agent: azsdk-python-identity/1.15.0 Python/3.10.13 (Linux-5.15.0-1053-azure-x86_64-with-glibc2.35)\r\nAccept-Encoding: gzip, deflate, br, zstd\r\nAccept: /\r\nConnection: keep-alive\r\nsecret: xxxxxxxxxxxxxxxxxxxxx\r\n\r\n' reply: 'HTTP/1.1 200 OK\r\n' header: Content-Type: application/json header: Date: Fri, 26 Jan 2024 09:14:22 GMT header: Content-Length: 1679 send: b'POST /api/2.0/token/create HTTP/1.1\r\nHost: adb-my-host.azuredatabricks.net\r\nUser-Agent: python-requests/2.31.0\r\nAccept-Encoding: gzip, deflate, br, zstd\r\nAccept: /\r\nConnection: keep-alive\r\nContent-type: application/json\r\nAuthorization: Bearer eyJ0eXAiOiJKV1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\r\nContent-Length: 78\r\n\r\n' send: b'{"comment": "Token created from python code", "lifetime_seconds": 86400}' reply: 'HTTP/1.1 200 OK\r\n' header: server: databricks header: x-databricks-reason-phrase: OK header: x-xss-protection: 1; mode=block header: date: Fri, 26 Jan 2024 09:14:23 GMT header: x-frame-options: SAMEORIGIN header: strict-transport-security: max-age=31536000; includeSubDomains; preload header: x-content-type-options: nosniff header: x-databricks-server-name: cons-webapp-44 header: strict-transport-security: max-age=31536000; includeSubDomains; preload header: x-content-type-options: nosniff header: content-type: application/json header: x-databricks-org-id: 5428309926879282 header: content-encoding: gzip header: vary: Accept-Encoding header: strict-transport-security: max-age=31536000; includeSubDomains; preload header: x-content-type-options: nosniff header: transfer-encoding: chunked
So here it confirms that I have a created a token, and this Service Principal user is in group with all rights ticked on. (sql + cluster usage/modification/deletion)
- Connecting to the Warehouse
send: b'POST /sql/1.0/warehouses/dcf9951cfebb7d84 HTTP/1.1\r\nHost: adb-my-host.azuredatabricks.net\r\nAccept-Encoding: identity\r\nContent-Type: application/x-thrift\r\nContent-Length: 118\r\nUser-Agent: PyDatabricksSqlConnector/3.0.1\r\nAuthorization: Bearer
\r\n\r\n' send: b'\x80\x01\x00\x01\x00\x00\x00\x0bOpenSession\x00\x00\x00\x00\x0c\x00\x01\r\x00\x04\x0b\x0b\x00\x00\x00\x01\x00\x00\x005spark.thriftserver.arrowBasedRowSet.timestampAsString\x00\x00\x00\x05false\n\x05\x02\x00\x00\x00\x00\x00\x00\xa5\x07\x02\x05\x05\x01\x00\x00' reply: 'HTTP/1.1 403 Forbidden\r\n' header: date: Fri, 26 Jan 2024 09:14:23 GMT header: x-frame-options: SAMEORIGIN header: x-xss-protection: 1; mode=block header: x-content-type-options: nosniff header: strict-transport-security: max-age=31536000; includeSubDomains; preload header: content-type: application/x-thrift header: x-thriftserver-error-message: PERMISSION_DENIED: is not authorized to use this SQL Endpoint. Please contact your administrator. header: x-databricks-org-id: xxxxxxxxxxxx header: strict-transport-security: max-age=31536000; includeSubDomains; preload header: x-content-type-options: nosniff header: server: databricks header: transfer-encoding: chunked
And this error loops for 900 seconds until I get the first error log on OpenSession
I will try to roll back to check if previous version works.
susodapop
ak-dwivedi
chkp-orsaa
aAEeLll
yunbodeng-db
main(server_hostname = args.server_hostname, http_path = args.http_path, schema = args.schema, access_token = args.access_token) File "extract.py", line 24, in main dbx_extractor.execute_sql(sql_query, server_hostname, http_path, access_token) File "/mnt/azureml/cr/j/2c91464c9e0743f5860e8f48b407683d/exe/wd/dbextractor.py", line 23, in execute_sql with db_sql.connect( File "/azureml-envs/azureml_9806bfc7c716eaf0df12bea8512e961b/lib/python3.8/site-packages/databricks/sql/init.py", line 50, in connect return Connection(server_hostname, http_path, access_token, **kwargs) File "/azureml-envs/azureml_9806bfc7c716eaf0df12bea8512e961b/lib/python3.8/site-packages/databricks/sql/client.py", line 189, in init self._session_handle = self.thrift_backend.open_session( File "/azureml-envs/azureml_9806bfc7c716eaf0df12bea8512e961b/lib/python3.8/site-packages/databricks/sql/thrift_backend.py", line 464, in open_session response = self.make_request(self._client.OpenSession, open_session_req) File "/azureml-envs/azureml_9806bfc7c716eaf0df12bea8512e961b/lib/python3.8/site-packages/databricks/sql/thrift_backend.py", line 393, in make_request self._handle_request_error(error_info, attempt, elapsed) File "/azureml-envs/azureml_9806bfc7c716eaf0df12bea8512e961b/lib/python3.8/site-packages/databricks/sql/thrift_backend.py", line 261, in _handle_request_error raise network_request_error databricks.sql.exc.RequestError: Error during request to server