search

databricks / databricks-sdk-py

Databricks SDK for Python (Beta)
https://databricks-sdk-py.readthedocs.io/
Apache License 2.0
358 stars 121 forks source link

[ISSUE] 'dict' object is not callable when m2m authentication #776

Open HugoDife opened 3 weeks ago

HugoDife commented 3 weeks ago

Description I´m trying to connect m2m to databricks following the instructions in Databricks but when running the code I am getting the following error:

INFO:databricks.sql.thrift_backend:Error during request to server: {"method": "OpenSession", "session-id": null, "query-id": null, "http-code": null, "error-message": "", "original-exception": "'dict' object is not callable", "no-retry-reason": "non-retryable error", "bounded-retry-delay": null, "attempt": "1/4", "elapsed-seconds": "0.00045418739318847656/900.0"}
Traceback (most recent call last):
  File "/Users/test/Projects/databricks-test/dabricks_test.py", line 134, in <module>
    measures3 = measure_databricks_sql_performance(dbsql_config_m2m_pro, "SELECT * FROM catalog.schema.table LIMIT 2")
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/test/Projects/databricks-test/dabricks_test.py", line 99, in measure_databricks_sql_performance
    with sql.connect(**dbsql_config, _retry_stop_after_attempts_count=4, _tls_trusted_ca_file="/Users/test/Desktop/certificate/zscaler.pem") as connection:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/test/miniconda3/envs/databricks/lib/python3.11/site-packages/databricks/sql/__init__.py", line 90, in connect
    return Connection(server_hostname, http_path, access_token, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/test/miniconda3/envs/databricks/lib/python3.11/site-packages/databricks/sql/client.py", line 247, in __init__
    self._open_session_resp = self.thrift_backend.open_session(
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/test/miniconda3/envs/databricks/lib/python3.11/site-packages/databricks/sql/thrift_backend.py", line 549, in open_session
    response = self.make_request(self._client.OpenSession, open_session_req)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/test/miniconda3/envs/databricks/lib/python3.11/site-packages/databricks/sql/thrift_backend.py", line 478, in make_request
    self._handle_request_error(error_info, attempt, elapsed)
  File "/Users/test/miniconda3/envs/databricks/lib/python3.11/site-packages/databricks/sql/thrift_backend.py", line 308, in _handle_request_error
    raise network_request_error

From what I could see the class OAuthCredentialsProvider extends from CredentialsProvider, however the class CredentialsProvider in the databricks-sql-connector dont match the output, one directly return dict while the other return Callable[[], Dict[str, str]].

Is there something I am missing? Shouldnt these two classes be compatible?

Reproduction Available in Databricks

Expected behavior Everything work and return the result query.

Is it a regression? Did this work in a previous version of the SDK? If so, which versions did you try? Didn´t test

Other Information

shivonchain commented 1 day ago

Getting the same error. Repro + stacktrace/debug logs below.

import logging
import os
import sys

from databricks import sql
from databricks.sdk.core import Config, oauth_service_principal

logging.basicConfig(
    stream=sys.stderr,
    level=logging.INFO,
    format="%(asctime)s [%(name)s][%(levelname)s] %(message)s",
)
logging.getLogger("databricks.sdk").setLevel(logging.DEBUG)
logging.getLogger("databricks.sql").setLevel(logging.DEBUG)
host = os.getenv("DATABRICKS_HOST")
http_path = os.getenv("DATABRICKS_HTTP_PATH")
client_id = os.getenv("DATABRICKS_OAUTH_CLIENT_ID")
client_secret = os.getenv("DATABRICKS_OAUTH_CLIENT_SECRET")
conn = sql.connect(
    server_hostname=host,
    http_path=http_path,
    credentials_provider=oauth_service_principal(
        Config(
            host=f"https://{host}",
            client_id=client_id,
            client_secret=client_secret,
        )
    ),
)
2024-10-23 12:28:02,003 [databricks.sdk][DEBUG] Attempting to configure auth: pat
2024-10-23 12:28:02,003 [databricks.sdk][DEBUG] Attempting to configure auth: basic
2024-10-23 12:28:02,003 [databricks.sdk][DEBUG] Attempting to configure auth: metadata-service
2024-10-23 12:28:02,004 [databricks.sdk][DEBUG] Attempting to configure auth: oauth-m2m
2024-10-23 12:28:03,414 [databricks.sdk.oauth][DEBUG] Retrieving token for <concealed by 1Password>
2024-10-23 12:28:04,137 [databricks.sql.thrift_backend][DEBUG] retry parameter: _retry_delay_min given_or_default 1.0
2024-10-23 12:28:04,137 [databricks.sql.thrift_backend][DEBUG] retry parameter: _retry_delay_max given_or_default 60.0
2024-10-23 12:28:04,137 [databricks.sql.thrift_backend][DEBUG] retry parameter: _retry_stop_after_attempts_count given_or_default 30
2024-10-23 12:28:04,137 [databricks.sql.thrift_backend][DEBUG] retry parameter: _retry_stop_after_attempts_duration given_or_default 900.0
2024-10-23 12:28:04,137 [databricks.sql.thrift_backend][DEBUG] retry parameter: _retry_delay_default given_or_default 5.0
2024-10-23 12:28:04,149 [databricks.sql.thrift_backend][DEBUG] Sending request: OpenSession(<REDACTED>)
2024-10-23 12:28:04,149 [databricks.sql.thrift_backend][INFO] Error during request to server: {"method": "OpenSession", "session-id": null, "query-id": null, "http-code": null, "error-message": "", "original-exception": "'dict' object is not callable", "no-retry-reason": "non-retryable error", "bounded-retry-delay": null, "attempt": "1/30", "elapsed-seconds": "0.0002009868621826172/900.0"}
Traceback (most recent call last):
  File "/Users/shivgupta/projects/biztech-dea/btda_dagster/file.py", line 22, in <module>
    conn = sql.connect(
           ^^^^^^^^^^^^
  File "/Users/shivgupta/projects/biztech-dea/venv/lib/python3.12/site-packages/databricks/sql/__init__.py", line 90, in connect
    return Connection(server_hostname, http_path, access_token, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shivgupta/projects/biztech-dea/venv/lib/python3.12/site-packages/databricks/sql/client.py", line 233, in __init__
    self._open_session_resp = self.thrift_backend.open_session(
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shivgupta/projects/biztech-dea/venv/lib/python3.12/site-packages/databricks/sql/thrift_backend.py", line 578, in open_session
    response = self.make_request(self._client.OpenSession, open_session_req)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shivgupta/projects/biztech-dea/venv/lib/python3.12/site-packages/databricks/sql/thrift_backend.py", line 507, in make_request
    self._handle_request_error(error_info, attempt, elapsed)
  File "/Users/shivgupta/projects/biztech-dea/venv/lib/python3.12/site-packages/databricks/sql/thrift_backend.py", line 337, in _handle_request_error
    raise network_request_error
databricks.sql.exc.RequestError: Error during request to server
shivonchain commented 1 day ago

This may be related to the discussion here: https://github.com/databricks/databricks-sql-python/issues/423

Fixed using @kravets-levko's example below

from databricks.sdk.core import Config, oauth_service_principal
from databricks import sql
import os

server_hostname = os.getenv("DATABRICKS_SERVER_HOSTNAME")

def credential_provider():
  config = Config(
    host          = f"https://{server_hostname}",
    client_id     = os.getenv("DATABRICKS_CLIENT_ID"),
    client_secret = os.getenv("DATABRICKS_CLIENT_SECRET")
  )

  def get_headers():
    return oauth_service_principal(config)               

  return get_headers

with sql.connect(server_hostname      = server_hostname,
                 http_path            = os.getenv("DATABRICKS_HTTP_PATH"),
                 credentials_provider = credential_provider) as connection: