search

microsoft / sql-spark-connector

Apache Spark Connector for SQL Server and Azure SQL
Apache License 2.0
273 stars 116 forks source link

Is there a way to set transaction isolation level for a session? #262

Closed aurokk closed 2 months ago

aurokk commented 2 months ago

Hi!

I want to read data from a db without any locks cuz that db is used by a service running in production, so i want to set transaction isolation level for the session to read_uncommitted or sth. Is there any way to do that?

By default it's read_committed

SELECT transaction_isolation_level FROM sys.dm_exec_sessions where session_id = @@SPID

+---------------------------+
|transaction_isolation_level|
+---------------------------+
|                          2|
+---------------------------+

ps: 2 = read_committed

I configure the read operation this way:

session.read.format("com.microsoft.sqlserver.jdbc.spark")
            .option("isolationLevel", "READ_UNCOMMITTED")
            .option("mssqlIsolationLevel", "READ_UNCOMMITTED")
            .option(
                "url",
                f"jdbc:sqlserver://{project.connectionString.host};"
                f"databaseName={project.connectionString.databaseName}",
            )
            .option("user", project.connectionString.user)
            .option("password", project.connectionString.password)
            .option("query", query)
            .load()

upd 1: tried to set sessionInitStatement

session.read.format("com.microsoft.sqlserver.jdbc.spark")
            .option("isolationLevel", "READ_UNCOMMITTED")
            .option("mssqlIsolationLevel", "READ_UNCOMMITTED")
            .option("sessionInitStatement", "SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;")
            .option(
                "url",
                f"jdbc:sqlserver://{project.connectionString.host};"
                f"databaseName={project.connectionString.databaseName}",
            )
            .option("user", project.connectionString.user)
            .option("password", project.connectionString.password)
            .load()

result
+---------------------------+
|transaction_isolation_level|
+---------------------------+
|                          2|
+---------------------------+

upd 2: tried to add SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED to query directly

SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; SELECT transaction_isolation_level FROM sys.dm_exec_sessions where session_id = @@SPID

---------------------------------------------------------------------------
SparkConnectGrpcException                 Traceback (most recent call last)
Cell In[6], [line 1](vscode-notebook-cell:?execution_count=6&line=1)
----> [1](vscode-notebook-cell:?execution_count=6&line=1) Sql.query(session, vault, "SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; SELECT transaction_isolation_level FROM sys.dm_exec_sessions where session_id = @@SPID").show()

File /usr/local/lib/python3.11/site-packages/pyspark/sql/connect/dataframe.py:996, in DataFrame.show(self, n, truncate, vertical)
    [995](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/dataframe.py:995) def show(self, n: int = 20, truncate: Union[bool, int] = True, vertical: bool = False) -> None:
--> [996](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/dataframe.py:996)     print(self._show_string(n, truncate, vertical))

File /usr/local/lib/python3.11/site-packages/pyspark/sql/connect/dataframe.py:753, in DataFrame._show_string(self, n, truncate, vertical)
    [741](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/dataframe.py:741)     except ValueError:
    [742](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/dataframe.py:742)         raise PySparkTypeError(
    [743](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/dataframe.py:743)             error_class="NOT_BOOL",
    [744](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/dataframe.py:744)             message_parameters={
   (...)
    [747](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/dataframe.py:747)             },
    [748](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/dataframe.py:748)         )
    [750](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/dataframe.py:750) pdf = DataFrame.withPlan(
    [751](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/dataframe.py:751)     plan.ShowString(child=self._plan, num_rows=n, truncate=_truncate, vertical=vertical),
    [752](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/dataframe.py:752)     session=self._session,
--> [753](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/dataframe.py:753) ).toPandas()
    [754](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/dataframe.py:754) assert pdf is not None
    [755](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/dataframe.py:755) return pdf["show_string"][0]

File /usr/local/lib/python3.11/site-packages/pyspark/sql/connect/dataframe.py:1663, in DataFrame.toPandas(self)
   [1661](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/dataframe.py:1661)     raise Exception("Cannot collect on empty session.")
   [1662](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/dataframe.py:1662) query = self._plan.to_proto(self._session.client)
-> [1663](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/dataframe.py:1663) return self._session.client.to_pandas(query)

File /usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:873, in SparkConnectClient.to_pandas(self, plan)
    [869](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:869) (self_destruct_conf,) = self.get_config_with_defaults(
    [870](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:870)     ("spark.sql.execution.arrow.pyspark.selfDestruct.enabled", "false"),
    [871](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:871) )
    [872](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:872) self_destruct = cast(str, self_destruct_conf).lower() == "true"
--> [873](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:873) table, schema, metrics, observed_metrics, _ = self._execute_and_fetch(
    [874](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:874)     req, self_destruct=self_destruct
    [875](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:875) )
    [876](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:876) assert table is not None
    [878](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:878) schema = schema or from_arrow_schema(table.schema, prefer_timestamp_ntz=True)

File /usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:1283, in SparkConnectClient._execute_and_fetch(self, req, self_destruct)
   [1280](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:1280) schema: Optional[StructType] = None
   [1281](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:1281) properties: Dict[str, Any] = {}
-> [1283](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:1283) for response in self._execute_and_fetch_as_iterator(req):
   [1284](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:1284)     if isinstance(response, StructType):
   [1285](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:1285)         schema = response

File /usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:1264, in SparkConnectClient._execute_and_fetch_as_iterator(self, req)
   [1262](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:1262)                     yield from handle_response(b)
   [1263](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:1263) except Exception as error:
-> [1264](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:1264)     self._handle_error(error)

File /usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:1503, in SparkConnectClient._handle_error(self, error)
   [1490](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:1490) """
   [1491](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:1491) Handle errors that occur during RPC calls.
   [1492](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:1492) 
   (...)
   [1500](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:1500) Throws the appropriate internal Python exception.
   [1501](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:1501) """
   [1502](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:1502) if isinstance(error, grpc.RpcError):
-> [1503](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:1503)     self._handle_rpc_error(error)
   [1504](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:1504) elif isinstance(error, ValueError):
   [1505](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:1505)     if "Cannot invoke RPC" in str(error) and "closed" in str(error):

File /usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:1539, in SparkConnectClient._handle_rpc_error(self, rpc_error)
   [1537](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:1537)             info = error_details_pb2.ErrorInfo()
   [1538](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:1538)             d.Unpack(info)
-> [1539](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:1539)             raise convert_exception(info, status.message) from None
   [1541](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:1541)     raise SparkConnectGrpcException(status.message) from None
   [1542](https://vscode-remote+dev-002dcontainer-002b7b22686f737450617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e61222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226465736b746f702d6c696e7578227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f6b6f63686e65762f5265706f732f636f636f6e612f2e646576636f6e7461696e65722f30302d737061726b2d636f6e6e6563742d72656d6f74652f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d.vscode-resource.vscode-cdn.net/usr/local/lib/python3.11/site-packages/pyspark/sql/connect/client/core.py:1542) else:

SparkConnectGrpcException: (com.microsoft.sqlserver.jdbc.SQLServerException) Incorrect syntax near the keyword 'SET'.

P.s.: i don't think it's spark connect related cuz i set a query as an option, but not 100% sure :-)

Env:

aurokk commented 2 months ago

Looks like this functionality is not in the scope of the connector, so i'm closing the issue. https://github.com/microsoft/sql-spark-connector/blob/master/src/main/scala/com/microsoft/sqlserver/jdbc/spark/DefaultSource.scala#L26

dazfuller commented 2 months ago

Have you tried just issuing a table hint in the query? So something like

select col1, col2 from table (NOLOCK) where col1 = 'condition'