[Feature] Support OCI Dataflow as a backend for dbt-spark

[davidallan]

Is this your first time submitting a feature request?

• [X] I have read the expectations for open source contributors
• [X] I have searched the existing issues, and I could not find an existing issue for this feature
• [X] I am requesting a straightforward extension of existing dbt-spark functionality, rather than a Big Idea better suited to a discussion

Describe the feature

OCI Dataflow is an Oracle serverless Spark solution that also has a SQL Endpoint accessible via Simba ODBC driver for example. The Simba OCI driver is similar to the Databricks one but the properties are slightly different. One of the changes is that sparkServerType property passed in the connection must have value DFI - in the code here it is hard-wired to the value 3. The odbc code path also fails if cluster is not set or endpoint is not set, these are not needed for the OCI Simba ODBC driver (along with validation failure). if creds.cluster is not None: ... elif creds.endpoint is not None: ... else: ...raises error

A working format for the Simba OCI ODBC driver has the JDBC URL as; jdbc:spark://yourendpoint.oci.oraclecloud.com/default;SparkServerType=DFI;httpPath=cliservice

I have made my own local modifications to the connections.py code (in adapters/spark/connections.py) and been able to use Delta format in OCI Dataflow SQL endpoint to do incremental extracts and merges for example. See here; https://github.com/davidallan/dbt-spark/blob/main/dbt/adapters/spark/connections.py

Describe alternatives you've considered

No response

Who will this benefit?

This will open up DBT to the OCI serverless spark platform.

Are you interested in contributing this feature?

Yes, interested in doing this.

Anything else?

Here is the doc on OCI SQL Endpoints for OCI Dataflow; https://docs.oracle.com/en-us/iaas/data-flow/using/sql-endpoints-connect.htm

[github-actions[bot]]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[amychen1776]

At this time we are unfortunately unable to prioritize this work but I will keep this issue open to track feedback/requests for this.