Evaluate Databricks Destination and scope path to beta

[grishick]

We need to answer the following questions about our databricks destination connector:

• does it work as expected (as defined in the documentation)?
• what limitations it has currently, how critical each of the limitations are, and how hard it is to remove each limitation?
• how does the user experience of our databricks destination compare to other connectors (Stich, Fivetran) and what are the UX gaps?
• how important each UX gap is and how hard it will be to close?

When filing issues that come out of this research, please link them to this epic

[grishick]

timeboxing to 13 points

[suhomud]

Opened issues

• Currently, this connector requires 30+MB of memory for each stream https://github.com/airbytehq/airbyte/issues/11424. Notes: I see it as a most have before the Beta.
• Destination Databricks - currently uses int type when long needed (cast to 'null') https://github.com/airbytehq/airbyte/issues/21943.
• Destination Databricks - Destination process message delivery failed https://github.com/airbytehq/airbyte/issues/21860
• Is Airbyte really only able to transfer less than 2MB/s? https://github.com/airbytehq/airbyte/issues/21318. Notes: Would be solved with file based buffering? Could it be not related to the buffering?
• Destination Databricks: Support Google Cloud/GCS for Delta staging tables https://github.com/airbytehq/airbyte/issues/21292. Notes: Related to Question number 3. Currently we supports only S3 and Azure data sources.
• Destination Databricks: Datatypes missing in SQL CREATE-statement https://github.com/airbytehq/airbyte/issues/21193. Notes: there is an PR from community contributor https://github.com/airbytehq/airbyte/pull/21238.
• Destination Databricks: Replicate data into new catalog instead of metastore https://github.com/airbytehq/airbyte/issues/20953
• Unable to authenticate Azure blob with Databricks Lakehouse connector https://github.com/airbytehq/airbyte/issues/19011. Notes: From the comment section I see This is a documentation issue. It needs to be an account level SAS rather than a container level SAS.. So most-likely we need to check and modify the documentation which describe permissions setup.
• Destination Databricks - Does not escape commas in JSON Fields https://github.com/airbytehq/airbyte-internal-issues/issues/1107. Notes: Could be related to https://github.com/airbytehq/airbyte/issues/21193
• Update destination-databricks to handle the new data types protocol https://github.com/airbytehq/airbyte/issues/18548
• Destination S3: Credentials not being accepted, but are valid and working on CLI https://github.com/airbytehq/airbyte/issues/18277. Notes: Another ticket about credentials not being accepted so the docs should be revisit and possibly the check method required revisiting
• Enable integration test for Databricks with Azure storage https://github.com/airbytehq/airbyte-internal-issues/issues/1046. Notes: checked this one and it's still relevant the Azure tests are marked with @Disabled
• Destination S3: Correctly generate int64 values https://github.com/airbytehq/airbyte/issues/17564. Notes: from the comment section this issue affects databricks also
• Add SSH tunneling ability to Databricks Destination connector https://github.com/airbytehq/airbyte/issues/16309. Notes: based on @grishick comment. Should we close it?
• Pass open-source specific user agent parameter in Databricks connector https://github.com/airbytehq/airbyte-internal-issues/issues/643
• Destination Databricks: failed to convert JSON to Avro https://github.com/airbytehq/airbyte/issues/12949
• Make SecondSync acceptance test run for staging destinations https://github.com/airbytehq/airbyte/issues/12147
• Destination Databricks: use s3 location on create database https://github.com/airbytehq/airbyte/issues/11635
• Apply buffering changes to Databricks Destination https://github.com/airbytehq/airbyte/issues/11424

  Opened PRs

• Allow Incremental Deduped sync https://github.com/airbytehq/airbyte/pull/14445. Notes: The downside of this Pr is that it supports only csv staging and no Parquet. Also it’s conflicting with File-base buffer and if we want to support it the PR needs to be almost re-written.
• Destination Databricks: replicate into specified catalog using three level namespace https://github.com/airbytehq/airbyte/pull/20954.

• Destination Databricks: handle optional and object types in input schema (azure only) https://github.com/airbytehq/airbyte/pull/21238.

  Questions

  1. Databricks connector use in-memory buffer. Should we migrate to File-based buffer?
  2. Databricks service has Delta table modes:
     • Managed storage (no source data is needed)
     • External storage (like S3, Azure)

  Currently we are supporting External storage only so should we implement Managed storage in a scope of moving to Beta?

3. Current implementation supports S3 and Azure external storages. Should we extend it before moving to Beta? The full list of supported sources can be found here

[grishick]

Still TBD: manually test the connector

[grishick]

1. Databricks connector use in-memory buffer. Should we migrate to File-based buffer?

Yes

Currently we are supporting External storage only so should we implement Managed storage in a scope of moving to Beta?

Yes

Current implementation supports S3 and Azure external storages. Should we extend it before moving to Beta? The full list of supported sources can be found here

No. S3 and Azure are OK for Beta.