search

microsoft / Purview-ADB-Lineage-Solution-Accelerator

A connector to ingest Azure Databricks lineage into Microsoft Purview
MIT License
92 stars 56 forks source link

Databricks lineage with mounts not getting captured for 11.3 LTS version #165

Closed Kishor-Radhakrishnan closed 1 year ago

Kishor-Radhakrishnan commented 1 year ago

Describe the bug Databricks lineage with mounts not getting captured for 11.3 LTS version.

Same code with same configurations working fine for 9.1 LTS version

To Reproduce

  1. Followed steps mentioned in Connector only deployment.
  2. Set up interactive cluster with 11.3 LTS version.
  3. Lineage not captured when using mount paths on ADLS Gen1 ( /mnt/mount_name)
  4. Lineage captured fine when used direct ADLS Gen1 URI instead of mount ( adl://path)
  5. Lineage captured properly in 9.1 LTS version.

Attaching logs from function app dbfs_out_file.txt

wjohnson commented 1 year ago

Thank you for your interest and use of the solution accelerator!

11.3 LTS is not officially supported yet but will be in our next release.

If you would like to, you should be able to manually upgrade your OpenLineage jar to 0.18.0 and that should support mount point extraction on 11.3.

Please let us know if you are able to install the upgraded OpenLineage jar and test this on your 11.3 cluster.

On Wed, Feb 15, 2023 at 1:40 AM Kishor Radhakrishnan < @.***> wrote:

Describe the bug Databricks lineage with mounts not getting captured for 11.3 LTS version.

Same code with same configurations working fine for 9.1 LTS version

To Reproduce

  1. Followed steps mentioned in Connector only deployment.
  2. Set up interactive cluster with 11.3 LTS version.
  3. Lineage not captured when using mount paths on ADLS Gen1 ( /mnt/mount_name)
  4. Lineage captured fine when used direct ADLS Gen1 URI instead of mount ( adl://path)
  5. Lineage captured properly in 9.1 LTS version.

Attaching logs from function app dbfs_out_file.txt https://github.com/microsoft/Purview-ADB-Lineage-Solution-Accelerator/files/10740476/dbfs_out_file.txt

— Reply to this email directly, view it on GitHub https://github.com/microsoft/Purview-ADB-Lineage-Solution-Accelerator/issues/165, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAICLF4DI5V4MGES32LDJL3WXSCAHANCNFSM6AAAAAAU4PZ5QA . You are receiving this because you are subscribed to this thread.Message ID: @.*** com>

Kishor-Radhakrishnan commented 1 year ago

with latest OpenLineage jar 0.18.0 version , lineage worked perfectly in 11.3 LTS version .

wjohnson commented 1 year ago

Glad to hear it!

On Fri, Feb 17, 2023 at 4:29 AM Kishor Radhakrishnan < @.***> wrote:

with latest OpenLineage jar 0.18.0 version , lineage worked perfectly inn 11.3 LTS version .

— Reply to this email directly, view it on GitHub https://github.com/microsoft/Purview-ADB-Lineage-Solution-Accelerator/issues/165#issuecomment-1434441633, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAICLF3UWU5AUEUJ3I527CLWX5HKPANCNFSM6AAAAAAU4PZ5QA . You are receiving this because you commented.Message ID: <microsoft/Purview-ADB-Lineage-Solution-Accelerator/issues/165/1434441633@ github.com>