search

microsoft / Purview-ADB-Lineage-Solution-Accelerator

A connector to ingest Azure Databricks lineage into Microsoft Purview
MIT License
90 stars 55 forks source link

Token Authentication error #230

Open jijack opened 5 months ago

jijack commented 5 months ago

I have an Authentication error on purview token while testing the lineage solution accellerator.

Steps to reproduce the behavior:

  1. Follow the base deploy guide only: https://github.com/microsoft/Purview-ADB-Lineage-Solution-Accelerator/blob/release/2.3/deploy-base.md
  2. At the end of the step: https://github.com/microsoft/Purview-ADB-Lineage-Solution-Accelerator/blob/release/2.3/deploy-base.md#install-openlineage-on-your-databricks-cluster Then I try to run a sample databricks test notebook (pulls data from lake parquet A to a new parquet on lake B with spark using commands spark.read.format("parquet").load(dbfs_path) and df.write.parquet(dbfs_path, mode="overwrite") ) with an interactive cluster configured with spark conf and the init script provided (not in DBFS as told in the guide but in workspace, because ad dbricks say init scripts in DBFS are deprecated)

This error pops out when monitoring PurviewOut function after running the dbricks notebook:

image

From the error trace i would say i've entered a wrong token, but everything looks normal to me, may i ask which should be the correct way to debug this and ensure the various keys are set correctly? Thanks!