Databricks python activity lineage issue

[Kishor-Radhakrishnan]

Describe the bug We have some ADF orchectrated python activity jobs which trigger a python file in dbfs.

Lineage for the same is not getting reflected in purview. I can see some events related to that is coming to purview out function with some errors.

To Reproduce Steps to reproduce the behavior:

1. Create a python activity job from ADF .
2. Observe the lineage

Logs

{"_producer":"https://github.com/OpenLineage/OpenLineage/tree/0.18.0/integration/spark","_schemaURL":"https://openlineage.io/spec/1-0-5/OpenLineage.json#/$defs/RunFacet","spark-version":"3.0.1","openlineage-spark-version":"0.18.0"}}},"job":{"namespace":"adb-2636379187030124.4#default","name":"adb-2636379187030124_4_azuredatabricks_net_execute_create_data_source_table_as_select_command_smds_C_ALT_CODE","facets":{}},"inputs":[{"namespace":"dbfs","name":"/mnt/datalake-prod/raw/smds/initial_reload/MDM_INFM_SMDS.C_ALT_CODE","facets":{"dataSource":{"_producer":"https://github.com/OpenLineage/OpenLineage/tree/0.18.0/integration/spark","_schemaURL":"https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet","name":"dbfs","uri":"dbfs"},"schema":{"_producer":"https://github.com/OpenLineage/OpenLineage/tree/0.18.0/integration/spark","_schemaURL":"https://openlineage.io/spec/facets/1-0-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet","fields":[{"name":"ROWID_OBJECT","type":"string"},{"name":"CREATOR","type":"string"},{"name":"CREATE_DATE","type":"string"},{"name":"UPDATED_BY","type":"string"},{"name":"LAST_UPDATE_DATE","type":"string"},{"name":"CONSOLIDATION_IND","type":"string"},{"name":"DELETED_IND","type":"string"},{"name":"DELETED_BY","type":"string"},{"name":"DELETED_D 2023-02-28T05:18:29Z [Information] Executed 'Functions.PurviewOut' (Succeeded, Id=cb9445d3-a201-4f65-b2e3-c54fcdcb51c7, Duration=117ms) 2023-02-28T05:18:29Z [Error] AdbClient-GetSingleAdbJobAsync: error, message: Unexpected character encountered while parsing value: {. Path 'tasks[0].libraries[0].pypi', line 1, position 1516. 2023-02-28T05:18:32Z [Information] Executing 'Functions.PurviewOut' (Reason='(null)', Id=181ec998-7e09-4ba1-ad01-b5474fdb9e6e) 2023-02-28T05:18:32Z [Information] Trigger Details: PartionId: 0, Offset: 124585262552, EnqueueTimeUtc: 2023-02-28T05:18:31.5100000Z, SequenceNumber: 3045, Count: 1 2023-02-28T05:18:32Z [Verbose] Sending invocation id:181ec998-7e09-4ba1-ad01-b5474fdb9e6e 2023-02-28T05:18:32Z [Verbose] Posting invocation id:181ec998-7e09-4ba1-ad01-b5474fdb9e6e on workerId:a70f5206-8b51-40ea-9b85-2916ca30d772

Screenshots If applicable, add screenshots to help explain your problem.

Python file location is in dbfs

[wjohnson]

Hi, @Kishor-Radhakrishnan thank you for bringing this to our attention!

Would you please try the hotfix/librariesInTasks.

With the fix, I was able to create a Python File job in ADF and have it run on a cluster configured to work with the solution accelerator and then observe lineage in the Microsoft Purview UI.

[Kishor-Radhakrishnan]

@wjohnson Thanks for the quick fix . I deployed the fix and I can see lineage for one of our dev flow.

Will do further validations.

[Kishor-Radhakrishnan]

[wjohnson]

Thank you so much Kishor! We've merged this into main and will get this into the release branch as soon as we get the last few issues you've reported.