search

open-metadata / OpenMetadata

OpenMetadata is a unified metadata platform for data discovery, data observability, and data governance powered by a central metadata repository, in-depth column level lineage, and seamless team collaboration.
https://open-metadata.org
Apache License 2.0
5.39k stars 1.02k forks source link

Don't build metadata lineage with MSSQL Server when queries contain square brackets. #16424

Closed Uzifly closed 2 months ago

Uzifly commented 4 months ago

Affected module Ingestion(metadata, lineage)

Describe the bug The view definition contains queries like from [database].[schema].[table] in square brackets. Such requests are not processed by the parser and no connections are built. If we remove the square brackets in the query, then the lineage appear.

To Reproduce This problem occurs when ingesting metadata from MSSQL Server. Specify a [schema] or [table] in the view definition in square brackets and you won't get the lineage.

Expected behavior Perhaps for some reason the method in /ingestion/src/metadata/ingestion/lineage/parser.py insensitive_match does not work

Version:

Uzifly commented 4 months ago

I got debug information: Method 

insensitive_replace(
                clean_table.schema.raw_name, r"\[(.*)\]", r"\1"
            )

returns "db].[schema". You need to fix regexp. UPD:r"\[(.*?)\]", r"\1" - I think this might solve the problem

SumanMaharana commented 2 months ago

Hello @Uzifly this issue is fixed within the sqlfluff repo with this PR Closing this issue. Thanks