open-metadata / OpenMetadata

OpenMetadata is a unified metadata platform for data discovery, data observability, and data governance powered by a central metadata repository, in-depth column level lineage, and seamless team collaboration.
https://open-metadata.org
Apache License 2.0
5.5k stars 1.04k forks source link

Profiler BigQuery error: 'Schema pattern not allowed' #18503

Closed Cir02 closed 3 days ago

Cir02 commented 3 days ago

Affected module Does it impact the UI, backend or Ingestion Framework?

Profiler error in BigQuery connection with ADC.

Describe the bug After launch openmetadata_ingestion for metadata and lineage, both of them are okey. When launching the pipeline for profiler with Enable Debug Log enabled and Schema Filter Pattern point to only one schema, we get the error: {'Schema pattern not allowed: Bigquery_dev_3.xxx.business_zone.xxx': 'Schema pattern not allowed'}. Not getting the right profiler for this schema.

To Reproduce

Nomal profiler pipeline only include one schema of the database.

image

image

Expected behavior get a full profiler of all columns and sample data

Version:

Additional context all roles for bigquery connection in documentation are granted.

ulixius9 commented 3 days ago

@Cir02 you need to disable the Use FQN for filtering flag, once that is done you will be able to run profiler with provided schema.

you can read the description about Use FQN for filtering to know more.

Cir02 commented 3 days ago

thx!