Closed fredriv closed 1 year ago
@fredriv @ulixius9 we are applying lowercase normalizer in ES to avoid this issue
"type": "keyword",
"normalizer": "lowercase_normalizer"
},```
@harshach I believe we are directly calling the backend api instead of querying to ES first here
I get a 404 when trying to look up the table with the lowercase ID:
curl -i http://localhost:8585/api/v1/tables/name/Warehouse_Dev.raw_dev.fisk_cloudsql_public.ost
HTTP/1.1 404 Not Found
Date: Thu, 22 Sep 2022 07:50:19 GMT
Content-Type: application/json
Content-Length: 100
{"code":404,"message":"table instance for Warehouse_Dev.raw_dev.fisk_cloudsql_public.ost not found"}
But it works when using the uppercase ID:
curl -I http://localhost:8585/api/v1/tables/name/Warehouse_Dev.RAW_DEV.FISK_CLOUDSQL_PUBLIC.OST
HTTP/1.1 200 OK
Date: Thu, 22 Sep 2022 07:53:06 GMT
Content-Type: application/json
Content-Length: 2316
@harshach I believe we are directly calling the backend api instead of querying to ES first here
@ulixius9 ES gets called in fqn.build
to get the FQN name. We then call the API directly, but we should have been able to find the entity through ES first
I confirmed with the user that this is happening using PostgreSQL as DB for the OM server.
https://github.com/open-metadata/OpenMetadata/pull/9079 has fixed this.
@fredriv, please, reopen the issue if the error still happens on 0.13.1
.
Affected module Backend and ingestion framework
Describe the bug I've ingested tables from Postgres and Snowflake into Open Metadata. Tables are synced between Postgres and Snowflake using Fivetran.
I'm trying to use the new Fivetran connector to set up lineage between the Postgres and Snowflake tables. The connector manages to find the Fivetran pipelines, but it fails to locate the correct Snowflake table in Open Metadata since the Snowflake tables use UPPER CASE for the IDs whereas Fivetran (Postgres?) seems to use lower case. So the connector tries to look up a Snowflake table based on FQN
Warehouse_Dev.raw_dev.fisk_cloudsql_public.ost
which fails because it should beWarehouse_Dev.RAW_DEV.FISK_CLOUDSQL_PUBLIC.OST
To Reproduce
Expected behavior
I expected the Fivetran connector to find both the Postgres and Snowflake tables and set up lineage between them.
Version:
openmetadata-ingestion[docker]==0.12.0.2
Additional context
Excerpt from the Fivetran connector debug log: