open-metadata / OpenMetadata

OpenMetadata is a unified metadata platform for data discovery, data observability, and data governance powered by a central metadata repository, in-depth column level lineage, and seamless team collaboration.
https://open-metadata.org
Apache License 2.0
5.58k stars 1.05k forks source link

Bug in Profiler with Postgresql 11.19 #12501

Closed isoctcolo closed 1 year ago

isoctcolo commented 1 year ago

Affected module Does it impact the UI, backend or Ingestion Framework? Profiler Describe the bug A clear and concise description of what the bug is. Profiler fails [2023-07-18 08:30:29,437] {metadata_rest.py:111} WARNING - Failed to sink profiler & test data for Datavault Dev Old.postgres.learning.lms_users: Non-standard token 'NaN': enable JsonReadFeature.ALLOW_NON_NUMERIC_NUMBERS to allow at [Source: (org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream); line: 1, column: 44532] (through reference chain: org.openmetadata.schema.api.data.CreateTableProfile["columnProfile"]->java.util.ArrayList[44]) To Reproduce

Screenshots or steps to reproduce It fails with this table structure: CREATE TABLE learning.lms_users ( user_id text NOT NULL, username text NULL, first_name text NULL, last_name text NULL, email text NULL, uuid text NULL, is_manager bool NULL, fullname text NULL, lang_code text NULL, last_access_date timestamp NULL, last_update timestamp NULL, creation_date timestamp NULL, status text NULL, avatar text NULL, "language" text NULL, expiration_date text NULL, "level" text NULL, email_validation_status text NULL, send_notification text NULL, timezone text NULL, date_format text NULL, field_4 text NULL, field_5 text NULL, field_8 text NULL, field_9 text NULL, field_12 text NULL, field_13 text NULL, field_16 text NULL, field_17 text NULL, field_20 text NULL, field_23 text NULL, field_24 text NULL, field_25 text NULL, encoded_username text NULL, multidomains text NULL, managers text NULL, role_id text NULL, role_title text NULL, active_subordinates_count int8 NULL, actions text NULL, expired bool NULL, "manager_names.1.manager_title" text NULL, "manager_names.1.manager_name" text NULL, "manager_names.1.manager_username" text NULL, "manager_names.1.manager_id" float8 NULL, field_26 text NULL, field_28 text NULL, field_29 text NULL, newsletter_optout text NULL, newsletter_optout_date text NULL, CONSTRAINT lms_users_pk PRIMARY KEY (user_id) );

Expected behavior A clear and concise description of what you expected to happen. Successful profiler run

Version:

Additional context Add any other context about the problem here.

ayush-shah commented 1 year ago

@isoctcolo feel free to contribute for the same as we don't have many requests for the same. Looks like 11.19 / 11.21 is near EOL support, Thanks. Let me know if that works for you, or else will be closing the ticket in couple of days. We can reopen it if someone wants to work on it post that based on request.

isoctcolo commented 1 year ago

@ayush-shah please close it - you are right and 11.x is a special version for maintaining