airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.42k stars 3.98k forks source link

[PYTHON NORMALIZATION] Destination Redshift: Make possible to process SUPER Data Type during normalization #9411

Closed alexandertsukanov closed 2 years ago

alexandertsukanov commented 2 years ago

Tell us about the problem you're trying to solve

In scope of https://github.com/airbytehq/airbyte/issues/5050 was requested to add support of Redshift SUPER type. In scope of PR https://github.com/airbytehq/airbyte/pull/9407 was added support for SUPER data type on Java side.

Describe the solution you’d like

In scope of this ticket the support of SUPER Datatype for Normalization should be added.

Additional context

The PR for JAVA side created here https://github.com/airbytehq/airbyte/pull/9407

SQL Queries Examples for SUPER datatype

INSERT

insert into _airbyte_raw_some_table (_airbyte_ab_id, _airbyte_data)
values (3, JSON_PARSE('{"r_nations":[
      {"n_comment":" haggle. carefully final deposits detect slyly agai",
         "n_nationkey":0,
         "n_name":"ALGERIA"
      },
      {"n_comment":"ven packages wake quickly. regu",
         "n_nationkey":5,
         "n_name":"ETHIOPIA"
      },
      {"n_comment":" pending excuses haggle furiously deposits. pending, express pinto beans wake fluffily past t",
         "n_nationkey":14,
         "n_name":"KENYA"
      },
      {"n_comment":"rns. blithely bold courts among the closely regular packages use furiously bold platelets?",
         "n_nationkey":15,
         "n_name":"MOROCCO"
      },
      {"n_comment":"s. ironic, unusual asymptotes wake blithely r",
         "n_nationkey":16,
         "n_name":"MOZAMBIQUE"
      }
   ]
}'));

SELECT

select _airbyte_data.r_nations[0]
from _airbyte_raw_some_table;
alexandertsukanov commented 2 years ago

You can play with my DB created in Integration Test Redshift instance. otsukanov_test.source_schema