search

transferwise / pipelinewise-target-redshift

Singer.io Target for Amazon Redshift - PipelineWise compatible
https://transferwise.github.io/pipelinewise/
Other
12 stars 65 forks source link

Support for custom column types in configuration #19

Open mlavoie-sm360 opened 5 years ago

mlavoie-sm360 commented 5 years ago

Describe the feature

The default column type values can be prohibitive but are very convenient to get started quickly. It would be nice to be able to provide specific column types for specific columns in YML config to further configure the creation of the Redshift tables.

...
# ------------------------------------------------------------------------------
# Source to target Schema mapping
# ------------------------------------------------------------------------------
schemas:

  - source_schema: "source"             # Source schema (aka. database) in MySQL/ MariaDB with tables
    target_schema: "target"             # Target schema in the destination Data Warehouse

    tables:
      - table_name: "a_table"
        replication_method: "FULL_TABLE"
        column_definitions:
          - column: "column_1"
             type: "VARCHAR(512)"
          - column: "column_10"
             type: "DECIMAL(2,1)"
...

Who will this benefit?

This can benefit anyone using data types that do not fit the provided defaults for column types.

koszti commented 5 years ago

this is a little bit complicated and needs to be in sync with Columns Versioning on Schema Changes

How we should alter the table if the column type is changing in the source db? Should we just ignore auto-versioning the column if the column types defined in the YAML file? What is the expected behaviour if selected column type is not a good fit for the data or cannot be converted automatically?