Redshift: allow user-defined schemas for export and sync table operations

patrick-fs commented 5 years ago

Scenario 1: "search_path" value provided for schema configuration in config.toml/redshift/DatabaseSchema

Given there is a DatabaseSchema configuration parameter in config.toml/redshift When DatabaseSchema = ”search_path” and Hauser is executed Then the tables defined in config.toml/redshift/ExportTable and config.toml/redshift/SyncTable are both created under the schema defined by the database's Search Path configuration.

Scenario 2: Non-empty and non "search_path" value provided for schema configuration in config.toml/redshift/DatabaseSchema

Given there is a DatabaseSchema configuration parameter in config.toml/redshift When DatabaseSchema = ”some_schema” and Hauser is executed Then the tables defined in config.toml/redshift/ExportTable and config.toml/redshift/SyncTable are both created under the “some_schema” schema

Scenario 3: No DatabaseSchema parameter provided in config.toml

Given a DatabaseSchema configuration parameter is not defined or DatabaseSchema = "" in config.toml/redshift When Hauser is executed Then throw an error explaining that DatabaseSchema must be provided in config.toml/redshift and include a link to redshift config documentation in the error message

Developer Notes

Be aware of backwards compatibility with existing Hauser users. The existing Hauser implementation explicitly creates export and sync tables under the public schema. Here's how I think we can support the existing behavior and also effectively communicate the change during runtime:

To support existing behavior, add a DatabaseSchema = "public" entry to example-config.toml/redshift
Scenario 3 above is intended to communicate new behavior to existing users

patrick-fs commented 5 years ago

UPDATE: I changed the first and third scenarios. Whereas I originally wrote Scenario 1 to apply the search_path when DatabaseSchema = "", I came to the conclusion that this is non-interpretable behavior and decided to make the search_path behavior explicit. I added DatabaseSchema = "" to Scenario 3.

patrick-fs commented 5 years ago

This should satisfy the schema usage patterns documented here: https://www.postgresql.org/docs/10/ddl-schemas.html#DDL-SCHEMAS-PATTERNS

patrick-fs commented 5 years ago

PR merged

fullstorydev / hauser