search

meltano / sdk

Write 70% less code by using the SDK to build custom extractors and loaders that adhere to the Singer standard: https://sdk.meltano.com
https://sdk.meltano.com
Apache License 2.0
87 stars 64 forks source link

feat: Configurable object/column name formatting options for targets #2490

Open dluo-sig opened 2 weeks ago

dluo-sig commented 2 weeks ago

Feature scope

Targets (data type handling, batching, SQL object generation, tests, etc.)

Description

It would be helpful if there is a way to configure the column and object names that get mapped from the extractor.

  1. direct mapping - names as-is
  2. snake_case
  3. PascalCase

Nice to have: some targets also have the ability to specify column names that are outside of the normal specs. For example, SQL Server can have column names with any character when qualified with square brackets, PostgreSQL/Snowflake can do the same with double quotes, etc. This would be an additional option to use the appropriate identifiers across the board when referencing objects/fields at the target. It would not be desirable to always force this option, as for example, Snowflake becomes case-sensitive when double quotes are provided.

visch commented 2 weeks ago

Yes, good part is there's already a function for this https://github.com/meltano/sdk/blob/63340915751b798bade84572a6d6bda2da9e2d1b/singer_sdk/sinks/sql.py#L134

I overrode this as I wanted "direct mapping" in your list to be the default not snake_case see https://github.com/MeltanoLabs/target-postgres/blob/1e59be2750961876d52b8e69cf05c5eb06cb13b4/target_postgres/sinks.py#L280-L282

There was a longer discussion about this here https://github.com/meltano/sdk/issues/1205

I think the idea of a config option to choose between them is a good one!

dluo-sig commented 2 weeks ago

@edgarrmondragon had proposed using the humps library for this.