NorthConcepts / DataPipeline-Examples

DataPipeline Examples
Apache License 2.0
17 stars 5 forks source link

DP-3689-Create-example-to-Join-CSV-Files-Using-H2-Database #193

Closed Bryan-Noel closed 11 months ago

Bryan-Noel commented 11 months ago

@DeleTaylor can't seem to respond directly to your comment about using a SchemaTransformer so adding my response here. Using a SchemaTransformer works but I still had to set the type of the FieldDef DoB to DATE on line 67 otherwise the output would be something like 1964-09-02 00:00:00.0 when the input is only 09-02-1964. This is because the inferred type for the column during GenerateEntityFromDataset#generateEntity is DATETIME resulting in the generated ddl to use TableColumnType.TIMESTAMP for the column.

DeleTaylor commented 11 months ago

@DeleTaylor can't seem to respond directly to your comment about using a SchemaTransformer so adding my response here. Using a SchemaTransformer works but I still had to set the type of the FieldDef DoB to DATE on line 67 otherwise the output would be something like 1964-09-02 00:00:00.0 when the input is only 09-02-1964. This is because the inferred type for the column during GenerateEntityFromDataset#generateEntity is DATETIME resulting in the generated ddl to use TableColumnType.TIMESTAMP for the column.

@Bryan-Noel This might be a bug in the type inference logic. I'll take a look. 👍