search

estuary / connectors

Connectors for capturing data from external data sources
Other
38 stars 10 forks source link

materializations: support base64 encoded strings as binary columns #1573

Open williamhbaker opened 2 months ago

williamhbaker commented 2 months ago

In https://github.com/estuary/connectors/pull/1563 and https://github.com/estuary/connectors/pull/1572, support for base64-encoded strings to be materialized as binary columns was added to materialize-databricks, for non-key fields. This required some wrangling of the staged textual data into a binary column, and managing pre-existing materializations which have always materialized these fields as strings previously.

At some point we should do the same for other materializations. This is not likely to be a quick thing for any of them, so it will take some time, and for now I am just logging this issue as something to follow up on. It would be significantly easier if we had a good strategy for changing materialized column types in a backward-compatible way, which may eventually emerge.

williamhbaker commented 1 month ago

Support for key fields of binary types was added to materialize-databricks in https://github.com/estuary/connectors/pull/1575, so having that capability across the board would be good.