ConduitIO / conduit-commons

Shared code between different Conduit repositories.
Apache License 2.0
3 stars 2 forks source link

paramgen: add support for custom field (un)marshalling #142

Open hariso opened 1 week ago

hariso commented 1 week ago

Feature description

The problem: Here's an example from a connector middleware struct:

type SourceWithSchemaExtraction struct {
    SchemaType schema.Type `json:"sdk.schema.extract.type" validate:"inclusion=avro" default:"avro"`
}

where schema.Type is an alias of schema.Type in conduit-commons.

When parsing SourceWithSchemaExtraction with sdk.Util.ParseConfig, we get an error saying that there's a type mismatch, because the default value is "avro" (a string), and the target value is an int. The reason is that in Config.Validate() we validate the types.

Possible solution:

  1. Remove type validation in Config.Validate() altogether. In the Connector SDK, it's called only when parsing a config, and after that we decode the map into a struct, where the types are validated too.
  2. Use one of mapstructure's hooks (link), so that developers can implement a MarshalText/UnmarshalText method and customize parsing.
hariso commented 6 days ago

We have a workaround for this, but it's a useful feature that we plan to add in one of the next releases.