airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.91k stars 4.09k forks source link

Add Actor Definition Context to LaunchDarkly #25455

Closed pedroslopez closed 1 year ago

pedroslopez commented 1 year ago

What

In order to apply feature flags for overriding connector versions, as described in the Multiple Connector Versions tech spec, we'll need to be able to target based on the actor definition IDs. These IDs refer to connectors available in our Registry, are stored in the airbyte database's actor_definition table and are referenced internally usually via StandardDestinationDefinition and StandardSourceDefinition.

Since we have a similar internal representation for Source / Destination (actors) and there are already corresponding Feature Flag contexts for these (Source, Destination), I would suggest adding two new contexts:

These just need a UUID to be used as key.

pedroslopez commented 1 year ago

Alternatively, we could extend the current Source and Destination contexts with a source definition id or destination definition id attribute, respectively.

colesnodgrass commented 1 year ago

@pedroslopez Adding SourceDefinition and DestintationDefinition contexts seems like the best approach.

A potential issue will adding the definition-id to the existing Source and Destination contexts-types is that the context-kind for Source and SourceDefinition would be identical. This would force us to differentiate between them based on the attribute defined in the LD rule to determine which would context be used. I.e. key for Source and definition for SourceDefinition, which isn't incorrect but differs enough from the current way we have implemented contexts that I believe it would only cause confusion. The last thing we want is for feature-flags to be confusing.


There is one additional step (besides the PR adding these new context types) that is necessary to ensure these context-types are actually supported. These contexts will need to be manually added to the LD Context list by someone that has LD administrative privileges. Make sure whatever kind value you define in the code matches the kind value defined in LD.