faros-ai / airbyte-local-cli

CLI for running Airbyte sources & destinations locally without Airbyte server
Apache License 2.0
31 stars 3 forks source link

Rework destination cfg to support both Faros editions #44

Closed ypc-faros closed 2 years ago

ypc-faros commented 2 years ago

Description

Provide description here

The current logic to write the destination config handles the Faros destination (farosai/airbyte-faros-destination) as a special case. We assume the Faros API URL, API key and graph to have been specified via --dst.faros_api_url,--dst.faros_api_key and --dst.graph respectively, and we use these values + "edition": "cloud" as the destination configuration.

This overwrites other values that might have been specified via --dst.edition_configs and is also Faros Cloud edition specific (doesn't support the Community Edition case).

This PR fixes the overwriting issue, while keeping the behavior that the special args (--dst.faros_api_url,--dst.faros_api_key and --dst.graph) are still included in the appropriate place in the configuration, for compatibility with clients are already using it. If we could live with a breaking change, we could get rid of this logic and just have clients specify these args according to their paths in the spec (--dst.edition_configs.faros_api_url,--dst.edition_configs.faros_api_key and --dst.edition_configs.graph).

There is no special logic needed to support Faros Community Edition(or Cloud Edition, except for the sake of keeping compatibility). Clients just need to pass the config values as needed. E.g.,

--dst.edition_configs '{"edition": "community","hasura_admin_secret": "admin", "hasura_url":"http://localhost:8080"}' \

or

--dst.edition_configs.edition 'community' \
--dst.edition_configs.hasura_admin_secret 'admin' \
--dst.edition_configs.hasura_url 'http://localhost:8080' \