faros-ai / airbyte-local-cli

CLI for running Airbyte sources & destinations locally without Airbyte server
Apache License 2.0
31 stars 3 forks source link

FAI-7388 Copy Faros API cfg from destination cfg to source cfg + Make writeConfig more robust #84

Closed ypc-faros closed 1 year ago

ypc-faros commented 1 year ago

Description

Copy Faros API cfg from destination cfg to source cfg. This is only done when the source is farosai/airbyte-faros-feeds-source and the destination is the Faros destination. In this case, users of the CLI can omit Faros API config arguments from the source args.

./airbyte-local.sh \
  --src farosai/airbyte-faros-feeds-source:latest \
  --src.feed_cfg.feed_name "github-feed" \
  --src.feed_cfg.auth_cfg.auth "token" \
  --src.feed_cfg.auth_cfg.personal_access_token $GH_TOKEN \
  --src.feed_cfg.org_repo_list '["faros-ai/airbyte-connectors"]'  \
  --src.feed_cfg.repos_query_mode.query_mode "FarosGraph" \
  --dst farosai/airbyte-faros-destination:latest \
  --dst.edition_configs.api_key $FAROS_API_KEY \
  --dst.edition_configs.api_url "https://prod.api.faros.ai" \
  --dst.edition_configs.graph "ypc-test" \
  --dst.edition_configs.graphql_api "v2" \
  --debug

Screenshot 2023-09-22 at 10 13 38 AM Screenshot 2023-09-22 at 10 14 24 AM

While working on this, I upgraded my jq from 1.6 to 1.7 and found out that split behaves differently and this broke the writeConfig function. In writeConfig, we create a long string of the key/value pairs of config items passed via the CLI args, separated by a special separator. We then have a jq expression to split by the separator and merge all the key/value pairs to create the json config. This logic assumes that the number of elements after split is an even number. But sadly this is no longer true in version 1.7: Screenshot 2023-09-22 at 10 23 04 AM

To make this work in 1.7, we just trim the last occurrence of the separator.

Screenshot 2023-09-22 at 10 25 03 AM

Type of change

Related issues

Fix [#1]()

Migration notes

Describe migration notes if any

Extra info

Add any additional information

sonarcloud[bot] commented 1 year ago

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
No Duplication information No Duplication information