search

meltano / sdk

Write 70% less code by using the SDK to build custom extractors and loaders that adhere to the Singer standard: https://sdk.meltano.com
https://sdk.meltano.com
Apache License 2.0
94 stars 68 forks source link

Use an `--env-config` flag instead of `--config=ENV` to tell the tap to parse environment variables #199

Open MeltyBot opened 3 years ago

MeltyBot commented 3 years ago

Migrated from GitLab: https://gitlab.com/meltano/sdk/-/issues/200

Originally created by @edgarrmondragon on 2021-08-24 19:04:04

Summary

Currently, the way to tell a tap-based SDK to parse environment variables for settings is to pass a --config=ENV option to the tap invocation

my-tap --config=firstConfig.json --config=ENV -config=secondConfig.json

I suggest supporting a boolean --env-config instead.

Proposed benefits

Be more explicit about environment parsing option for settings and don't rely on the hardcoded ENV string.

Also, the position of --config=ENV in the multi-value sequence is not important, so in practice it acts only as a flag. It'd be good to make that understanding more explicit.

Proposal details

Add a new --env-config option to the tap CLI decorators:

click.option(
    "--env-config",
    is_flag=True,
    help="Use environment variables for configuration.",
)

The corresponding env_config boolean variable in the function would then be passed directly to the Tap constructor's parse_env_config argument.

Best reasons not to build

--env-config is not standard for Singer taps, so this would act as an extension to the common CLI options.

cc @aaronsteers

MeltyBot commented 2 years ago

View 6 previous comments from the original issue on GitLab

stale[bot] commented 1 year ago

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

edgarrmondragon commented 1 year ago

Still relevant

stale[bot] commented 2 months ago

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

edgarrmondragon commented 2 months ago

Still relevant

edgarrmondragon commented 2 months ago

Added this to the Office Hours board. I'm interested in polling folks:

  1. Do they ever use multiple --config inputs?
  2. Do they frequently use --config=ENV?
  3. Would a flag like --read-env be useful (this issue)?
  4. Would short flags be (e.g. -d for --discover) be appreciated?