Open mirnawong1 opened 11 months ago
This would be easiest to maintain if there were some process that autogenerates the page(s) with the flags + sub commands.
Here is a script that uses click
to determine which CLI flags (and environment variables) are relevant to each sub-command.
@aranke and I discussed this issue today while exploring options for https://github.com/dbt-labs/dbt-core/issues/9575.
Here's a pairing of scripts that produces that markdown table:
python generate_cli_flags_artifact.py
python generate_markdown.py > pivot_table.md
generate_cli_flags_artifact.py
is from here.
generate_markdown.py
import pandas as pd
import json
def list_to_pd(result_list):
return pd.DataFrame(result_list)
def extract_commands_and_params(filename):
with open(filename, "r") as file:
data = json.load(file)
commands_list = []
parent_command = "dbt"
if data.get("command") and data["command"].get("commands"):
add_subcommands(data["command"]["commands"], parent_command, commands_list)
return commands_list
def add_subcommands(commands, parent_command="dbt", commands_list=[]):
# base case
if not commands:
return
for command_name, command_info in commands.items():
full_command_name = f"{parent_command} {command_name}"
if command_info["params"]:
for param in command_info["params"]:
flag_name = f"{param['name']}"
commands_list.append({"subcommand": full_command_name, "option": flag_name})
else:
# If there are no params, still add the command with an empty option
commands_list.append({"subcommand": full_command_name, "option": ""})
# recursively explore any sub-commands
if command_info.get("commands"):
add_subcommands(command_info["commands"], full_command_name, commands_list)
def main():
# Assuming the JSON data is stored in a file named dbt-core-cli-flags.json
filename = "dbt-core-cli-flags.json"
click_options = extract_commands_and_params(filename)
result_df = list_to_pd(click_options)
# Represents the presence of a pair with the value 1
result_df["count"] = 1
# Create a pivot table
table = pd.pivot_table(
result_df,
values="count",
index=["option"],
columns=["subcommand"],
aggfunc="sum",
fill_value=0,
)
# Convert 1's and 0's to applicable icons
table.replace(to_replace=1, value="✅", inplace=True)
table.replace(to_replace=0, value="", inplace=True)
# Export to Markdown
print(table.to_markdown(stralign="center"))
if __name__ == "__main__":
main()
https://github.com/dbt-labs/docs.getdbt.com/pull/4814 resolved https://github.com/dbt-labs/docs.getdbt.com/issues/2894 and added the big table of available flags (global configs).
But there are still non-global CLI flag / environment variable combos that aren't documented. One example is --empty
/ DBT_EMPTY
which is only available for compile
, run
, and build
.
Contributions
What page(s) or areas on docs.getdbt.com are affected?
We don’t have a clear docs page/section outlining the available flags and which dbt command they can use.
this seems to be a pretty important gap not covered in the docs.
What changes are you suggesting?
Having a table that lists all of the flags and sub commands, what commands those flags are compatible with, and which editing tool supports those flags.
Additional information
https://docs.getdbt.com/reference/global-configs/failing-fast