feat: support matrices and stdout for all output types

[trxcllnt]

• Support stdout for all output types via the new --stdout flag
• Allow passing --file_key, --output, or --matrix (or any combination of the three)
• Prepends the comment to the header for all file types
• Support multiple --file_key and --output arguments

Fixes https://github.com/rapidsai/dependency-file-generator/issues/46

Enables the dependencies.yaml changes in this branch.

Example generating requirements.txt:

$ rapids-dependency-file-generator --output requirements -f py_build_cudf -f py_run_cudf --matrix "cuda=11.8;arch=$(uname -m)" --stdout
# This file is generated by `rapids-dependency-file-generator`.
# To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.
cmake>=3.23.1,!=3.25.0
cython>=0.29,<0.30
git+https://github.com/python-streamz/streamz.git@master
ninja
numpy>=1.21,<1.24
protoc-wheel
pyarrow==11.0.0.*
rmm-cu11==23.6.*
scikit-build>=0.13.1,<0.17.2
# This file is generated by `rapids-dependency-file-generator`.
# To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.
--extra-index-url=https://pypi.nvidia.com
cachetools
cubinlinker-cu11
cuda-python>=11.7.1,<12.0
cupy-cuda11x>=12.0.0
fsspec>=0.6.0
numba>=0.56.4,<0.57
numpy>=1.21,<1.24
nvtx>=0.2.1
packaging
pandas>=1.3,<1.6.0dev0
protobuf>=4.21.6,<4.22
ptxcompiler-cu11
pyarrow==11.*
rmm-cu11==23.6.*
typing_extensions

Example generating pyproject.toml

$ rapids-dependency-file-generator --output pyproject -f py_build_cudf -f py_run_cudf --matrix "cuda=11.8;arch=$(uname -m)" --stdout
# This file is generated by `rapids-dependency-file-generator`.
# To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.
# Copyright (c) 2021-2023, NVIDIA CORPORATION.

[build-system]
build-backend = "setuptools.build_meta"
requires = [
    "cmake>=3.23.1,!=3.25.0",
    "cython>=0.29,<0.30",
    "ninja",
    "numpy>=1.21,<1.24",
    "protoc-wheel",
    "pyarrow==11.0.0.*",
    "rmm-cu11==23.6.*",
    "scikit-build>=0.13.1,<0.17.2",
    "setuptools",
    "wheel",
]

[project]
name = "cudf-cu11"
version = "23.6.0"
description = "cuDF - GPU Dataframe"
readme = { file = "README.md", content-type = "text/markdown" }
authors = [
    { name = "NVIDIA Corporation" },
]
license = { text = "Apache 2.0" }
requires-python = ">=3.9"
dependencies = [
    "cachetools",
    "cubinlinker-cu11",
    "cuda-python>=11.7.1,<12.0",
    "cupy-cuda11x>=12.0.0",
    "fsspec>=0.6.0",
    "numba>=0.56.4,<0.57",
    "numpy>=1.21,<1.24",
    "nvtx>=0.2.1",
    "packaging",
    "pandas>=1.3,<1.6.0dev0",
    "protobuf>=4.21.6,<4.22",
    "ptxcompiler-cu11",
    "pyarrow==11.*",
    "rmm-cu11==23.6.*",
    "typing_extensions",
]
classifiers = [
    "Intended Audience :: Developers",
    "Topic :: Database",
    "Topic :: Scientific/Engineering",
    "License :: OSI Approved :: Apache Software License",
    "Programming Language :: Python",
    "Programming Language :: Python :: 3.9",
    "Programming Language :: Python :: 3.10",
]

[project.optional-dependencies]
test = [
    "fastavro>=0.22.9",
    "hypothesis",
    "mimesis>=4.1.0",
    "msgpack",
    "pyorc",
    "pytest",
    "pytest-benchmark",
    "pytest-cases",
    "pytest-cov",
    "pytest-xdist",
    "python-snappy>=0.6.0",
    "scipy",
    "tokenizers==0.13.1",
    "transformers==4.24.0",
    "tzdata",
]

[project.urls]
Homepage = "https://github.com/rapidsai/cudf"
Documentation = "https://docs.rapids.ai/api/cudf/stable/"

[tool.setuptools]
license-files = ["LICENSE"]

[tool.isort]
line_length = 79
multi_line_output = 3
include_trailing_comma = true
force_grid_wrap = 0
combine_as_imports = true
order_by_type = true
known_dask = [
    "dask",
    "distributed",
    "dask_cuda",
]
known_rapids = [
    "rmm",
]
known_first_party = [
    "cudf",
]
default_section = "THIRDPARTY"
sections = [
    "FUTURE",
    "STDLIB",
    "THIRDPARTY",
    "DASK",
    "RAPIDS",
    "FIRSTPARTY",
    "LOCALFOLDER",
]
skip = [
    "thirdparty",
    ".eggs",
    ".git",
    ".hg",
    ".mypy_cache",
    ".tox",
    ".venv",
    "_build",
    "buck-out",
    "build",
    "dist",
    "__init__.py",
]

[ajschmidt8]

I think these changes are considered breaking because of the new stdout flag/behavior.

It will break test scripts like this: https://github.com/rapidsai/cudf/blob/72c067726ccfb6e87033d34ab07b4dc79b5e4a3e/ci/test_python_common.sh#L10-L14

@trxcllnt, can you add a BREAKING CHANGE note to your PR body as mentioned here https://github.com/rapidsai/dependency-file-generator/blob/main/CONTRIBUTING.md? That will make sure the next release increments the major version.

Our CI images pin to the current major version of dfg: https://github.com/rapidsai/ci-imgs/blob/cac1028880574b466ed37a4aec8aaf93d3eab0b2/Dockerfile#L117-L119

But we'll need a way to incrementally roll this out to each repository before we update the version in our CI images so that we don't break CI for everyone.

I thought about this in the past, but never took any action on it due to time constraints.

I think one way we can fix it is to add a new optional input, install_dfg_version, to the relevant shared workflows here: https://github.com/rapidsai/shared-action-workflows/tree/branch-23.06/.github/workflows.

install_dfg_version will be a version specifier for rapids-dependency-file-generator. When it's set, it will install the specified version. When it is not set, it will simply do nothing.

Then each repo can manually opt-in to the new major version. Once all the repos are using the new version, we can update the CI image version accordingly and then go back and clean up all of the optional install_dfg_version arguments in each repository.

[vyasr]

@trxcllnt apologies for the delay in reviewing this PR. I propose that we we split this up so that the uncontroversial pieces can get merged quickly, if you're open to that.

• I think everyone is happy with adding support for stdout to all file types (especially pyproject.toml)
• My guess is that generalizing the CLI to support only passing a subset of arguments is probably uncontroversial in theory, but a bit more problematic because it's a breaking change. I know you said you and @ajschmidt8 had discussed that piece a bit further, and maybe there's a path forward there that involves refactoring the Python functionality and then using it in a new CLI? That would be another good option.
• The project name/CUDA suffix piece will need some more debate, and I don't want to hold up you getting the other useful pieces in over that.

[vyasr]

@trxcllnt would you like some help finishing this up? Let me know if you want to chat about it or need some extra person-hour help.

[bdice]

@trxcllnt @vyasr Can we get this PR to a completed state? It keeps biting us that we don't have pyproject matrix support. I can help with a review, if that's what is needed, but it seems like there is some code work to be done still.

[vyasr]

I'll defer to @trxcllnt here. He had an idea of how best to rewrite the generator to better support this behavior.

[vyasr]

Replaced by #74