This PR introduces a new, hierarchical configuration system for Arroyo, replacing the ad-hoc environment variable system currently in place. This will make it easier to configure the system, understand the available configuration options, and add new ones.
The configuration file can be specified by passing --config to the binary, or by placing the config file in $(user-config-dir)/arroyo/config.toml (for example, on Linux this is ~/.config/arroyo/config.toml, on MacOS ~/Library/Application\ Support/arroyo).
Configurations can be overridden with environment variables with the prefix ARROYO__. To convert a toml config key to env var, paths (.) are replace with double-underscores (__) while hyphens are replaced with single underscores (_). So for example, the key controller.rpc-port would be expressed as ARROYO__CONTROLLER__RPC_PORT.
Most existing env var configs are also supported (like CHECKPOINT_URL or DATABASE_NAME) but will produce a warning if used, and support will be removed in 0.12. The helm config format is unchanged.
This PR introduces a new, hierarchical configuration system for Arroyo, replacing the ad-hoc environment variable system currently in place. This will make it easier to configure the system, understand the available configuration options, and add new ones.
An example config file looks like this:
The configuration file can be specified by passing
--config
to the binary, or by placing the config file in $(user-config-dir)/arroyo/config.toml (for example, on Linux this is~/.config/arroyo/config.toml
, on MacOS~/Library/Application\ Support/arroyo
).Configurations can be overridden with environment variables with the prefix
ARROYO__
. To convert a toml config key to env var, paths (.
) are replace with double-underscores (__
) while hyphens are replaced with single underscores (_
). So for example, the keycontroller.rpc-port
would be expressed asARROYO__CONTROLLER__RPC_PORT
.Most existing env var configs are also supported (like
CHECKPOINT_URL
orDATABASE_NAME
) but will produce a warning if used, and support will be removed in 0.12. The helm config format is unchanged.