Add a configuration file

wojas commented 3 years ago

Output of `rest-server --version`

v0.10 plus local changes

What should rest-server do differently?

Add a structured configuration file to rest-server. This would make it easier to allow more complex configuration, like per-user or repo settings.

What are you trying to do? What is your use case?

Examples of features the would become easier to implement with this feature:

TLS hot reload (#94)
TLS client authentication (#73)
Set --append-only and --max-size per user (#74)
More complicated permission schemes (user 'foo' can also access repo 'bar')
Multiple authentication backends (#70, #111)

This change is backward compatible. We keep supporting existing commandline flags and these override global settings in the configuration file. We may not add flags for all future new features. A new --config flag will specify the location of the configuration file, and will be optional.

Example configs

A config file only needs to contain items with non-default values. An empty configuration file would have the same effect as running rest-server without additional flags.

Simple example with items set to their defaults:

path: .
append_only: false
private_repos: false
listen: ":8000"

More complicated example with non-default settings and per-user overrides:

path: /var/lib/rest-server
append_only: true
private_repos: true
listen: ":8443"

# See https://github.com/PowerDNS/go-tlsconfig
tls:
  cert_file: path/to/cert.pem
  key_file: path/to/key.pem
  watch_certs: true
  watch_certs_poll_interval: 5s

quota:
  max_size: 100GB 

metrics:
  enabled: true
  no_auth: true

auth:
  htpasswd_file: /etc/rest-server/htpasswd

# Per-user overrides
users:
  # Can write to all repos, including overwriting
  admin:
    private_repos: false
    append_only: false

  # You could forgo the htpasswd file altogether, if you want
  user123:
    # htpasswd -nbB ignored-username my-password | cut -d: -f2
    password_hash: "$2y$05$sS5LUnabJeuixKQo7BtIyun7CfCSie239gAxrPWD4RviewfuCAgPa"

PR coming up.

rawtaz commented 3 years ago

Can't we please have a good ol' .ini file instead of a .yaml file? Both are well established formats, but the former is less messy. Also, there's been talk about config file format in the config file issue in the restic repo, but I suppose you know that already :)

dimejo commented 3 years ago

Can't we please have a good ol' .ini file instead of a .yaml file? Both are well established formats, but the former is less messy. Also, there's been talk about config file format in the config file issue in the restic repo, but I suppose you know that already :)

If I recall correctly @fd0 was favoring TOML which is like an extended version of INI.

fd0 commented 3 years ago

Oh, no, TOML is too unusual. My favorite would be HCL, but as much as I despise YAML it's probably the most widely used configuration file format.

wojas commented 3 years ago

I was naively hoping that I would not need to type all of this. Oh well.. 😄

There are many different config file formats and one can spend hours discussing the merits of each of them, as with programming languages, editors and bike shed colors.

YAML nowadays appears to be the most common format for configuration files, with projects like Docker Compose, Kubernetes, Ansible, various CI systems, OpenAPI/Swagger and more using it. I have used it extensively in other projects and it proved pleasant both from a development and from a configuration point of view.

It is not without its warts, but most of that can be managed by providing an example configuration with extensive comments and examples. Most users will never really have to understand YAML, they can just uncomment and change the examples. If they do need to know more, YAML documentation and tutorials are readily available on the internet.

From my experience, gopkg.in/yaml.v2 is also mature and generates pretty good error messages if you get something wrong.

As a bonus, YAML is a superset of JSON, which allows users to write a config like this:

{
  "path": ".",
  "append_only": false,
  "private_repos": false,
  "listen": ":8000"
}

or as less strict JSON, with comments:

{
  "path": ".",              # comments are very useful in config languages
  "append_only": false,
  "private_repos": false,
  "listen": ":8000",        # note the trailing comma here
}

INI files only look simple, they can get very complex or extremely limiting when you need structured config items, like lists or objects. There are also many dialects with subtly different parsing rules.

TOML is a variant that tries to somehow combine YAML with an INI file syntax, but I personally found it non-obvious for more complicated things when I read the documentation some time ago. It is also less common.

HCL is pretty cool if you need a formal configuration language for more complicated systems. It is however less common, and the people that do know it (Terraform users) are almost certainly also familiar with YAML. Choosing HCL would require extensively documenting the format of the configuration language with many usage examples, whereas with YAML I expect that a good example config and a reference to YAML documentation will get users quite far.

restic / rest-server