[Feature Request] Support relative paths in defaults keys.

[alegonz]

🚀 Feature Request

Hi, thank you for this great tool.

It would be useful if one could use relatives paths with .. in defaults keys.

Although currently this is somewhat possible with some workarounds (see below for details), overriding defaults via the CLI does not work throwing a error like hydra.errors.OverrideParseException: LexerNoViableAltException: ../../db@db=postgresql.

Motivation

Is your feature request related to a problem? Please describe.

By being able to use .. in defaults keys, one could organize primary configs in project subfolders under the root configuration folder (a project may involve multiple primary configs), and one could also reuse configurations that are common to primary configs across projects and configurations that are common to primary configs within a project.

Consider the following repo structure:

• Assume multiple projects are developed under the same repo.
• All the configurations are under conf.
• The primary config of each project is in a corresponding subfolder under conf.
  • For example the "foo" project has its primary configuration file in conf/projects/foo/config.yaml.
  • There may be other such primary configs under conf/projects/foo.
• Configuration files that are common across projects are under conf, next to conf/projects.
  • conf/db in the example below.
• Configuration files that are common within a project are under the projects subfolder.
  • conf/projects/foo/info.yaml in the example below.

.
├── app.py
└── conf
    ├── db
    │   ├── mysql.yaml
    │   └── postgresql.yaml
    └── projects
        └── foo
            ├── config.yaml
            └── info.yaml

Contents of each file:

# conf/db/mysql.yaml
driver: mysql
user: omry
password: secret

# conf/db/postgresql.yaml
driver: postgresql
user: postgres_user
password: drowssap

# conf/projects/foo/config.yaml
# @package _global_
defaults:
  # Within-project common configuration
  - info@info
  # Across-projects common configuration
  # Use @db to avoid nesting with empty strings as keys.
  # An alternative is to use a @package directive in the config file.
  - ../../db@db: mysql
  - _self_

somekey: someval

# conf/projects/foo/info.yaml
project_name: Foo Project

# app.py
#!/usr/bin/env python3
import argparse
from pathlib import Path
from hydra import initialize_config_dir, compose

parser = argparse.ArgumentParser(description="Example app.")
parser.add_argument("--config", type=Path, help="Path for the input data config(.yaml).")
parser.add_argument(
    "overrides",
    nargs=argparse.REMAINDER,
    help="You can pass multiple arguments to set or override configuration parameters "
    "with `key=val` syntax, e.g. 'foo=123 foo.bar=xyz'",
    metavar="...",
)
args = parser.parse_args()

config_dir = Path.cwd() / "conf"
with initialize_config_dir(config_dir=str(config_dir.resolve()), version_base="1.1"):
    cfg = compose(
        config_name=str(args.config.resolve().relative_to(config_dir.resolve())),
        overrides=args.overrides
    )
print(cfg)

With this it is possible to do this which yields the expected, desired result:

~$ ./app.py --config conf/projects/foo/config.yaml
{'info': {'project_name': 'Foo Project'}, 'db': {'driver': 'mysql', 'user': 'omry', 'password': 'secret'}, 'somekey': 'someval'}

However, overrides don't work:

(.venv) ./app.py --config conf/projects/foo/config.yaml ../../db@db=postgresql
Traceback (most recent call last):
  File "/**redacted**/./app.py", line 19, in <module>
    cfg = compose(
  File "/**redacted**/.venv/lib/python3.10/site-packages/hydra/compose.py", line 38, in compose
    cfg = gh.hydra.compose_config(
  File "/**redacted**/.venv/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 594, in compose_config
    cfg = self.config_loader.load_configuration(
  File "/**redacted**/.venv/lib/python3.10/site-packages/hydra/_internal/config_loader_impl.py", line 142, in load_configuration
    return self._load_configuration_impl(
  File "/**redacted**/.venv/lib/python3.10/site-packages/hydra/_internal/config_loader_impl.py", line 244, in _load_configuration_impl
    parsed_overrides, caching_repo = self._parse_overrides_and_create_caching_repo(
  File "/**redacted**/.venv/lib/python3.10/site-packages/hydra/_internal/config_loader_impl.py", line 228, in _parse_overrides_and_create_caching_repo
    parsed_overrides = parser.parse_overrides(overrides=overrides)
  File "/**redacted**/.venv/lib/python3.10/site-packages/hydra/core/override_parser/overrides_parser.py", line 96, in parse_overrides
    raise OverrideParseException(
hydra.errors.OverrideParseException: LexerNoViableAltException: ../../db@db=postgresql
                           ^
See https://hydra.cc/docs/1.2/advanced/override_grammar/basic for details

The workarounds employed to make work relative paths in defaults are:

1. Add # @package _global_ at the top of the primary config. This is to avoid nesting the configuration in a key hierarchy that follows the folder hierarchy of the primary config path. That is, if we pass "conf/projects/foo/config.yaml" to hydra's compose, without the directive the entire config would be put under a "projects.foo" key, but with the directive it will be put directly at the root of the dict.

2. Specify a package with @ for the config group that contains ... This is to avoid the content of the config being placed under a hierarchy of keys made of empty strings (this relates to #2878) . An alternative is to use a @package directive in the config file.

Pitch

Describe the solution you'd like

It would be useful if one could do use relative paths in defaults without the workarounds (which are not obvious and I discovered after some trial-and-error) and if it could support overrding them via the CLI.

Describe alternatives you've considered

I can't think of other alternatives to achieve a similar same repo structure without needing relative paths.

Are you willing to open a pull request? (See CONTRIBUTING)

Yes, I'd be happy to contribute. However, I would need some guidance since I'm not familiar with the implementation details.

Additional context

Add any other context or screenshots about the feature request here.

[alegonz]

I edited the description with some more information I had forgotten to add. Please refer to the latest version of the description.