[tune] utility function for converting dicts to hparam search spaces

[sumanthratna]

Describe your feature request

I'm writing a framework that allows users to tune hparams, where the users would provide a config file describing the search space. A function for converting Python dicts to the Ray hparam config would be a very useful addition to the 1.0 API.

the function should be:

• file format agnostic
  • by accepting a dict instead of file path, users can provide a JSON, YAML, etc. and leave it up to developers using Ray to load the file path to a Python dict
• as flexible as possible (this is a given)
  • e.g., it should work with nested configs (it's pretty easy to convert search spaces with a depth of 1)

Pros:

• establishes a convention for user-specified search spaces
  • WandB puts all hparams under parameters, whereas my current setup splits hparams into 3 categories: preprocessing, algorithm, model
• integrates well with the 1.0 API
  • the 1.0 API is search alg- and scheduler-agnostic, so a search alg- and scheduler-agnostic config file makes sense
• could allow for useful domains and samplers (bonus conversions)
  • e.g., sampling multiple values from a categorical (slack thread)

    Cons:

• more code to maintain
  • I don't expect the sloc for this to be that big (compared to Ray's codebase)
• might be considered an unnecessary addition; instead, leave it to developers using Ray to convert configs to the correct format

---

CC @richardliaw

[sumanthratna]

e.g., if the "bonus" conversions are implemented,

hparams:
  losses:
    type: categorical
    sample_size: rand  # choose the sample size randomly; randrange(1, len(values)+1)
    values:
      - mse
      - l1
      - n11
  dropout:
    type: boolean

would become:

config = {
    "_losses__all": ["mse", "l1", "n11"],
    "_losses__sample_size": tune.sample_from(
        lambda spec: random.randrange(
            1, 
            len(spec.config._losses__all)+1,
        ),
    ),
    "losses": tune.sample_from(
        lambda spec: random.choices(
            spec.config._losses__all,
            spec.config._losses__sample_size,
        ),
    ),
    "dropout": tune.choice([True, False]),
}

[richardliaw]

hmm, sounds interesting - should be much simpler though:

config = {
   "config1": {"tune.uniform": [args]},
   "config2": {"tune.sample_from": "lambda spec: random.choices(...)"}
}

[sumanthratna]

hmm, sounds interesting - should be much simpler though:

config = {
   "config1": {"tune.uniform": [args]},
   "config2": {"tune.sample_from": "lambda spec: random.choices(...)"}
}

@richardliaw i'm not sure what you mean–are you suggesting that the dict you sent should be the input for the conversion function?

[richardliaw]

sorry - as input to the conversion function

[richardliaw]

(if you were to upstream it). Otherwise, perhaps it makes sense to just to build the conversion on your project (with whatever input format makes the most sense for you) for now!

[sumanthratna]

hmm in my case, pathologists with close to no CS experience might be specifying the search space via a config file, so I'd like to avoid syntax like lambda.

for now, I'll keep this in my project codebase; I'm still leaving this open at P3 because I think at some point (post-1.0) it would be cool to implement this in tune

[richardliaw]

very cool! sounds good.

[krfricke]

This should be covered by the Tune search space API conversion: https://docs.ray.io/en/latest/tune/api_docs/search_space.html