search

icaros-usc / pyribs

A bare-bones Python library for quality diversity optimization.
https://pyribs.org
MIT License
205 stars 31 forks source link

Support diversity optimization in Scheduler.tell #473

Closed btjanaka closed 6 days ago

btjanaka commented 1 week ago

Description

We are working on supporting diversity optimization algorithms such as novelty search (see #472). One difference from other algorithms is that such algorithms do not require objectives. However, the current API to Scheduler.tell() is:

tell(objective, measures, **fields)

i.e., it assumes objective and measures will be provided. This PR seeks to make it possible to call scheduler.tell(measures) in addition to the current scheduler.tell(objective, measures).

Use Cases

There are a couple of use cases that the scheduler must satisfy:

  1. QD optimization (i.e., current behavior should not break):
    • Positional arguments: scheduler.tell(objective, measures)
    • Keyword arguments: scheduler.tell(objective=objective, measures=measures)
    • Mixed (a rather extreme case): scheduler.tell(objective, measures=measures)
  2. Diversity optimization:
    • Positional arguments: scheduler.tell(measures)
    • Keyword arguments: scheduler.tell(measures=measures)
  3. Diversity optimization but where an objective has been added as an extra field:
    • Positional arguments: scheduler.tell(measures, objective=objective)
    • Keyword arguments: scheduler.tell(measures=measures, objective=objective)

Ideally, we would also be robust to a case where we need to only have objectives in the future:

  1. Objective optimization:
    • Positional arguments: scheduler.tell(objective)
    • Keyword arguments: scheduler.tell(objective=objective)

Potential Solutions

  1. Change scheduler API to tell(*args, **kwargs) and add a problem_type to each archive.
    • args and kwargs are interpreted based on the type of the archive. If the problem type is quality_diversity, then the args are interpreted as objectives and measures. If the problem type is diversity_optimization, then the args are interpreted as measures. If the problem type is single_objective, the args are interpreted as objective. kwargs will be treated the same as fields in all cases.
    • Pros: Backwards-compatible and handles all 4 use cases above.
    • Cons: Makes the archives a bit more complex, may be a bit confusing to users because the signature will not be very informative. However, I think we can get away with this with good docstrings and documentation.
  2. Allow the current objective argument to be treated as measures, and assign a default argument of None to the current objective and measures -> i.e., tell(objective=None, measures=None, **fields)
    • Inspired by how numpy's integers can be either rng.integers(high) or rng.integers(low, high) -> see here
    • Pros: Minimal changes to current API, and still quite interpretable for users.
    • Cons: On the other hand, it may be confusing to see that objective can be set to measures. This will also fail case 3 above, i.e., scheduler.tell(measures, objective=objective) will throw an error because measures is actually objective due to the positional argument, so objective is being passed in twice.
  3. Require the user to pass in objective=None when performing diversity optimization and maintain the same scheduler.tell API.
    • Pros: This requires the fewest changes to the API. It also provides a good model for what EvolutionStrategyEmitter and other emitters should do -- all these classes can now implement behavior for objective=None. It also does not require modifying the archives to have a problem_type or archive_type attribute. Furthermore, objective can still be provided if that is a field in the archive.
    • Cons: It is a bit verbose to have scheduler.tell(None, measures) or scheduler.tell(objective=None, measures=measures) but I think users can understand that.
    • Optionally, we can also set a default of objective=None and measures=None so that one can pass just measures, e.g., scheduler.tell(measures=...) or just objectives scheduler.tell(objective=). However, I think we may not want to add this feature for now as it requires adding default values to a lot of places.
  4. Add a new parameter to scheduler.tell called mode or something similar to indicate diversity optimization is in effect, i.e., scheduler.tell(measures, mode="diversity")
    • Pros: This is very explicit and makes it clear that diversity optimization is happening without objectives.
    • Cons: Rather verbose.

Decision

I believe Solution 3 is the best solution, since it involves the fewest changes to the API and also contains the changes to the scheduler.

TODO

Status