automl / neps

Neural Pipeline Search (NePS): Helps deep learning experts find the best neural pipeline.
https://automl.github.io/neps/
Apache License 2.0
39 stars 11 forks source link

refactor(SearchSpace): Switch to `clone()` for search space. #94

Closed eddiebergman closed 2 months ago

eddiebergman commented 2 months ago

This PR attempts to clean up and document the SearchSpace setup as well as a few other minor optimizations. This is to make progress towards an 1) ask-and-tell interface for benchmarking algorithmic changes/decisions easily and fast, 2) importantly fast, the latest set of changes tried to be self contained to their modules but have sped up things from 6.2seconds after improvements from for the basic example to 1.2seconds and now to 0.7seconds (excluding import time).

All of this is not super important when considering the time to train models but this is unfortunately the difference of being able to quickly ablate and benchmark changes required for both CI and developing new research methods.


The most important contributions of this PR is to avoid the use of deepcopy on SearchSpace objects and instead resort to using clone() where possible. This simply just calls the constructor with the arugments and sets the value if any.

  1. Object construction tends to be a little faster than deepcopying
  2. If migrating away from the joint use of the SearchSpace as both a space definition and config, then now it's detectable as to where these clones occur.

Here the changes this PR makes to a simple sampling in the script below which does a simple generate 1000 configs, 100 times.

Mean time: 0.1886 seconds -> 0.1258
Std time: 0.0378 seconds -> 0.0377
Min time: 0.1547 seconds -> 0.113
Max time: 0.267 seconds -> 0.2273

I imagine switch to a non-copy and dedicated configuration object could drop this down to 0.05. For reference, some recent optimization work for ConfigSpace which removed cython based things has it at around 0.3 seconds.

This is ignoring the fact most optimizers/acq. functions then have to extract all the information and put them in some array structure which is not show in the figure above, and then de-allocate all the clones.


Along with that, there's now documentation and typing so it's been re-enabled in ruff and mypy.


I don't really know what's going on with the graphs under the hood but after some debugging and investigations, these don't share a lot of similarities with a Parameter and it's more of a overlap of behaviour then really belonging to the heirarchy.

Future PR's from this insight might consider making them seperate classes with no shared implementation such that consuming code can knowingly decide what to do on detection of a GraphParameter, e.g. you may not immediatly know you can't mutate some of them or that it can be vectorized to a number/set of numbers, which would be important to know for a PFN-surrogate for example.


from __future__ import annotations

import time

import numpy as np

# ruff: noqa: T201
import neps
from neps.search_spaces.search_space import SearchSpace

def main():
    ss = SearchSpace(
        a=neps.IntegerParameter(0, 10),
        b=neps.IntegerParameter(1, 1000, log=True),
        c=neps.FloatParameter(0, 10, log=False),
        d=neps.FloatParameter(1, 1000, log=True),
        e=neps.CategoricalParameter(["a", "b", "c"]),
    )

    times = []
    for _ in range(100):
        start = time.time()
        _samples = [ss.sample() for _ in range(1000)]
        times.append(time.time() - start)

    print(f"Mean time: {sum(times) / len(times):.4f} seconds")
    print(f"Std time: {np.std(times):.4f} seconds")
    print(f"Min time: {min(times):.4f} seconds")
    print(f"Max time: {max(times):.4f} seconds")

if __name__ == "__main__":
    main()