search

google / vizier

Python-based research interface for blackbox and hyperparameter optimization, based on the internal Google Vizier Service.
https://oss-vizier.readthedocs.io
Apache License 2.0
1.48k stars 96 forks source link

Get suggestions based on existing measurements #284

Closed Ark-kun closed 2 years ago

Ark-kun commented 2 years ago

What is the easiest way to get Vizier suggestions based on existing measurements? I believe that somewhere inside Vizier there is a something that would let me do this without much overhead or boilerplate.

xingyousong commented 2 years ago

Hi @Ark-kun, there are a few ways, depending on how far down stack you're interested in.

At the highest level, you'd use the User API to tune your tasks and obtain suggestions.

However, if you're purely interested in the algorithm implementations, this would be the Designer API, with our currently implemented designers found here, with their enumerations in the service here.

Ark-kun commented 2 years ago

At the highest level, you'd use the User API to tune your tasks and obtain suggestions.

It looks like this route involves starting a new server and communicating with it's endpoint. This looks like a bit of overhead.

I've read the documentation about Policies, Designers, Pythia, but I'm still somewhat confused.

Policy does not seem to fit since it's geared towards long-term multi-stage operation. Policy seems to be related to asking for suggestions multiple times while I only need to ask once.

Designer seems a bit closer, but it seems to be just an abstract interface. The documentation shows how I can create my own Designer while I just want to use Vizier's designer. To implement a designer I need to write my own make_parameters function. But it looks like that function is what I'm actually seeking. However the sample make_parameters function in the documentation is very limited and rudimentary. I need a full-fledged one.

I need Vizier's make_parameters function.

Looking at the built-in policies...

xingyousong commented 2 years ago

Ah I see what you mean - currently the Default Vizier algorithm (i.e. specific version of Bayesian Optimization, GP-UCB) is still a work in progress, as we're trying to port the C++ version (that's used internally) into Python + Jax. Please stay tuned, as this will be released in the upcoming weeks!

Ark-kun commented 2 years ago

The currently implemented designers found here, with their enumerations in the service here.

Thank you. It looks like your first comment provided what I wanted. It looks like I should be able to instantiate a designer without any additional set-up, then update it with known information and get new suggestions.

currently the Default Vizier algorithm (i.e. specific version of Bayesian Optimization, GP-UCB) is still a work in progress, as we're trying to port the C++ version (that's used internally) into Python + Jax. Please stay tuned, as this will be released in the upcoming weeks!

Great news. I'll keep tuned.

BTW, here is the way I'm using Vizier. Couple of years ago I've build Kubeflow Pipelines components and pipelines for Google Cloud AI Platform Vizier/Optimizer. Recently I was updating it to Vertex version of Vizier and found this Vizier OSS project. Having a portable stand-alone implementation is great. Thank you a lot for this project!

xingyousong commented 2 years ago

BTW, here is the way I'm using Vizier. Couple of years ago I've build Kubeflow Pipelines components and pipelines for Google Cloud AI Platform Vizier/Optimizer. Recently I was updating it to Vertex version of Vizier and found this Vizier OSS project.

Good to know! I'm not sure if it helps, but if your pipeline uses the OSS Platform Client, then it should be very easy to switch back-and-forth between OSS and Vertex, as they all follow the same Cross-Platform abstraction.