Algo Budget Information

[Delaunay]

I think all algos should have some form of budget information

• Number of unique configuration that will be sampled at the end of the optimization
• Total number of epochs
• Number of unique configuration per stage if multiple stage
• Parallelism: (weighted average of the possible number of parallel trials running per stages)

Why

• It gives useful comparison points (i.e RS is unmatched for parallelism, but hyperband allows you to do a lot less epochs)
• Allows users to know specifics about the algo themselves (i.e One may need to track more information that what Orion is doing and knowing how many trials will be created & how many epochs for each can greatly help)

Example:

Hyperband

• Unique config: 300
• Total number of epochs: 1254 (2 (300 - 12 - 3) + 12 (30 - 3) + 120 * 3 `
• Stages: [{'epoch': 2, 'count': 300}, {'epoch': 12, 'count': 30}, {'epoch': 120, 'count': 3}]
• Parallelism: 11

Random search

• Unique config: 300
• Total Number or epochs: 36000
• Stages [{'epoch': 120, 'count': 300}]
• Parallelism: 300

BayeOpt (Sequential)

Sample a new config every 2 epochs

• Unique Config: 60
• Total number of epochs: 120
• Stages: [{'epoch': i * 2, 'count': 1} for i in range(60) ]
• Parallelism: 1

[bouthilx]

• Number of unique configuration that will be sampled at the end of the optimization

That's like experiment.max_trials but pushed into algo?

• Total number of epochs

This should be optional, it opens the black box and some problems may not have epochs.

• Number of unique configuration per stage if multiple stage

Do you see the stages as a way to represent the number of trials that will be sampled at a given time? That could be a handy way of representing this information indeed. Note that many algorithms will have dynamic stages. ASHA for instance may not follow the expected stages based on the results, how fast or how slow new best trials are found. Same for BayesOpt with parallel strategies for parallelism.

• Parallelism: (weighted average of the possible number of parallel trials running per stages)

Can you elaborate a bit more on this one? How do you compute it and what would it be used for?

[Delaunay]

yeah not all algos will provide the same information but I think a lot could be similar. for ASHA it would provide the upper bound

Parallelism the the number of independent trial that could be run in parallel averaged out per stages.

So random search is just all the configs. I made an error for hyperband the parallelism would be 11.

Epoch	Count	Weighted	formula
2	300	600	epoch * count
12	30	360
120	3	360
	Parallelism	11	(600 + 360 + 360) / 120

It is just a metric to highlight the property of a given optimizer

[guillaume-chevalier]

Related tasks for sparking discussion:

• https://github.com/Neuraxio/Neuraxle/issues/302 (I'd like to bring attention on especially this one)
• https://github.com/Neuraxio/Neuraxle/issues/299
• https://github.com/Neuraxio/Neuraxle/issues/301
• https://github.com/Neuraxio/Neuraxle/issues/258

On our side, we'd like the 4 things above to work together as a whole. 4th one is optional and depends on framework used as a hyperparameter repository.

[Delaunay]

@guillaume-chevalier Have you ever used Orion before ? Hyperband is already implemented by Orion.

[guillaume-chevalier]

@Delaunay I have not yet tried Orion definitely, although I've taken a few looks to its code.

By the way I've discussed with @bouthilx and it'd be cool to have a common hyperparameter library we could both import to have our algo's hyperparams compatible with each other.

Next steps would be to write adapters of hyperparam optimization algorithms to do the bridge in one direction and the other one too. This could be done as well for the persistence.

By the way guys I'd suggest to move your MongoDB things to another plug-in package outside of your main Orion repo to avoid having to absolutely rely on MongoDB even for users who don't use it.