mlos_bench: Trial repetition

microsoft / MLOS

MLOS is a project to enable autotuning for systems.

https://microsoft.github.io/MLOS

MIT License

139 stars 66 forks source link

mlos_bench: Trial repetition #756

Open yshady opened 4 months ago

yshady commented 4 months ago

I would like to run each config 30 times but not 30 at same time. For example, a default config should be running 30 times but not concurrently.

Currently trial repeat param for mlos_bench will run the trial repeat number all at the same ts_start

bpkroth commented 4 months ago

Chatted about this, there is some confusion over some storage entries, but they aren't running concurrently (yet).

We should

[ ] add ts_create to the DB to note when a trial was "created" vs when its (scheduled to) start(ed)
[ ] add support for specifying "(random) splay between" trial repeats in the configs
[ ] enable the scheduler to only start trials with current or past ts_start

bpkroth commented 4 months ago

@motus I don't think all of this is handled yet, so reopening.

yshady commented 3 months ago

@eujing can you see if trial repeat with master is still broken? I observed it acts like a minimum bound and not as a maximum

eujing commented 3 months ago

@yshady trial repeat seems to be working at an exact value when using the main branch. I have been using 1, and getting new configs proposed every time. Maybe this is not an issue anymore in the main branch.

yshady commented 3 months ago

I was using 5 or more

bpkroth commented 3 months ago

Should probably just write a test for this.