panahiparham / ml-experiment-definition

MIT License
2 stars 3 forks source link

Allow ability to extend a sweep with a prior default value #5

Closed andnp closed 2 months ago

andnp commented 2 months ago

Consider the QRC algorithm implementation where we assume beta=1.0 by default. We execute an experiment like:

from ml_experiment.DefinitionPart import DefinitionPart

part = DefinitionPart(name='qrc')
part.add_sweepable_property('alpha', (2**-i for i in range(5, 9)))
part.commit()

then later we decide to sweep beta:

from ml_experiment.DefinitionPart import DefinitionPart

part = DefinitionPart(name='qrc')
part.add_sweepable_property('alpha', (2**-i for i in range(5, 9)))
part.add_sweepable_property('beta', [0.5, 1.0, 2.0])
part.commit()

A problem occurs where every result is now considered invalidated, despite the fact that we already have results for the beta=1.0 case from the first experiment execution.


This PR addresses exactly this issue. The PR is structured in a sequence of 3 commits, it might be useful to go commit-by-commit (i.e. in order to see where the Maybe monad fits in).

You can test this PR with the following (run once to generate the v0 table, uncomment and generate the v1):

from ml_experiment.DefinitionPart import DefinitionPart

part = DefinitionPart(name='qrc')
part.add_sweepable_property('alpha', (2**-i for i in range(5, 9)))
# part.add_sweepable_property(
#   'beta', [0.5, 1.0, 2.0],
#   assume_prior_value=1.0,
# )
part.commit()

We expect the beta=1.0 rows to have ids 0, 1, 2, 3 and the full range of the v1 table to be [0, 11]. The original implementation would have ids [4, 15].