rupertford / melody

Lightweight python parameter search software for performance analysis and tuning
BSD 3-Clause "New" or "Revised" License
2 stars 0 forks source link

Complex parameter types #7

Open jimboid opened 7 years ago

jimboid commented 7 years ago

This might have to be something that happens on the domain specific side, but I wonder if there is anything generic we can do in melody core such that this becomes much simpler and more "melody like (melodious??)".

If I have a parameter that I'm interested in sweeping, say parameter "A" which sweeps say between 0.1 and 0.5 but by doing this I have to make one or more parameters "B" and "C" sweep the same range (not permute) so A = B = C. Do we currently have a way to do this?

rupertford commented 7 years ago

No there is no way to do this at the moment from the front end of melody so it would have to be done in a domain specific (back-end) way you suggest.

I've thought about the idea of constraints between inputs but have left that for the timebeing as I couldn't come up with a simple solution.

I was originally thinking that some sort of constraint language that a user could write their constraints in would be required and then melody would have to honour those constraints and that sounds hard to me!

However, if there are particular cases, such as the one above, that occur often, we might be able to support this in a more limited way. In your case we could have Same(A,B) and Same(A,C) options as inputs.

However, in general, constraints could be arbitrary things e.g. B = 2*A + 6, or, if not C and D>2 or D<1 then E; else F.

Perhaps melody could optionally support a constraint callback function which told melody if a particular combination was valid if we want general constraints?

Or perhaps we could support the notion of derived values and get a callback function to compute the derived values for us?

jimboid commented 7 years ago

Yes that looks pretty hard, might be worth keeping this for the future and see if any other projects have a similar need.

I'll try and do this in my launcher script at some point.

rupertford commented 7 years ago

OKl, I'll leave this as a live question in case it becomes more important for you, or someone else expresses a need.

rupertford commented 7 years ago

Following a discussion with James, the last of the options I gave a few posts back i.e. "... support the notion of derived values ..." seems to be a reasonable solution. So, for the example above we might implement something like ...

' from melody.inputs import FloatRange, DerivedEquals from melody.search import BruteForce from melody.main import Melody

def test_function(values):

values provides a value for a, b and c

...
return ...

INPUTS = [ FloatRange(name="A", low=0.1, high=0.5, step=0.05), DerivedEquals(names=["B","C"], derived_from="A")]

MELODY = Melody(inputs=INPUTS, function=test_function, method=BruteForce) MELODY.search() ' where there is a Melody 'Derived' base class and at least one (DerivedEquals) supported subclass. Users can then create their own Derived subclass if they like as long as they conform to the Derived base class API.