IQVIA-ML / TreeParzen.jl

TreeParzen.jl, a pure Julia hyperparameter optimiser with MLJ integration
Other
35 stars 5 forks source link

Suggestions fundamentally broken #50

Open yalwan-iqvia opened 4 years ago

yalwan-iqvia commented 4 years ago

It seems to me that while the user can supply suggestions to be evaluated by the system, doing so is pointless as of the current state since it cannot use the results for modelling (the evaluations are not associated with any part of the conditional graph)

It would be good to demonstrate this with some worked examples -- e.g. craft a system that samples and has a suggestion with a perfect result and all others with garbage results. Idea is that if doing it via suggestions it won't try to sample more from near suggestions. Then do it again by forging a space and then mucking about with internal values so that it has a proper association with the graph and show that under same circumstances (forged good values at forced suggestions) cause optimiser to want to go near the suggestions region.

Fix is likely to be extremely complex when considering arbitrarily structured spaces ....