heidmic / suprb

GNU General Public License v3.0
6 stars 3 forks source link

Add further mixing variants (all weighted sum mixing based on constant weights) #136

Open heidmic opened 1 year ago

heidmic commented 1 year ago

We want to add more variants to https://github.com/heidmic/suprb/blob/master/suprb/solution/mixing_model.py

Current mixing is making a model prediction based on all matching classifier's predictions times their inverse errors divided by their experiences (weighted sum).

What we want to test:

  1. Add a hyperparameter that governs that only x rules can partake in mixing
    • [x] the best x rules (highest fitness)
    • [x] x random rules
    • [x] x rules randomly chosen using fitness-weighted roulette wheel selection
  2. Cap the experience factor to some limit
    • [x] rules of experience y times dimensionality are equivalent in their weight to rules of experience > z times dimensionality (assuming identical errors)
    • [x] try without accounting for dimensionality
    • [ ] are rules with experience smaller than z times dimensionality reliable at all? (e.g. z==0.5*d or z==5 assuming no relations to dim)
  3. Combine these two
  4. Add an option to weigh experience against error with a hyperparameter
heidmic commented 1 year ago

There is one point left open are rules with experience smaller than z times dimensionality reliable at all? (e.g. z==0.5*d or z==5 assuming no relations to dim). This basically asks whether we need a lower bound as well. Maybe something that decreases based on a function (probably nonlinearly). But it might also be detrimental as it strongly discourages specific rules. To implement this in a good way, we would probably need to do something like estimating the "relevant" experience based on some kind of neighbourhood estimation