s1dlx / meh

Merging Execution Helper
MIT License
37 stars 10 forks source link

feat: Distribution crossover #14

Closed ljleb closed 1 year ago

ljleb commented 1 year ago

This one is a little bit weird, I am not sure it is really useful. However, the concept is pretty fun:

  1. Reshape and sort models a and b in the order of model c's weights
  2. Take the fourrier transform of both resulting distributions
  3. Merge the frequency components according to a linear filter (see below)
  4. Turn the merged frequencies back into a weights distribution
  5. Reorder and reshape according to c's weights order

The filter used in the spectral domain can be parameterized here: https://www.desmos.com/calculator/wsikh5hmyf Everything in red is the ratio of the frequencies of model b that contribute to the result. x=0 is the lowest frequency component and x=1 is the highest.

When beta = 0, alpha is a hard ratio of the frequency components. a takes all the frequencies below alpha and b takes all of the frequencies above. In this case you could say a contributes the low frequencies and b the high frequencies.