search

hunar4321 / RLS-neural-net

Recursive Leasting Squares (RLS) with Neural Network for fast learning
MIT License
48 stars 9 forks source link

No issue just a finding that i had... #1

Open snapo opened 10 months ago

snapo commented 10 months ago

First of all a lot of thanks for publishing your code. In your video you mention it is unknown how many layers / neurons should be sufficient. So i run some very basic tests on RLS. From my finding i estimate to get 100% accuracy you would require 1.1x the neurons of all possible outcomes.

To do the test i chose Majority bit as a test. where you have 3-12 input bits and the more bits are 1 the output is one, if more bits are 0 on the input , then the output should be 0 too. I run it for different neuron counts and different majority gates sizes all in a one-shot (so training data only seen once) like mentioned in the video and the github description.

Here is the adjusted code i used:

import numpy as np
import itertools

class RlsNode:
    """
    Recursive Least Squares Estimation (This is the update part of Kalman filter)
    """
    def __init__(self, _m, _name):
        self.name = _name;
        self.M = _m;
        self.w = np.random.rand(1, _m);
        self.P = np.eye(_m)/1;

    def RlsPredict(self, v):
        return np.dot(self.w , v);

    def RlsLearn(self, v, e):
        pv = np.dot(self.P, v)
        vv = np.dot(pv.T, v)        
        eMod = pv/vv;
        ee = eMod * e;        
        self.w = self.w + ee
        outer = np.outer(eMod , pv);
        self.P = self.P - outer

class Net:
    """ 
    neural network (single hidden layer where the weights in first layer (iWh) are randomly initialized)
    """
    def __init__(self, _input_size, _neurons):
        self.input_size = _input_size;
        self.neurons = _neurons;
        self.iWh = (np.random.rand(_neurons, _input_size)-0.5)*1
        self.nodes = [];

    def CreateOutputNode(self, _name):
        nn = RlsNode(self.neurons, _name);
        self.nodes.append(nn)

    def sigmoid(self, x):
        return 1 / (1 + np.e ** -x)

    def FeedForwardL1(self, v):
        vout = np.dot(self.iWh, v);
        # tout = np.tanh(vout);
        tout = self.sigmoid(vout);
        return tout + 0.0000000000000001 ## adding a small value to avoid dividion by zero in the upcoming computations!

    ### RLS layer (Trainable weights using RLS algorthim)
    def FeedForwardL2(self, tout):
        yhats = [];
        for i in range(len(self.nodes)):
            p = self.nodes[i].RlsPredict(tout);
            yhats.append(p[0]);
        return np.asarray(yhats)

    ### Error Evaluation
    def Evaluate(self, ys, yhats):
        errs = ys - yhats
        return errs

    def Learn(self, acts, errs):
        for i in range(len(self.nodes)):
            self.nodes[i].RlsLearn(acts, errs[i]) #change to errs[0][i] if indexing error happen

for l in [80,128,156,225,256,285,450,512,560,900,1024,1124,1800,2048,2248,3600,4096,4400]:
    print(f" --------------------- Neurons {l} ---------------------")

    for r in range(3,13):
        # #### Example Usage ###            
        lengthmaj = r # Create list and results of the majority gates with 11 bits
        x = [list(input_bits) for input_bits in itertools.product([0, 1], repeat=lengthmaj)]
        y = [int(sum(input_bits) >= lengthmaj/2) for input_bits in x]

        ## configuring the network
        n_input = lengthmaj
        n_neurons = l
        n_output = 1
        net = Net(n_input, n_neurons)
        for i in range(n_output):
            net.CreateOutputNode(i)

        ## training
        N = len(x) ## you only need one iteration over the samples to learn (RLS is a near one-shot learning!)
        #for x in range(0,1): # train 5 epochs
        for i in range(N):
            inputt = x[i][:]
            L1 = net.FeedForwardL1(inputt);
            yhats = net.FeedForwardL2(L1)
            errs = net.Evaluate(y[i], yhats)
            net.Learn(L1, errs)

        ## evaluate after learning
        yh= [];
        countright = 0
        for i in range(N):
            inputt = x[i][:] 
            L1 = net.FeedForwardL1(inputt);
            yhats = net.FeedForwardL2(L1)
            prediction = yhats[0]
            if prediction > 0.5 and y[i] == 1:
                countright += 1
            if prediction < 0.5 and y[i] == 0:
                countright += 1

            #print(f"input {inputt} predicted output: {abs(round(yhats[0], 2))} | truth output {y[i]} | Counter : {countright}")

        print(f"Majority Gate width: {r} -- Lenght targets: {len(y)} -- Total correct Counter : {countright} -- Neurons used : {n_neurons} -- Percentage correct : {round(countright / len(y) * 100, 2)}%")

Here is the output with the accuracys:

 --------------------- Neurons 80 ---------------------
Majority Gate width: 3 -- Lenght targets: 8 -- Total correct Counter : 8 -- Neurons used : 80 -- Percentage correct : 100.0%
Majority Gate width: 4 -- Lenght targets: 16 -- Total correct Counter : 16 -- Neurons used : 80 -- Percentage correct : 100.0%
Majority Gate width: 5 -- Lenght targets: 32 -- Total correct Counter : 32 -- Neurons used : 80 -- Percentage correct : 100.0%
Majority Gate width: 6 -- Lenght targets: 64 -- Total correct Counter : 64 -- Neurons used : 80 -- Percentage correct : 100.0%
Majority Gate width: 7 -- Lenght targets: 128 -- Total correct Counter : 45 -- Neurons used : 80 -- Percentage correct : 35.16%
Majority Gate width: 8 -- Lenght targets: 256 -- Total correct Counter : 127 -- Neurons used : 80 -- Percentage correct : 49.61%
Majority Gate width: 9 -- Lenght targets: 512 -- Total correct Counter : 282 -- Neurons used : 80 -- Percentage correct : 55.08%
Majority Gate width: 10 -- Lenght targets: 1024 -- Total correct Counter : 363 -- Neurons used : 80 -- Percentage correct : 35.45%
Majority Gate width: 11 -- Lenght targets: 2048 -- Total correct Counter : 1005 -- Neurons used : 80 -- Percentage correct : 49.07%
Majority Gate width: 12 -- Lenght targets: 4096 -- Total correct Counter : 2163 -- Neurons used : 80 -- Percentage correct : 52.81%
 --------------------- Neurons 128 ---------------------
Majority Gate width: 3 -- Lenght targets: 8 -- Total correct Counter : 8 -- Neurons used : 128 -- Percentage correct : 100.0%
Majority Gate width: 4 -- Lenght targets: 16 -- Total correct Counter : 16 -- Neurons used : 128 -- Percentage correct : 100.0%
Majority Gate width: 5 -- Lenght targets: 32 -- Total correct Counter : 32 -- Neurons used : 128 -- Percentage correct : 100.0%
Majority Gate width: 6 -- Lenght targets: 64 -- Total correct Counter : 64 -- Neurons used : 128 -- Percentage correct : 100.0%
Majority Gate width: 7 -- Lenght targets: 128 -- Total correct Counter : 122 -- Neurons used : 128 -- Percentage correct : 95.31%
Majority Gate width: 8 -- Lenght targets: 256 -- Total correct Counter : 138 -- Neurons used : 128 -- Percentage correct : 53.91%
Majority Gate width: 9 -- Lenght targets: 512 -- Total correct Counter : 291 -- Neurons used : 128 -- Percentage correct : 56.84%
Majority Gate width: 10 -- Lenght targets: 1024 -- Total correct Counter : 656 -- Neurons used : 128 -- Percentage correct : 64.06%
Majority Gate width: 11 -- Lenght targets: 2048 -- Total correct Counter : 992 -- Neurons used : 128 -- Percentage correct : 48.44%
Majority Gate width: 12 -- Lenght targets: 4096 -- Total correct Counter : 2290 -- Neurons used : 128 -- Percentage correct : 55.91%
 --------------------- Neurons 156 ---------------------
Majority Gate width: 3 -- Lenght targets: 8 -- Total correct Counter : 8 -- Neurons used : 156 -- Percentage correct : 100.0%
Majority Gate width: 4 -- Lenght targets: 16 -- Total correct Counter : 16 -- Neurons used : 156 -- Percentage correct : 100.0%
Majority Gate width: 5 -- Lenght targets: 32 -- Total correct Counter : 32 -- Neurons used : 156 -- Percentage correct : 100.0%
Majority Gate width: 6 -- Lenght targets: 64 -- Total correct Counter : 64 -- Neurons used : 156 -- Percentage correct : 100.0%
Majority Gate width: 7 -- Lenght targets: 128 -- Total correct Counter : 128 -- Neurons used : 156 -- Percentage correct : 100.0%
Majority Gate width: 8 -- Lenght targets: 256 -- Total correct Counter : 131 -- Neurons used : 156 -- Percentage correct : 51.17%
Majority Gate width: 9 -- Lenght targets: 512 -- Total correct Counter : 286 -- Neurons used : 156 -- Percentage correct : 55.86%
Majority Gate width: 10 -- Lenght targets: 1024 -- Total correct Counter : 688 -- Neurons used : 156 -- Percentage correct : 67.19%
Majority Gate width: 11 -- Lenght targets: 2048 -- Total correct Counter : 1079 -- Neurons used : 156 -- Percentage correct : 52.69%
Majority Gate width: 12 -- Lenght targets: 4096 -- Total correct Counter : 2332 -- Neurons used : 156 -- Percentage correct : 56.93%
 --------------------- Neurons 225 ---------------------
Majority Gate width: 3 -- Lenght targets: 8 -- Total correct Counter : 8 -- Neurons used : 225 -- Percentage correct : 100.0%
Majority Gate width: 4 -- Lenght targets: 16 -- Total correct Counter : 16 -- Neurons used : 225 -- Percentage correct : 100.0%
Majority Gate width: 5 -- Lenght targets: 32 -- Total correct Counter : 32 -- Neurons used : 225 -- Percentage correct : 100.0%
Majority Gate width: 6 -- Lenght targets: 64 -- Total correct Counter : 64 -- Neurons used : 225 -- Percentage correct : 100.0%
Majority Gate width: 7 -- Lenght targets: 128 -- Total correct Counter : 128 -- Neurons used : 225 -- Percentage correct : 100.0%
Majority Gate width: 8 -- Lenght targets: 256 -- Total correct Counter : 123 -- Neurons used : 225 -- Percentage correct : 48.05%
Majority Gate width: 9 -- Lenght targets: 512 -- Total correct Counter : 239 -- Neurons used : 225 -- Percentage correct : 46.68%
Majority Gate width: 10 -- Lenght targets: 1024 -- Total correct Counter : 617 -- Neurons used : 225 -- Percentage correct : 60.25%
Majority Gate width: 11 -- Lenght targets: 2048 -- Total correct Counter : 1059 -- Neurons used : 225 -- Percentage correct : 51.71%
Majority Gate width: 12 -- Lenght targets: 4096 -- Total correct Counter : 1977 -- Neurons used : 225 -- Percentage correct : 48.27%
 --------------------- Neurons 256 ---------------------
Majority Gate width: 3 -- Lenght targets: 8 -- Total correct Counter : 8 -- Neurons used : 256 -- Percentage correct : 100.0%
Majority Gate width: 4 -- Lenght targets: 16 -- Total correct Counter : 16 -- Neurons used : 256 -- Percentage correct : 100.0%
Majority Gate width: 5 -- Lenght targets: 32 -- Total correct Counter : 32 -- Neurons used : 256 -- Percentage correct : 100.0%
Majority Gate width: 6 -- Lenght targets: 64 -- Total correct Counter : 64 -- Neurons used : 256 -- Percentage correct : 100.0%
Majority Gate width: 7 -- Lenght targets: 128 -- Total correct Counter : 128 -- Neurons used : 256 -- Percentage correct : 100.0%
Majority Gate width: 8 -- Lenght targets: 256 -- Total correct Counter : 256 -- Neurons used : 256 -- Percentage correct : 100.0%
Majority Gate width: 9 -- Lenght targets: 512 -- Total correct Counter : 178 -- Neurons used : 256 -- Percentage correct : 34.77%
Majority Gate width: 10 -- Lenght targets: 1024 -- Total correct Counter : 482 -- Neurons used : 256 -- Percentage correct : 47.07%
Majority Gate width: 11 -- Lenght targets: 2048 -- Total correct Counter : 1199 -- Neurons used : 256 -- Percentage correct : 58.54%
Majority Gate width: 12 -- Lenght targets: 4096 -- Total correct Counter : 2002 -- Neurons used : 256 -- Percentage correct : 48.88%
 --------------------- Neurons 285 ---------------------
Majority Gate width: 3 -- Lenght targets: 8 -- Total correct Counter : 8 -- Neurons used : 285 -- Percentage correct : 100.0%
Majority Gate width: 4 -- Lenght targets: 16 -- Total correct Counter : 16 -- Neurons used : 285 -- Percentage correct : 100.0%
Majority Gate width: 5 -- Lenght targets: 32 -- Total correct Counter : 32 -- Neurons used : 285 -- Percentage correct : 100.0%
Majority Gate width: 6 -- Lenght targets: 64 -- Total correct Counter : 64 -- Neurons used : 285 -- Percentage correct : 100.0%
Majority Gate width: 7 -- Lenght targets: 128 -- Total correct Counter : 128 -- Neurons used : 285 -- Percentage correct : 100.0%
Majority Gate width: 8 -- Lenght targets: 256 -- Total correct Counter : 256 -- Neurons used : 285 -- Percentage correct : 100.0%
Majority Gate width: 9 -- Lenght targets: 512 -- Total correct Counter : 337 -- Neurons used : 285 -- Percentage correct : 65.82%
Majority Gate width: 10 -- Lenght targets: 1024 -- Total correct Counter : 574 -- Neurons used : 285 -- Percentage correct : 56.05%
Majority Gate width: 11 -- Lenght targets: 2048 -- Total correct Counter : 816 -- Neurons used : 285 -- Percentage correct : 39.84%
Majority Gate width: 12 -- Lenght targets: 4096 -- Total correct Counter : 1767 -- Neurons used : 285 -- Percentage correct : 43.14%
 --------------------- Neurons 450 ---------------------
Majority Gate width: 3 -- Lenght targets: 8 -- Total correct Counter : 8 -- Neurons used : 450 -- Percentage correct : 100.0%
Majority Gate width: 4 -- Lenght targets: 16 -- Total correct Counter : 16 -- Neurons used : 450 -- Percentage correct : 100.0%
Majority Gate width: 5 -- Lenght targets: 32 -- Total correct Counter : 32 -- Neurons used : 450 -- Percentage correct : 100.0%
Majority Gate width: 6 -- Lenght targets: 64 -- Total correct Counter : 64 -- Neurons used : 450 -- Percentage correct : 100.0%
Majority Gate width: 7 -- Lenght targets: 128 -- Total correct Counter : 128 -- Neurons used : 450 -- Percentage correct : 100.0%
Majority Gate width: 8 -- Lenght targets: 256 -- Total correct Counter : 256 -- Neurons used : 450 -- Percentage correct : 100.0%
Majority Gate width: 9 -- Lenght targets: 512 -- Total correct Counter : 255 -- Neurons used : 450 -- Percentage correct : 49.8%
Majority Gate width: 10 -- Lenght targets: 1024 -- Total correct Counter : 479 -- Neurons used : 450 -- Percentage correct : 46.78%
Majority Gate width: 11 -- Lenght targets: 2048 -- Total correct Counter : 904 -- Neurons used : 450 -- Percentage correct : 44.14%
Majority Gate width: 12 -- Lenght targets: 4096 -- Total correct Counter : 2075 -- Neurons used : 450 -- Percentage correct : 50.66%
 --------------------- Neurons 512 ---------------------
Majority Gate width: 3 -- Lenght targets: 8 -- Total correct Counter : 8 -- Neurons used : 512 -- Percentage correct : 100.0%
Majority Gate width: 4 -- Lenght targets: 16 -- Total correct Counter : 16 -- Neurons used : 512 -- Percentage correct : 100.0%
Majority Gate width: 5 -- Lenght targets: 32 -- Total correct Counter : 32 -- Neurons used : 512 -- Percentage correct : 100.0%
Majority Gate width: 6 -- Lenght targets: 64 -- Total correct Counter : 64 -- Neurons used : 512 -- Percentage correct : 100.0%
Majority Gate width: 7 -- Lenght targets: 128 -- Total correct Counter : 128 -- Neurons used : 512 -- Percentage correct : 100.0%
Majority Gate width: 8 -- Lenght targets: 256 -- Total correct Counter : 256 -- Neurons used : 512 -- Percentage correct : 100.0%
Majority Gate width: 9 -- Lenght targets: 512 -- Total correct Counter : 359 -- Neurons used : 512 -- Percentage correct : 70.12%
Majority Gate width: 10 -- Lenght targets: 1024 -- Total correct Counter : 545 -- Neurons used : 512 -- Percentage correct : 53.22%
Majority Gate width: 11 -- Lenght targets: 2048 -- Total correct Counter : 898 -- Neurons used : 512 -- Percentage correct : 43.85%
Majority Gate width: 12 -- Lenght targets: 4096 -- Total correct Counter : 2725 -- Neurons used : 512 -- Percentage correct : 66.53%
 --------------------- Neurons 560 ---------------------
Majority Gate width: 3 -- Lenght targets: 8 -- Total correct Counter : 8 -- Neurons used : 560 -- Percentage correct : 100.0%
Majority Gate width: 4 -- Lenght targets: 16 -- Total correct Counter : 16 -- Neurons used : 560 -- Percentage correct : 100.0%
Majority Gate width: 5 -- Lenght targets: 32 -- Total correct Counter : 32 -- Neurons used : 560 -- Percentage correct : 100.0%
Majority Gate width: 6 -- Lenght targets: 64 -- Total correct Counter : 64 -- Neurons used : 560 -- Percentage correct : 100.0%
Majority Gate width: 7 -- Lenght targets: 128 -- Total correct Counter : 128 -- Neurons used : 560 -- Percentage correct : 100.0%
Majority Gate width: 8 -- Lenght targets: 256 -- Total correct Counter : 256 -- Neurons used : 560 -- Percentage correct : 100.0%
Majority Gate width: 9 -- Lenght targets: 512 -- Total correct Counter : 512 -- Neurons used : 560 -- Percentage correct : 100.0%
Majority Gate width: 10 -- Lenght targets: 1024 -- Total correct Counter : 500 -- Neurons used : 560 -- Percentage correct : 48.83%
Majority Gate width: 11 -- Lenght targets: 2048 -- Total correct Counter : 927 -- Neurons used : 560 -- Percentage correct : 45.26%
Majority Gate width: 12 -- Lenght targets: 4096 -- Total correct Counter : 2245 -- Neurons used : 560 -- Percentage correct : 54.81%
 --------------------- Neurons 900 ---------------------
Majority Gate width: 3 -- Lenght targets: 8 -- Total correct Counter : 8 -- Neurons used : 900 -- Percentage correct : 100.0%
Majority Gate width: 4 -- Lenght targets: 16 -- Total correct Counter : 16 -- Neurons used : 900 -- Percentage correct : 100.0%
Majority Gate width: 5 -- Lenght targets: 32 -- Total correct Counter : 32 -- Neurons used : 900 -- Percentage correct : 100.0%
Majority Gate width: 6 -- Lenght targets: 64 -- Total correct Counter : 64 -- Neurons used : 900 -- Percentage correct : 100.0%
Majority Gate width: 7 -- Lenght targets: 128 -- Total correct Counter : 128 -- Neurons used : 900 -- Percentage correct : 100.0%
Majority Gate width: 8 -- Lenght targets: 256 -- Total correct Counter : 256 -- Neurons used : 900 -- Percentage correct : 100.0%
Majority Gate width: 9 -- Lenght targets: 512 -- Total correct Counter : 512 -- Neurons used : 900 -- Percentage correct : 100.0%
Majority Gate width: 10 -- Lenght targets: 1024 -- Total correct Counter : 628 -- Neurons used : 900 -- Percentage correct : 61.33%
Majority Gate width: 11 -- Lenght targets: 2048 -- Total correct Counter : 1002 -- Neurons used : 900 -- Percentage correct : 48.93%
Majority Gate width: 12 -- Lenght targets: 4096 -- Total correct Counter : 1798 -- Neurons used : 900 -- Percentage correct : 43.9%
 --------------------- Neurons 1024 ---------------------
Majority Gate width: 3 -- Lenght targets: 8 -- Total correct Counter : 8 -- Neurons used : 1024 -- Percentage correct : 100.0%
Majority Gate width: 4 -- Lenght targets: 16 -- Total correct Counter : 16 -- Neurons used : 1024 -- Percentage correct : 100.0%
Majority Gate width: 5 -- Lenght targets: 32 -- Total correct Counter : 32 -- Neurons used : 1024 -- Percentage correct : 100.0%
Majority Gate width: 6 -- Lenght targets: 64 -- Total correct Counter : 64 -- Neurons used : 1024 -- Percentage correct : 100.0%
Majority Gate width: 7 -- Lenght targets: 128 -- Total correct Counter : 128 -- Neurons used : 1024 -- Percentage correct : 100.0%
Majority Gate width: 8 -- Lenght targets: 256 -- Total correct Counter : 256 -- Neurons used : 1024 -- Percentage correct : 100.0%
Majority Gate width: 9 -- Lenght targets: 512 -- Total correct Counter : 512 -- Neurons used : 1024 -- Percentage correct : 100.0%
Majority Gate width: 10 -- Lenght targets: 1024 -- Total correct Counter : 556 -- Neurons used : 1024 -- Percentage correct : 54.3%
Majority Gate width: 11 -- Lenght targets: 2048 -- Total correct Counter : 937 -- Neurons used : 1024 -- Percentage correct : 45.75%
Majority Gate width: 12 -- Lenght targets: 4096 -- Total correct Counter : 2169 -- Neurons used : 1024 -- Percentage correct : 52.95%
 --------------------- Neurons 1124 ---------------------
Majority Gate width: 3 -- Lenght targets: 8 -- Total correct Counter : 8 -- Neurons used : 1124 -- Percentage correct : 100.0%
Majority Gate width: 4 -- Lenght targets: 16 -- Total correct Counter : 16 -- Neurons used : 1124 -- Percentage correct : 100.0%
Majority Gate width: 5 -- Lenght targets: 32 -- Total correct Counter : 32 -- Neurons used : 1124 -- Percentage correct : 100.0%
Majority Gate width: 6 -- Lenght targets: 64 -- Total correct Counter : 64 -- Neurons used : 1124 -- Percentage correct : 100.0%
Majority Gate width: 7 -- Lenght targets: 128 -- Total correct Counter : 128 -- Neurons used : 1124 -- Percentage correct : 100.0%
Majority Gate width: 8 -- Lenght targets: 256 -- Total correct Counter : 256 -- Neurons used : 1124 -- Percentage correct : 100.0%
Majority Gate width: 9 -- Lenght targets: 512 -- Total correct Counter : 512 -- Neurons used : 1124 -- Percentage correct : 100.0%
Majority Gate width: 10 -- Lenght targets: 1024 -- Total correct Counter : 1024 -- Neurons used : 1124 -- Percentage correct : 100.0%
Majority Gate width: 11 -- Lenght targets: 2048 -- Total correct Counter : 854 -- Neurons used : 1124 -- Percentage correct : 41.7%
Majority Gate width: 12 -- Lenght targets: 4096 -- Total correct Counter : 1944 -- Neurons used : 1124 -- Percentage correct : 47.46%
 --------------------- Neurons 1800 ---------------------
Majority Gate width: 3 -- Lenght targets: 8 -- Total correct Counter : 8 -- Neurons used : 1800 -- Percentage correct : 100.0%
Majority Gate width: 4 -- Lenght targets: 16 -- Total correct Counter : 16 -- Neurons used : 1800 -- Percentage correct : 100.0%
Majority Gate width: 5 -- Lenght targets: 32 -- Total correct Counter : 32 -- Neurons used : 1800 -- Percentage correct : 100.0%
Majority Gate width: 6 -- Lenght targets: 64 -- Total correct Counter : 64 -- Neurons used : 1800 -- Percentage correct : 100.0%
Majority Gate width: 7 -- Lenght targets: 128 -- Total correct Counter : 128 -- Neurons used : 1800 -- Percentage correct : 100.0%
Majority Gate width: 8 -- Lenght targets: 256 -- Total correct Counter : 256 -- Neurons used : 1800 -- Percentage correct : 100.0%
Majority Gate width: 9 -- Lenght targets: 512 -- Total correct Counter : 512 -- Neurons used : 1800 -- Percentage correct : 100.0%
Majority Gate width: 10 -- Lenght targets: 1024 -- Total correct Counter : 1024 -- Neurons used : 1800 -- Percentage correct : 100.0%
Majority Gate width: 11 -- Lenght targets: 2048 -- Total correct Counter : 826 -- Neurons used : 1800 -- Percentage correct : 40.33%
Majority Gate width: 12 -- Lenght targets: 4096 -- Total correct Counter : 2004 -- Neurons used : 1800 -- Percentage correct : 48.93%
 --------------------- Neurons 2048 ---------------------
Majority Gate width: 3 -- Lenght targets: 8 -- Total correct Counter : 8 -- Neurons used : 2048 -- Percentage correct : 100.0%
Majority Gate width: 4 -- Lenght targets: 16 -- Total correct Counter : 16 -- Neurons used : 2048 -- Percentage correct : 100.0%
Majority Gate width: 5 -- Lenght targets: 32 -- Total correct Counter : 32 -- Neurons used : 2048 -- Percentage correct : 100.0%
Majority Gate width: 6 -- Lenght targets: 64 -- Total correct Counter : 64 -- Neurons used : 2048 -- Percentage correct : 100.0%
Majority Gate width: 7 -- Lenght targets: 128 -- Total correct Counter : 128 -- Neurons used : 2048 -- Percentage correct : 100.0%
Majority Gate width: 8 -- Lenght targets: 256 -- Total correct Counter : 256 -- Neurons used : 2048 -- Percentage correct : 100.0%
Majority Gate width: 9 -- Lenght targets: 512 -- Total correct Counter : 512 -- Neurons used : 2048 -- Percentage correct : 100.0%
Majority Gate width: 10 -- Lenght targets: 1024 -- Total correct Counter : 1024 -- Neurons used : 2048 -- Percentage correct : 100.0%
Majority Gate width: 11 -- Lenght targets: 2048 -- Total correct Counter : 980 -- Neurons used : 2048 -- Percentage correct : 47.85%
Majority Gate width: 12 -- Lenght targets: 4096 -- Total correct Counter : 2041 -- Neurons used : 2048 -- Percentage correct : 49.83%
 --------------------- Neurons 2248 ---------------------
Majority Gate width: 3 -- Lenght targets: 8 -- Total correct Counter : 8 -- Neurons used : 2248 -- Percentage correct : 100.0%
Majority Gate width: 4 -- Lenght targets: 16 -- Total correct Counter : 16 -- Neurons used : 2248 -- Percentage correct : 100.0%
Majority Gate width: 5 -- Lenght targets: 32 -- Total correct Counter : 32 -- Neurons used : 2248 -- Percentage correct : 100.0%
Majority Gate width: 6 -- Lenght targets: 64 -- Total correct Counter : 64 -- Neurons used : 2248 -- Percentage correct : 100.0%
Majority Gate width: 7 -- Lenght targets: 128 -- Total correct Counter : 128 -- Neurons used : 2248 -- Percentage correct : 100.0%
Majority Gate width: 8 -- Lenght targets: 256 -- Total correct Counter : 256 -- Neurons used : 2248 -- Percentage correct : 100.0%
Majority Gate width: 9 -- Lenght targets: 512 -- Total correct Counter : 512 -- Neurons used : 2248 -- Percentage correct : 100.0%
Majority Gate width: 10 -- Lenght targets: 1024 -- Total correct Counter : 1024 -- Neurons used : 2248 -- Percentage correct : 100.0%
Majority Gate width: 11 -- Lenght targets: 2048 -- Total correct Counter : 2048 -- Neurons used : 2248 -- Percentage correct : 100.0%
Majority Gate width: 12 -- Lenght targets: 4096 -- Total correct Counter : 1929 -- Neurons used : 2248 -- Percentage correct : 47.09%
 --------------------- Neurons 3600 ---------------------
Majority Gate width: 3 -- Lenght targets: 8 -- Total correct Counter : 8 -- Neurons used : 3600 -- Percentage correct : 100.0%
Majority Gate width: 4 -- Lenght targets: 16 -- Total correct Counter : 16 -- Neurons used : 3600 -- Percentage correct : 100.0%
Majority Gate width: 5 -- Lenght targets: 32 -- Total correct Counter : 32 -- Neurons used : 3600 -- Percentage correct : 100.0%
Majority Gate width: 6 -- Lenght targets: 64 -- Total correct Counter : 64 -- Neurons used : 3600 -- Percentage correct : 100.0%
Majority Gate width: 7 -- Lenght targets: 128 -- Total correct Counter : 128 -- Neurons used : 3600 -- Percentage correct : 100.0%
Majority Gate width: 8 -- Lenght targets: 256 -- Total correct Counter : 256 -- Neurons used : 3600 -- Percentage correct : 100.0%
Majority Gate width: 9 -- Lenght targets: 512 -- Total correct Counter : 512 -- Neurons used : 3600 -- Percentage correct : 100.0%
Majority Gate width: 10 -- Lenght targets: 1024 -- Total correct Counter : 1024 -- Neurons used : 3600 -- Percentage correct : 100.0%
Majority Gate width: 11 -- Lenght targets: 2048 -- Total correct Counter : 2048 -- Neurons used : 3600 -- Percentage correct : 100.0%
Majority Gate width: 12 -- Lenght targets: 4096 -- Total correct Counter : 2115 -- Neurons used : 3600 -- Percentage correct : 51.64%
 --------------------- Neurons 4096 ---------------------
Majority Gate width: 3 -- Lenght targets: 8 -- Total correct Counter : 8 -- Neurons used : 4096 -- Percentage correct : 100.0%
Majority Gate width: 4 -- Lenght targets: 16 -- Total correct Counter : 16 -- Neurons used : 4096 -- Percentage correct : 100.0%
Majority Gate width: 5 -- Lenght targets: 32 -- Total correct Counter : 32 -- Neurons used : 4096 -- Percentage correct : 100.0%
Majority Gate width: 6 -- Lenght targets: 64 -- Total correct Counter : 64 -- Neurons used : 4096 -- Percentage correct : 100.0%
Majority Gate width: 7 -- Lenght targets: 128 -- Total correct Counter : 128 -- Neurons used : 4096 -- Percentage correct : 100.0%
Majority Gate width: 8 -- Lenght targets: 256 -- Total correct Counter : 256 -- Neurons used : 4096 -- Percentage correct : 100.0%
Majority Gate width: 9 -- Lenght targets: 512 -- Total correct Counter : 512 -- Neurons used : 4096 -- Percentage correct : 100.0%
Majority Gate width: 10 -- Lenght targets: 1024 -- Total correct Counter : 1024 -- Neurons used : 4096 -- Percentage correct : 100.0%
Majority Gate width: 11 -- Lenght targets: 2048 -- Total correct Counter : 2048 -- Neurons used : 4096 -- Percentage correct : 100.0%
Majority Gate width: 12 -- Lenght targets: 4096 -- Total correct Counter : 1591 -- Neurons used : 4096 -- Percentage correct : 38.84%
 --------------------- Neurons 4400 ---------------------
Majority Gate width: 3 -- Lenght targets: 8 -- Total correct Counter : 8 -- Neurons used : 4400 -- Percentage correct : 100.0%
Majority Gate width: 4 -- Lenght targets: 16 -- Total correct Counter : 16 -- Neurons used : 4400 -- Percentage correct : 100.0%
Majority Gate width: 5 -- Lenght targets: 32 -- Total correct Counter : 32 -- Neurons used : 4400 -- Percentage correct : 100.0%
Majority Gate width: 6 -- Lenght targets: 64 -- Total correct Counter : 64 -- Neurons used : 4400 -- Percentage correct : 100.0%
Majority Gate width: 7 -- Lenght targets: 128 -- Total correct Counter : 128 -- Neurons used : 4400 -- Percentage correct : 100.0%
Majority Gate width: 8 -- Lenght targets: 256 -- Total correct Counter : 256 -- Neurons used : 4400 -- Percentage correct : 100.0%
Majority Gate width: 9 -- Lenght targets: 512 -- Total correct Counter : 512 -- Neurons used : 4400 -- Percentage correct : 100.0%
Majority Gate width: 10 -- Lenght targets: 1024 -- Total correct Counter : 1024 -- Neurons used : 4400 -- Percentage correct : 100.0%
Majority Gate width: 11 -- Lenght targets: 2048 -- Total correct Counter : 2048 -- Neurons used : 4400 -- Percentage correct : 100.0%
Majority Gate width: 12 -- Lenght targets: 4096 -- Total correct Counter : 4095 -- Neurons used : 4400 -- Percentage correct : 99.98%

Considering using it for a large language model (just for fun) which has unlimited meassureable outputs it would be impossible to use this network as the size would be in the 10^80 in size neurons. It would learn it perfectly all languages and code, but we would be unable to run it anywhere with current compute.

Am i doing something wrong, or is this the expected behaviour? Did you also run some tests on it?

Update: did run a test if epochs matter, and statistically they dont matter, 1 epoch is enough , but the neurons have to be at least 10% greater than the different possible outputs. Did run the test from 1-512 epochs...

Best regards

hunar4321 commented 10 months ago

Hi thanks for the test. I didn't run tests but your conclusion about using large number of neurons seems correct. The reason is not related specifically to RLS but it's related to all single layered feed-forward networks. The advantage of this RLS method is its ability to converge and learn the weights fast with few iterations. The disadvantage here is that it's a single layered network. Therefore, it has all the issues that single layered networks has. Single layered networks require large number of parameters to model patterns that has large number of features. I explained why this is the case in my GPT video at the time mark: 11:15 here. Hope that helps: https://youtu.be/l-CjXFmcVzY?t=675

snapo commented 10 months ago

I will play around with it a little more, because i realy find it interesting and if one finds a way to have multiple layers it might be a very very very good and quick learner.... What might help you is that i think the brain has "like microprocessors" one or multiple clocks ticks (the spike you see when you meassure it)....

considering binary this would mean we would have to add somehow a binary clock to the input like 0,1,0,1,0,1 (for example a 128bit value something we never achive) per datapoint.

I saw this effect is helping if you try to predict a decreasing sinus soidal wave that has spikes at specific points. The example you have can predict everything that is even in the prediction, but it cant predict uneven things like every 3. value as it moves across time. if one adds the binary timer it is able to predict also the uneven numbers.

Again thanks for all your hard work....