Testing rnsnorm

[adi-lb-phoenix]

This is in continuation after : [(https://github.com/chsasank/llama.lisp/issues/63)] This will contain the steps taken to test the rnsnorm function.

Original Code :

void rmsnorm(float* o, float* x, float* weight, int size) {
    // calculate sum of squares
    float ss = 0.0f;
    for (int j = 0; j < size; j++) {
        ss += x[j] * x[j];
    }
    ss /= size;
    ss += 1e-5f;
    ss = 1.0f / sqrtf(ss);
    // normalize and scale
    for (int j = 0; j < size; j++) {
        o[j] = weight[j] * (ss * x[j]);
    }
}

[adi-lb-phoenix]

Modified code that collects data and stores it in files.

void rmsnorm(float* o, float* x, float* weight, int size) {

    // calculate sum of squares
    char *file_weights="testing/weight.txt";
    char *file_o = "testing/o.txt";
    char *file_x = "testing/x.txt";
    char *file_size = "testing/size.txt";

    FILE *f_size = fopen(file_size,"w");
    fprintf(f_size,"%d",size);
    fclose(f_size);

    FILE *fp=fopen(file_x,"w");
    float ss = 0.0f;
    for (int j = 0; j < size; j++) {
        ss += x[j] * x[j];
        fprintf(fp,"%f\n",x[j]);
    }
    fclose(fp);

    ss /= size;
    ss += 1e-5f;
    ss = 1.0f / sqrtf(ss);
    // normalize and scale
    FILE *f_weight = fopen(file_weights,"w");
    FILE *f_o = fopen(file_o,"w");
    for (int j = 0; j < size; j++) {
        o[j] = weight[j] * (ss * x[j]);
        fprintf(f_weight,"%f\n",x[j]);
        fprintf(f_o,"%f\n",x[j]);
    }
    fclose(f_weight);
    fclose(f_o);
}

[adi-lb-phoenix]

After having made changes to my rnsnorm function. I compiled the code using :

~/llm_inferences/llama2.c$ make run
gcc -O3 -o run run.c -lm
gcc -O3 -o runq runq.c -lm

Test run to check if the model is inferenced correctly : ./run stories15M.bin Once upon a time, there was a little girl named Lily. She loved to help her mommy in the kitchen. One day, they were making cookies together. Lily loved the smell of the cookies baking in the oven. After they finished, Lily's mommy asked her to find the sugar in the cupboard. Lily looked and looked, but she couldn't find the sugar. She thought she saw a stupid mistake. But then, Lily remembered that she had the sugar in the sink earlier that day. She ran to the sink and drank the sugar. She was so happy! Her mommy was so proud of her for finding the sugar. From then on, Lily always remembered to check the sink before using it. achieved tok/s: 111.498258

[adi-lb-phoenix]

Now I shall pass the data written to respective files to my c-lisp function to verify if the correct value is being computed.

[adi-lb-phoenix]

sample output received from rns.sexp : 23.058947 21.489456 19.908035 18.314674 16.709381 15.092152 13.462987 11.821887 10.168852 8.503881 6.826976 5.138135

sample output received from the same function in c file. 23.058008 21.488585 19.907225 18.313931 16.708702 15.091538 13.462440 11.821406 10.168439 8.503536 6.826698 5.137926

[adi-lb-phoenix]

In my C function when I changed the code from

ss += 1e-5f;
output : 
10.168439
8.503536
6.826698
5.137926
3.437219

to

float e = 2.718281828;
 ss += e-5.0;

output : 
10.168852
8.503881
6.826976
5.138135
3.437358

When I make the below change in my C file the output does match my rns.sexp file. But I havent yet found a way to match the original unchanged C function values to my rns.sexp

[adi-lb-phoenix]

Now I shall pass the data written to respective files to my c-lisp function to verify if the correct value is being computed.

I did not use the data collected this way. Instead we generated it.

How did I generate my inputs in C function ? I generated the values in c-lisp and stored the data in two different file . The first file was called weight_generated.txt and the second file was called x_generated.txt.

Function and command used to store data into weight_generated.txt :

    (define ((initialise void) (w (ptr float)) (x (ptr float)) (size int))
        (declare count int)
        (declare initialise_float float)
        (set initialise_float 1.0)
        (for ((set count 0) (lt count size) (set count (add count 1)))
            (store (ptradd w count) initialise_float)
            (set initialise_float (fadd initialise_float 1.0))
            (call fprint (load (ptradd w count)))

        )   
        ;(call putchar 10)

        (for ((set count 0) (lt count size) (set count (add count 1)))
            (store (ptradd x count) initialise_float)
            (set initialise_float (fsub initialise_float 1.0))
            ;(call fprint (load (ptradd x count)))

        )
        (ret)
    )

command used : guile ../../../utils/sexp-json.scm < rnsnorm/rns.sexp | python ../../../c-lisp.py | python ../../../brilisp.py | python ../../../llvm.py | bash ../../brilisp/run.sh {args} | cat > rnsnorm/weight_generated.txt

function and command used to store data into x_generated.txt :

    (define ((initialise void) (w (ptr float)) (x (ptr float)) (size int))
        (declare count int)
        (declare initialise_float float)
        (set initialise_float 1.0)
        (for ((set count 0) (lt count size) (set count (add count 1)))
            (store (ptradd w count) initialise_float)
            (set initialise_float (fadd initialise_float 1.0))
            ;(call fprint (load (ptradd w count)))

        )   
        ;(call putchar 10)

        (for ((set count 0) (lt count size) (set count (add count 1)))
            (store (ptradd x count) initialise_float)
            (set initialise_float (fsub initialise_float 1.0))
            (call fprint (load (ptradd x count)))

        )
        (ret)
    )

command used : guile ../../../utils/sexp-json.scm < rnsnorm/rns.sexp | python ../../../c-lisp.py | python ../../../brilisp.py | python ../../../llvm.py | bash ../../brilisp/run.sh {args} | cat > rnsnorm/x_generated.txt

For my C version of rnsnorm. I used the inputs from the above two files to calculate the output.

[adi-lb-phoenix]

sample output received from rns.sexp : 23.058947 21.489456 19.908035 18.314674 16.709381 15.092152 13.462987 11.821887 10.168852 8.503881 6.826976 5.138135

While calculating the outputs for rns.sexp. I did not print the outputs of the generated number for the floating point array "w" and "x". Instead I just stored them and used it for further calculations.

[chsasank]

I found where the issue is in your code:

1e-5 is the notation for 0.0005. Read the wiki: https://en.wikipedia.org/wiki/Scientific_notation#E_notation

That way you should get the same number.

[adi-lb-phoenix]

Made this small change in C-lisp as suggested (set ss (fadd ss 0.0005)) . Output from c-lisp :

.
.
.
6.826698
5.137926
3.437219

Output from C-file:

.
.
.
6.826698
5.137926
3.437219