search

karpathy / llama2.c

Inference Llama 2 in one file of pure C
MIT License
17.35k stars 2.07k forks source link

Once upon a time, there was a little girl named Lily #465

Open GilesBathgate opened 10 months ago

GilesBathgate commented 10 months ago

There seems to be a lot of bias in the models because of the repetition of this prefix in the training data. 

grep -c "Once upon a time, there was a little girl named Lily" tinystories.txt 
53467

Does anyone know of ways of making qualitative measurements of the training data, i.e diversity of text, (aside from how well it compresses)

GilesBathgate commented 10 months ago

Using some simple bash scripts I found:

Count  | Prefix
53,467 | Once upon a time, there was a little girl named Lily
30,640 | Once upon a time, there was a little girl named Sue
20,910 | Once upon a time, there was a little girl named Mia
19,419 | Once upon a time, there was a little girl named Lucy
17,045 | Once upon a time, there was a little girl named Amy
 4,546 | Once upon a time, there was a little girl named Sally
 2,409 | Once upon a time, there was a little girl named Jane
 1,729 | Once upon a time, there was a little girl named Emma
 1,414 | Once upon a time, there was a little girl named Lisa
 1,257 | Once upon a time, there was a little girl named Anna

Likewise Tim is very popular

Count   | Prefix
125,460 | Once upon a time, there was a little boy named Tim
  9,438 | Once upon a time, there was a little boy named Tom
    916 | Once upon a time, there was a little boy named Jack
    543 | Once upon a time, there was a little boy named Mark
    527 | Once upon a time, there was a little boy named Sam
    317 | Once upon a time, there was a little boy named Joe
    294 | Once upon a time, there was a little boy named John
    247 | Once upon a time, there was a little boy named Timmy
    170 | Once upon a time, there was a little boy named Max
    149 | Once upon a time, there was a little boy named Bob
GilesBathgate commented 10 months ago

Here is a quick hack to attempt to remove the bias: https://gist.github.com/GilesBathgate/a7a0a18276a2a79836cb6cb44d8656c2

GilesBathgate commented 9 months ago

So weird, that ChatGPT has this bias. There are many ways to show it aside from the obvious: https://chat.openai.com/share/f774a1d4-1940-4249-bf1a-642e1ab4ef8f

GilesBathgate commented 7 months ago

@karpathy Could this be a suitable starting point for implementing nano RLHF (where the 'Human feedback' is simply some kind of contrastive loss function that penalises repeating prefixes)

SenatorAP28 commented 7 months ago

Hello... It could be a suitable start and am new to the tech world

On Sat, Mar 9, 2024, 4:45 PM Giles Bathgate @.***> wrote:

@karpathy https://github.com/karpathy Could this be a suitable starting point for implementing nano RLHF (where the 'Human feedback' is simply some kind of contrastive loss function that penalises repeating prefixes)

— Reply to this email directly, view it on GitHub https://github.com/karpathy/llama2.c/issues/465#issuecomment-1986892649, or unsubscribe https://github.com/notifications/unsubscribe-auth/BBQ2R4N6PRTR5NTOHNYLYYTYXMVBHAVCNFSM6AAAAABA6CQXAOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBWHA4TENRUHE . You are receiving this because you are subscribed to this thread.Message ID: @.***>