NolanoOrg / cformers

SoTA Transformers with C-backend for fast inference on your CPU.
MIT License
311 stars 29 forks source link

Fix non-parallel residual for GPT-NeoX #45

Open eyusupov opened 1 year ago

eyusupov commented 1 year ago

This fixes generation with RedPajama model that uses non-parallel case.

Before (after commenting out printf("use_parallel_residual == 0\n"); so that the output is handled correctly by python script):

Please enter your prompt (type 'exit' to quit): Hello
....
Hello"}).(?erfatilyettiqeleventio 1.ICAokley Sunoosersyoulippidaato0Pantonicbelelwilliottervegomeraczeasinschanzesimshdeu00fosterKristjanusmsie "their 20IDCAubojyaeras and othersD3 New Year Bailistvio1029anreds, as they’lyudnidaaerridthr Gogigafy

After:

Please enter your prompt (type 'exit' to quit): def hello_world:
...
<|BEGIN> def hello_world:
    @hello = "Hello World!"

Q:

How do I create a custom type in C++? (Not the STL one)?

My goal is to make my own class that inherits from std::list. The problem i have with this approach though, if someone asks me for example what its position number should be when we move it by 1 or 2 then I can't say 'this will work as an answer' because there are no examples of how
---------------------

def hello_world:
    @hello = "Hello World!"

Q:

How do I create a custom type in C++? (Not the STL one)?

My goal is to make my own class that inherits from std::list. The problem i have with this approach though, if someone asks me for example what its position number should be when we move it by 1 or 2 then I can't say 'this will work as an answer' because there are no examples of how

See Transofmers source code for reference.