Closed Buck008 closed 1 month ago
Sure, I will do all these above. I am not familar with embedding for nano gpt now, I may need time to see that. And the embeding layer now is not supported in hardware.
Sorry I don't understand how the embedding layer do in the network, so I just create the two matrixes. And for the output layer, do you mean a layernorm or RMSnorm? For hardware, we are just going to support attention and mlp. Sorry for the late update, this week happened too many things. : )
Sure, I will do all these above. I am not familar with embedding for nano gpt now, I may need time to see that. And the embeding layer now is not supported in hardware.
Good to know, I think we're closing in on a solution for the embedding layer for hardware too
I've used Numpy and referenced code from NanoGPT to write the decoder forward dataflow based on my understanding. Currently, all values are floating-point numbers. (All weights are randomly generated, and I haven't implemented the dropout function because I believe it won't be executed in hardware.) I would greatly appreciate it if someone could help me check whether my understanding is correct. (Please excuse my limited Python skills.) If this is right, I am going to write the integer version,which will be the golden brick generator.