search

likejazz / llama3.cuda

llama3.cuda is a pure C/CUDA implementation for Llama 3 model.
MIT License
294 stars 20 forks source link

2,823 tokens/s seems extremely high! #4

Open romitjain opened 3 months ago

romitjain commented 3 months ago

Hey,

I think the tokens/s calculation might be incorrect. I can see that you are computing time by timing the CPU clock here: https://github.com/likejazz/llama3.cuda/blob/master/llama3.cu#L789

This might result in an incorrect number because the actual code is running on the GPU and the CPU is just dispatching the kernel. Hence, I suspect that the time that you are getting is CPU execution time of dispatching the kernel.

The correct way will be using cuda events. Reference: https://developer.nvidia.com/blog/how-implement-performance-metrics-cuda-cc/

likejazz commented 3 months ago

@romitjain oh, thank you for the clarification. could you please send me a patch?