search

alan-turing-institute / minGPT

A minimal PyTorch re-implementation of the OpenAI GPT (Generative Pretrained Transformer) training
MIT License
1 stars 0 forks source link

GPT2 in pure C/CUDA #6

Open llewelld opened 1 month ago

llewelld commented 1 month ago

Python is doing a lot of heavy lifting and hiding a lot of the complexity. It'd be interesting to compare the Python GPT code with Kaparthy's pure C/CUDA implementaton of the same:

https://github.com/karpathy/llm.c

There's a text-based walkthrough from Karpathy in the discussions on the repo:

https://github.com/karpathy/llm.c/discussions/481

llewelld commented 1 month ago

It might be interesting to convert this to use SYCL, which could potentially be even easier and it doesn't look like it's been done yet.