search

karpathy / llm.c

LLM training in simple, raw C/CUDA
MIT License
24.42k stars 2.76k forks source link

is max_seq_len configurable or hardcoded parameter? #569

Open morphpiece opened 5 months ago

morphpiece commented 5 months ago

Today I was going to train a gpt3_124m model, when I noticed that the max_seq_len is hardcoded here and at the same time, it's a configurable parameter here. Then I executed the training executable with and without -t 2048 and find that the number of model parameters stays the same at 124475904.

is there a 🐛 bounty? 😃

gordicaleksa commented 5 months ago

yeah, we're focused on using the exact setup of gpt-2/3 right now, the idea is not to make this as configurable as possible, many things are hardcoded which goes against the general SWE principles, but it's less important given the goal of this project.

morphpiece commented 5 months ago

Two points come to mind.

It's fine to hardcode at this stage (like vocabulary), but then it should not be available as a configurable parameter. That can throw someone off completely. It's better to fix it like vocabulary is fixed and not be available as a configurable input.

Secondly, since one of the usecases is GPT-3 (which uses max_seq_len of 2048), it is assumed that users will change this value from the hardcoded 1024 to 2048. In that respect, probably hardcoding this param needs to be reconsidered.

Thanks again for this amazing work!