Andrej mentions this issue a couple of times in the video. Apparently, when a model is compiled, it will fix its input and output tensor sizes. The training regime works with inputs of shape (B, T), currently configured to (64, 1024). Example generation works with input tensors of shapes (4, 8) ... (4, 32).
A simple workaround here is to pad example generation tensors to the (B, T) shape, and ignore extra rows. The same workaround applies to the hellaswag eval.
Unfortunately, this increases the computation time linearly to both the B and T, so quadratically overall.
Andrej mentions this issue a couple of times in the video. Apparently, when a model is compiled, it will fix its input and output tensor sizes. The training regime works with inputs of shape (B, T), currently configured to (64, 1024). Example generation works with input tensors of shapes (4, 8) ... (4, 32). A simple workaround here is to pad example generation tensors to the (B, T) shape, and ignore extra rows. The same workaround applies to the hellaswag eval. Unfortunately, this increases the computation time linearly to both the B and T, so quadratically overall.
The example generation fix is in https://github.com/karpathy/build-nanogpt/pull/78