zscwind
commented
1 year ago Torch versions as follow: pytorch-triton 2.0.0+0d7e753227 pypi_0 pypi torch 2.0.0.dev20230202+cu116 pypi_0 pypi torchaudio 2.0.0.dev20230201+cu116 pypi_0 pypi torchvision 0.15.0.dev20230201+cu116 pypi_0 pypi
Here is the cuda info: nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2021 NVIDIA Corporation Built on Fri_Dec_17_18:16:03_PST_2021 Cuda compilation tools, release 11.6, V11.6.55 Build cuda_11.6.r11.6/compiler.30794723_0
vocab_size not found in data/openwebtext/meta.pkl, using GPT-2 default of 50257 Initializing a new model from scratch number of parameters: 124.34M compiling the model... (takes a ~minute) To use data.metrics please install scikit-learn. See https://scikit-learn.org/stable/index.html Traceback (most recent call last): File "train.py", line 243, in
losses = estimate_loss()
File "/data/sczhang21/miniconda3/envs/gpt/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, kwargs)
File "train.py", line 204, in estimate_loss
logits, loss = model(X, Y)
File "/data/sczhang21/miniconda3/envs/gpt/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, *kwargs)
File "/data/sczhang21/miniconda3/envs/gpt/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 82, in forward
return self.dynamo_ctx(self._orig_mod.forward)(args, kwargs)
File "/data/sczhang21/miniconda3/envs/gpt/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 209, in _fn
return fn(*args, *kwargs)
File "/data/sczhang21/project/llm/nanoGPT/model.py", line 169, in forward
def forward(self, idx, targets=None):
File "/data/sczhang21/miniconda3/envs/gpt/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 209, in _fn
return fn(args, **kwargs)
File "/data/sczhang21/miniconda3/envs/gpt/lib/python3.8/site-packages/torch/_functorch/aot_autograd.py", line 2497, in forward
return compiled_fn(full_args)
File "/data/sczhang21/miniconda3/envs/gpt/lib/python3.8/site-packages/torch/_functorch/aot_autograd.py", line 1065, in new_fn
fw_outs = call_func_with_args(compiled_fw, args, disable_amp=disable_amp)
File "/data/sczhang21/miniconda3/envs/gpt/lib/python3.8/site-packages/torch/_functorch/aot_autograd.py", line 1021, in call_func_with_args
out = normalize_as_list(f(args))
File "/data/sczhang21/miniconda3/envs/gpt/lib/python3.8/site-packages/torch/_inductor/compile_fx.py", line 224, in run
return model(new_inputs)
File "/tmp/torchinductor_sczhang21/jr/cjrwte733i2pk5nmciviucuo3bylz2vkq45cbven6tmextru7u6t.py", line 936, in call
triton__10.run(buf70, buf37, buf46, buf60, buf69, arg6_1, buf74, 12288, 768, grid=grid(12288), stream=stream0)
File "/data/sczhang21/miniconda3/envs/gpt/lib/python3.8/site-packages/torch/_inductor/triton_ops/autotune.py", line 188, in run
result = launcher(
File "", line 6, in launcher
RuntimeError: Triton Error [CUDA]: invalid argument