Closed cjxjxjx closed 2 months ago
I can confirm that this can be reproduced in the docker environment. @FindHao Can you help take a look at this issue?
@xuzhao9 The problem also occurs on the previous version of TorchBench(ghcr.io/pytorch/torchbench:dev20230619). It looks like it is from the first time DALLE2 was included in TorchBench. I'm not sure if we can fix it on our side or from the upstream repo since we have limited control over the model's init.py. I'll have a try.
We are dropping DALLE2_pytorch because it does not support numpy 2.0: https://github.com/pytorch/benchmark/pull/2311
Issue Description I encounter a RuntimeError related to gradient computation when enabling accuracy checks during the training of DALLE2_pytorch in a GPU docker environment. The training runs without issues when the
--accuracy
flag is not used.Steps to Reproduce
python install.py DALLE2_pytorch
python run.py DALLE2_pytorch -d cuda -t train --accuracy
Expected Behavior The training process should run without errors and perform accuracy checks without causing runtime errors.
Actual Behavior The script executes successfully without the --accuracy flag. However, when the accuracy check is enabled, it fails with the following error message:
Additional Context PyTorch version: 2.2.2 CUDA version: 12.4.0.041