search

hpcaitech / Open-Sora

Open-Sora: Democratizing Efficient Video Production for All
https://hpcaitech.github.io/Open-Sora/
Apache License 2.0
20.34k stars 1.92k forks source link

RuntimeError: GET was unable to find an engine to execute this computation #445

Closed aulaywang closed 2 weeks ago

aulaywang commented 3 weeks ago

rank0: Traceback (most recent call last):
rank0: File "/data/code/Open-Sora/scripts/inference.py", line 175, in

rank0: File "/data/code/Open-Sora/scripts/inference.py", line 149, in main rank0: samples = scheduler.sample( rank0: File "/root/miniconda3/envs/opensora/lib/python3.10/site-packages/opensora/schedulers/iddpm/init.py", line 75, in sample rank0: samples = self.p_sample_loop( rank0: File "/root/miniconda3/envs/opensora/lib/python3.10/site-packages/opensora/schedulers/iddpm/gaussian_diffusion.py", line 481, in p_sample_loop rank0: for sample in self.p_sample_loop_progressive( rank0: File "/root/miniconda3/envs/opensora/lib/python3.10/site-packages/opensora/schedulers/iddpm/gaussian_diffusion.py", line 534, in p_sample_loop_progressive rank0: out = self.p_sample( rank0: File "/root/miniconda3/envs/opensora/lib/python3.10/site-packages/opensora/schedulers/iddpm/gaussian_diffusion.py", line 429, in p_sample rank0: out = self.p_mean_variance( rank0: File "/root/miniconda3/envs/opensora/lib/python3.10/site-packages/opensora/schedulers/iddpm/respace.py", line 95, in p_mean_variance rank0: return super().p_mean_variance(self._wrap_model(model), args, kwargs) rank0: File "/root/miniconda3/envs/opensora/lib/python3.10/site-packages/opensora/schedulers/iddpm/gaussian_diffusion.py", line 286, in p_mean_variance rank0: model_output = model(x, t, model_kwargs) rank0: File "/root/miniconda3/envs/opensora/lib/python3.10/site-packages/opensora/schedulers/iddpm/respace.py", line 127, in call rank0: return self.model(x, new_ts, kwargs) rank0: File "/root/miniconda3/envs/opensora/lib/python3.10/site-packages/opensora/schedulers/iddpm/init.py", line 96, in forward_with_cfg rank0: model_out = model.forward(combined, timestep, y, kwargs) rank0: File "/root/miniconda3/envs/opensora/lib/python3.10/site-packages/opensora/models/stdit/stdit.py", line 276, in forward rank0: x = self.x_embedder(x) # B, N, C: File "/root/miniconda3/envs/opensora/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl rank0: return self._call_impl(args, *kwargs) rank0: File "/root/miniconda3/envs/opensora/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl rank0: return forward_call(args, **kwargs)

rank0: File "/root/miniconda3/envs/opensora/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl rank0: return self._call_impl(*args, *kwargs) rank0: File "/root/miniconda3/envs/opensora/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl rank0: return forward_call(args, **kwargs) rank0: File "/root/miniconda3/envs/opensora/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 610, in forward rank0: return self._conv_forward(input, self.weight, self.bias) rank0: File "/root/miniconda3/envs/opensora/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 605, in _conv_forward rank0: return F.conv3d( rank0: RuntimeError: GET was unable to find an engine to execute this computation

zhengzangw commented 2 weeks ago

Could @ver217 have a look?

aulaywang commented 2 weeks ago

Sorry, I have solved it. Thank you.

ty625911724 commented 2 weeks ago

I meet the same problem. Could you please tell us how to solve this problem? Thanks

aulaywang commented 2 weeks ago

I meet the same problem. Could you please tell us how to solve this problem? Thanks

Actually I solved this problem after I updated the pytorch (there maybe some other bugs but not this bug). I did not totally understand the principle of this problem. You can follow the version like me: torch=2.3.0 torchaudio=2.1.2 torchvision=0.18.0