search

williamyang1991 / FRESCO

[CVPR 2024] FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation
https://www.mmlab-ntu.com/project/fresco/
Other
730 stars 71 forks source link

Got RuntimeError:: element 0 of tensors does not require grad and does not have a grad_fn #31

Open baomuhedao opened 7 months ago

baomuhedao commented 7 months ago

video to video translation... processing batch [1/7] with 8 frames FutureWarning: _encode_prompt() is deprecated and it will be removed in a future version. Use encode_prompt() instead. Also, be aware that the output format changed from a concatenated tensor to a tuple.

0%| | 0/15 [00:00<?, ?it/s] 0%| | 0/15 [00:00<?, ?it/s] !!! Exception during processing !!! Traceback (most recent call last): File "/data/comfy-ui/execution.py", line 151, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) File "/data/comfy-ui/execution.py", line 81, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) File "/data/comfy-ui/execution.py", line 74, in map_node_over_list results.append(getattr(obj, func)(slice_dict(input_data_all, i))) File "/data/comfy-ui/custom_nodes/comfyui-cartoon-stylization/nodes.py", line 68, in stylize FrescoService().stylize(video_file_path,output_dir,prompt) File "/data/comfy-ui/custom_nodes/comfyui-cartoon-stylization/fresco_service.py", line 39, in stylize keys = run_keyframe_translation(config) File "/data/comfy-ui/custom_nodes/comfyui-cartoon-stylization/fresco/run_fresco.py", line 239, in run_keyframe_translation latents = inference(pipe, controlnet, frescoProc, File "/data/comfy-ui/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, *kwargs) File "/data/comfy-ui/custom_nodes/comfyui-cartoon-stylization/fresco/src/pipe_FRESCO.py", line 202, in inference noise_pred = pipe.unet( File "/data/comfy-ui/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/data/comfy-ui/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, kwargs) File "/data/comfy-ui/custom_nodes/comfyui-cartoon-stylization/fresco/src/diffusion_hacked.py", line 778, in forward sample = optimize_feature(sample, flows, occs, correlation_matrix, File "/data/comfy-ui/custom_nodes/comfyui-cartoon-stylization/fresco/src/diffusion_hacked.py", line 487, in optimize_feature optimizer.step(closure) File "/data/comfy-ui/venv/lib/python3.10/site-packages/torch/optim/optimizer.py", line 373, in wrapper out = func(*args, *kwargs) File "/data/comfy-ui/venv/lib/python3.10/site-packages/torch/optim/optimizer.py", line 76, in _use_grad ret = func(self, args, kwargs) File "/data/comfy-ui/venv/lib/python3.10/site-packages/torch/optim/adam.py", line 143, in step loss = closure() File "/data/comfy-ui/custom_nodes/comfyui-cartoon-stylization/fresco/src/diffusion_hacked.py", line 480, in closure loss.backward() File "/data/comfy-ui/venv/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward torch.autograd.backward( File "/data/comfy-ui/venv/lib/python3.10/site-packages/torch/autograd/init.py", line 251, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

williamyang1991 commented 7 months ago

Do you use the suggested diffusers version (diffusers 0.19.3)? My code didn't report the warning

FutureWarning: _encode_prompt() is deprecated and it will be removed in a future version. Use encode_prompt() instead. Also, be aware that the output format changed from a concatenated tensor to a tuple.