kijai
commented
2 days ago That amount of RAM may not be enough, to avoid running out of memory with the VAE you should enable the tiling option.
Open Fahmie23 opened 2 days ago
kijai
commented
2 days ago That amount of RAM may not be enough, to avoid running out of memory with the VAE you should enable the tiling option.
Fahmie23
commented
1 day ago That amount of RAM may not be enough, to avoid running out of memory with the VAE you should enable the tiling option.
ouhhh, so its mean i need to increase my RAM more? not Vram?
I have a linux machine to do the research and testing for CogVideo. Below are the specifications for the linux machine.
GPU Model: NVIDIA L4 VRAM: 24GB RAM: 16GB
It is enough to run the model?. I got an OOM Error when running just CogVideoX-2B.
ComfyUI Error Report
Error Details
Exception Message: Allocation on device
Stack Trace
2024-11-13 08:45:42,072 - root - INFO - Total VRAM 22491 MB, total RAM 15986 MB 2024-11-13 08:45:42,072 - root - INFO - pytorch version: 2.5.1+cu121 2024-11-13 08:45:42,073 - root - INFO - Set vram state to: NORMAL_VRAM 2024-11-13 08:45:42,073 - root - INFO - Device: cuda:0 NVIDIA L4 : cudaMallocAsync 2024-11-13 08:45:44,450 - root - INFO - Using pytorch cross attention 2024-11-13 08:45:46,596 - root - INFO - [Prompt Server] web root: /home/fahmie/comfyvm/ComfyUI/web 2024-11-13 08:45:48,122 - ComfyUI-CogVideoXWrapper.custom_cogvideox_transformer_3d - INFO - sageattn not found, using sdpa 2024-11-13 08:45:48,140 - ComfyUI-CogVideoXWrapper.cogvideox_fun.transformer_3d - INFO - sageattn not found, using sdpa 2024-11-13 08:45:48,141 - ComfyUI-CogVideoXWrapper.cogvideox_fun.fun_pab_transformer_3d - INFO - sageattn not found, using sdpa 2024-11-13 08:45:48,155 - root - INFO - Total VRAM 22491 MB, total RAM 15986 MB 2024-11-13 08:45:48,155 - root - INFO - pytorch version: 2.5.1+cu121 2024-11-13 08:45:48,155 - root - INFO - Set vram state to: NORMAL_VRAM 2024-11-13 08:45:48,155 - root - INFO - Device: cuda:0 NVIDIA L4 : cudaMallocAsync 2024-11-13 08:45:48,526 - root - INFO - Import times for custom nodes: 2024-11-13 08:45:48,527 - root - INFO - 0.0 seconds: /home/fahmie/comfyvm/ComfyUI/custom_nodes/websocket_image_save.py 2024-11-13 08:45:48,528 - root - INFO - 0.0 seconds: /home/fahmie/comfyvm/ComfyUI/custom_nodes/ComfyUI-KJNodes 2024-11-13 08:45:48,528 - root - INFO - 0.0 seconds: /home/fahmie/comfyvm/ComfyUI/custom_nodes/rgthree-comfy 2024-11-13 08:45:48,528 - root - INFO - 0.1 seconds: /home/fahmie/comfyvm/ComfyUI/custom_nodes/ComfyUI-Manager 2024-11-13 08:45:48,529 - root - INFO - 0.3 seconds: /home/fahmie/comfyvm/ComfyUI/custom_nodes/ComfyUI-VideoHelperSuite 2024-11-13 08:45:48,529 - root - INFO - 0.7 seconds: /home/fahmie/comfyvm/ComfyUI/custom_nodes/ComfyUI-CogVideoXWrapper 2024-11-13 08:45:48,529 - root - INFO - 2024-11-13 08:45:48,542 - root - INFO - Starting server
2024-11-13 08:45:48,544 - root - INFO - To see the GUI go to: http://10.148.0.13:8188 2024-11-13 08:48:14,328 - root - INFO - got prompt 2024-11-13 08:48:14,959 - root - INFO - Requested to load SD3ClipModel_ 2024-11-13 08:48:14,960 - root - INFO - Loading 1 new model 2024-11-13 08:48:14,974 - root - INFO - loaded completely 0.0 4541.693359375 True 2024-11-13 08:49:17,174 - ComfyUI-CogVideoXWrapper.pipeline_cogvideox - INFO - Temporal tiling and context schedule disabled 2024-11-13 08:49:23,326 - root - ERROR - !!! Exception during processing !!! backend='inductor' raised: CalledProcessError: Command '['/usr/bin/gcc', '/tmp/tmpclz_9v3o/main.c', '-O3', '-shared', '-fPIC', '-o', '/tmp/tmpclz_9v3o/cuda_utils.cpython-310-x86_64-linux-gnu.so', '-lcuda', '-L/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/triton/backends/nvidia/lib', '-L/lib/x86_64-linux-gnu', '-I/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/triton/backends/nvidia/include', '-I/tmp/tmpclz_9v3o', '-I/usr/include/python3.10']' returned non-zero exit status 1.
Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
You can suppress this exception and fall back to eager by setting: import torch._dynamo torch._dynamo.config.suppress_errors = True
2024-11-13 08:49:23,331 - root - ERROR - Traceback (most recent call last): File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1446, in _call_user_compiler compiled_fn = compiler_fn(gm, self.example_inputs()) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_dynamo/repro/after_dynamo.py", line 129, in call compiled_gm = compiler_fn(gm, example_inputs) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/init.py", line 2234, in call return compilefx(model, inputs_, config_patches=self.config) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1521, in compile_fx return aot_autograd( File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 72, in call cg = aot_module_simplified(gm, example_inputs, self.kwargs) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1071, in aot_module_simplified compiled_fn = dispatch_and_compile() File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1056, in dispatch_and_compile compiledfn, = create_aot_dispatcher_function( File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 522, in create_aot_dispatcher_function return _create_aot_dispatcher_function( File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 759, in _create_aot_dispatcher_function compiled_fn, fw_metadata = compiler_fn( File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py", line 179, in aot_dispatch_base compiled_fw = compiler(fw_module, updated_flat_args) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1350, in fw_compiler_base return _fw_compiler_base(model, example_inputs, is_inference) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1421, in _fw_compiler_base return inner_compile( File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 475, in compile_fx_inner return wrap_compiler_debug(_compile_fx_inner, compiler_name="inductor")( File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_dynamo/repro/after_aot.py", line 85, in debug_wrapper inner_compiled_fn = compiler_fn(gm, example_inputs) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 661, in _compile_fx_inner compiled_graph = FxGraphCache.load( File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1334, in load compiled_graph = compile_fx_fn( File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 570, in codegen_and_compile compiled_graph = fx_codegen_and_compile(gm, example_inputs, fx_kwargs) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 878, in fx_codegen_and_compile compiled_fn = graph.compile_to_fn() File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_inductor/graph.py", line 1913, in compile_to_fn return self.compile_to_module().call File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_inductor/graph.py", line 1839, in compile_to_module return self._compile_to_module() File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_inductor/graph.py", line 1845, in _compile_to_module self.codegen_with_cpp_wrapper() if self.cpp_wrapper else self.codegen() File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_inductor/graph.py", line 1784, in codegen self.scheduler.codegen() File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_inductor/scheduler.py", line 3383, in codegen return self._codegen() File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_inductor/scheduler.py", line 3461, in _codegen self.get_backend(device).codegen_node(node) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_inductor/codegen/cuda_combined_scheduling.py", line 80, in codegen_node return self._triton_scheduling.codegen_node(node) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_inductor/codegen/simd.py", line 1155, in codegen_node return self.codegen_node_schedule(node_schedule, buf_accesses, numel, rnumel) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_inductor/codegen/simd.py", line 1364, in codegen_node_schedule src_code = kernel.codegen_kernel() File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_inductor/codegen/triton.py", line 2661, in codegen_kernel **self.inductor_meta_common(), File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_inductor/codegen/triton.py", line 2532, in inductor_meta_common "backend_hash": torch.utils._triton.triton_hash_with_backend(), File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/utils/_triton.py", line 53, in triton_hash_with_backend backend = triton_backend() File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/utils/_triton.py", line 45, in triton_backend target = driver.active.get_current_target() File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/triton/runtime/driver.py", line 23, in getattr self._initialize_obj() File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/triton/runtime/driver.py", line 20, in _initialize_obj self._obj = self._init_fn() File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/triton/runtime/driver.py", line 9, in _create_driver return actives[0]() File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/triton/backends/nvidia/driver.py", line 371, in init self.utils = CudaUtils() # TODO: make static File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/triton/backends/nvidia/driver.py", line 80, in init mod = compile_module_from_src(Path(os.path.join(dirname, "driver.c")).read_text(), "cuda_utils") File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/triton/backends/nvidia/driver.py", line 57, in compile_module_from_src so = _build(name, src_path, tmpdir, library_dirs(), include_dir, libraries) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/triton/runtime/build.py", line 48, in _build ret = subprocess.check_call(cc_cmd) File "/usr/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['/usr/bin/gcc', '/tmp/tmpclz_9v3o/main.c', '-O3', '-shared', '-fPIC', '-o', '/tmp/tmpclz_9v3o/cuda_utils.cpython-310-x86_64-linux-gnu.so', '-lcuda', '-L/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/triton/backends/nvidia/lib', '-L/lib/x86_64-linux-gnu', '-I/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/triton/backends/nvidia/include', '-I/tmp/tmpclz_9v3o', '-I/usr/include/python3.10']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/home/fahmie/comfyvm/ComfyUI/execution.py", line 323, in execute output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb) File "/home/fahmie/comfyvm/ComfyUI/execution.py", line 198, in get_output_data return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb) File "/home/fahmie/comfyvm/ComfyUI/execution.py", line 169, in _map_node_over_list process_inputs(input_dict, i) File "/home/fahmie/comfyvm/ComfyUI/execution.py", line 158, in process_inputs results.append(getattr(obj, func)(inputs)) File "/home/fahmie/comfyvm/ComfyUI/custom_nodes/ComfyUI-CogVideoXWrapper/nodes.py", line 866, in process latents = pipeline["pipe"]( File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, *kwargs) File "/home/fahmie/comfyvm/ComfyUI/custom_nodes/ComfyUI-CogVideoXWrapper/pipeline_cogvideox.py", line 875, in call noise_pred = self.transformer( File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, kwargs) File "/home/fahmie/comfyvm/ComfyUI/custom_nodes/ComfyUI-CogVideoXWrapper/custom_cogvideox_transformer_3d.py", line 723, in forward hidden_states, encoder_hidden_states = block( File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(args, kwargs) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 465, in _fn return fn(*args, kwargs) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(args, kwargs) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1269, in call return self._torchdynamo_orig_callable( File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1064, in call result = self._inner_convert( File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 526, in call return _compile( File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 924, in _compile guarded_code = compile_inner(code, one_graph, hooks, transform) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 666, in compile_inner return _compile_inner(code, one_graph, hooks, transform) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_utils_internal.py", line 87, in wrapper_function return function(*args, *kwargs) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 699, in _compile_inner out_code = transform_code_object(code, transform) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1322, in transform_code_object transformations(instructions, code_options) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 219, in _fn return fn(args, **kwargs) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 634, in transform tracer.run() File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2796, in run super().run() File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 983, in run while self.step(): File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 895, in step self.dispatch_table[inst.opcode](self, inst) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 580, in wrapper return handle_graph_break(self, inst, speculation.reason) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 649, in handle_graph_break self.output.compile_subgraph(self, reason=reason) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1142, in compile_subgraph self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1369, in compile_and_call_fx_graph compiled_fn = self.call_user_compiler(gm) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1416, in call_user_compiler return self._call_user_compiler(gm) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1465, in _call_user_compiler raise BackendCompilerFailed(self.compiler_fn, e) from e torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised: CalledProcessError: Command '['/usr/bin/gcc', '/tmp/tmpclz_9v3o/main.c', '-O3', '-shared', '-fPIC', '-o', '/tmp/tmpclz_9v3o/cuda_utils.cpython-310-x86_64-linux-gnu.so', '-lcuda', '-L/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/triton/backends/nvidia/lib', '-L/lib/x86_64-linux-gnu', '-I/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/triton/backends/nvidia/include', '-I/tmp/tmpclz_9v3o', '-I/usr/include/python3.10']' returned non-zero exit status 1.
Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
You can suppress this exception and fall back to eager by setting: import torch._dynamo torch._dynamo.config.suppress_errors = True
2024-11-13 08:49:23,333 - root - INFO - Prompt executed in 68.99 seconds 2024-11-13 08:49:47,912 - root - INFO - got prompt 2024-11-13 08:50:11,802 - ComfyUI-CogVideoXWrapper.pipeline_cogvideox - INFO - Temporal tiling and context schedule disabled 2024-11-13 08:55:25,214 - root - ERROR - !!! Exception during processing !!! Allocation on device 2024-11-13 08:55:25,221 - root - ERROR - Traceback (most recent call last): File "/home/fahmie/comfyvm/ComfyUI/execution.py", line 323, in execute output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb) File "/home/fahmie/comfyvm/ComfyUI/execution.py", line 198, in get_output_data return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb) File "/home/fahmie/comfyvm/ComfyUI/execution.py", line 169, in _map_node_over_list process_inputs(input_dict, i) File "/home/fahmie/comfyvm/ComfyUI/execution.py", line 158, in process_inputs results.append(getattr(obj, func)(inputs)) File "/home/fahmie/comfyvm/ComfyUI/custom_nodes/ComfyUI-CogVideoXWrapper/nodes.py", line 987, in decode frames = vae.decode(latents).sample File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/diffusers/utils/accelerate_utils.py", line 46, in wrapper return method(self, *args, *kwargs) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/diffusers/models/autoencoders/autoencoder_kl_cogvideox.py", line 1278, in decode decoded = self._decode(z).sample File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/diffusers/models/autoencoders/autoencoder_kl_cogvideox.py", line 1249, in _decode z_intermediate, conv_cache = self.decoder(z_intermediate, conv_cache=conv_cache) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, kwargs) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/diffusers/models/autoencoders/autoencoder_kl_cogvideox.py", line 970, in forward hidden_states, new_conv_cache[conv_cache_key] = up_block( File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(args, kwargs) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/diffusers/models/autoencoders/autoencoder_kl_cogvideox.py", line 647, in forward hidden_states, new_conv_cache[conv_cache_key] = resnet( File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, *kwargs) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/diffusers/models/autoencoders/autoencoder_kl_cogvideox.py", line 291, in forward hidden_states, new_conv_cache["norm1"] = self.norm1(hidden_states, zq, conv_cache=conv_cache.get("norm1")) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, *kwargs) File "/home/fahmie/comfyvm/ComfyUI/venv/lib/python3.10/site-packages/diffusers/models/autoencoders/autoencoder_kl_cogvideox.py", line 187, in forward new_f = norm_f conv_y + conv_b torch.OutOfMemoryError: Allocation on device
2024-11-13 08:55:25,222 - root - ERROR - Got an OOM, unloading all loaded models. 2024-11-13 08:55:25,437 - root - INFO - Prompt executed in 337.51 seconds
{"last_node_id":34,"last_link_id":64,"nodes":[{"id":30,"type":"CogVideoTextEncode","pos":{"0":500,"1":308},"size":{"0":413.2752380371094,"1":248.78025817871094},"flags":{},"order":2,"mode":0,"inputs":[{"name":"clip","type":"CLIP","link":54}],"outputs":[{"name":"conditioning","type":"CONDITIONING","links":[61],"slot_index":0,"shape":3},{"name":"clip","type":"CLIP","links":null}],"properties":{"Node name for S&R":"CogVideoTextEncode"},"widgets_values":["A golden retriever, sporting sleek black sunglasses, with its lengthy fur flowing in the breeze, sprints playfully across a rooftop terrace, recently refreshed by a light rain. The scene unfolds from a distance, the dog's energetic bounds growing larger as it approaches the camera, its tail wagging with unrestrained joy, while droplets of water glisten on the concrete behind it. The overcast sky provides a dramatic backdrop, emphasizing the vibrant golden coat of the canine as it dashes towards the viewer.\n\n",1,true]},{"id":31,"type":"CogVideoTextEncode","pos":{"0":501,"1":598},"size":{"0":411.2016296386719,"1":244.37738037109375},"flags":{},"order":3,"mode":0,"inputs":[{"name":"clip","type":"CLIP","link":56}],"outputs":[{"name":"conditioning","type":"CONDITIONING","links":[62],"slot_index":0,"shape":3},{"name":"clip","type":"CLIP","links":null}],"properties":{"Node name for S&R":"CogVideoTextEncode"},"widgets_values":["blur, watermark. ",1,true]},{"id":33,"type":"VHS_VideoCombine","pos":{"0":1320,"1":204},"size":[504.50164794921875,310],"flags":{},"order":6,"mode":0,"inputs":[{"name":"images","type":"IMAGE","link":59},{"name":"audio","type":"AUDIO","link":null,"shape":7},{"name":"meta_batch","type":"VHS_BatchManager","link":null,"shape":7},{"name":"vae","type":"VAE","link":null,"shape":7}],"outputs":[{"name":"Filenames","type":"VHS_FILENAMES","links":null,"shape":3}],"properties":{"Node name for S&R":"VHS_VideoCombine"},"widgets_values":{"frame_rate":8,"loop_count":0,"filename_prefix":"CogVideoX5B","format":"video/h264-mp4","pix_fmt":"yuv420p","crf":19,"save_metadata":true,"pingpong":false,"save_output":false,"videopreview":{"hidden":false,"paused":false,"params":{"filename":"CogVideoX5B_00002.mp4","subfolder":"","type":"temp","format":"video/h264-mp4","frame_rate":8},"muted":false}}},{"id":20,"type":"CLIPLoader","pos":{"0":101,"1":399},"size":{"0":352.55279541015625,"1":82},"flags":{},"order":0,"mode":0,"inputs":[],"outputs":[{"name":"CLIP","type":"CLIP","links":[54,56],"slot_index":0,"shape":3}],"properties":{"Node name for S&R":"CLIPLoader"},"widgets_values":["t5xxl_fp8_e4m3fn.safetensors","sd3"]},{"id":34,"type":"CogVideoSampler","pos":{"0":969,"1":113},"size":{"0":315.8404846191406,"1":410},"flags":{},"order":4,"mode":0,"inputs":[{"name":"pipeline","type":"COGVIDEOPIPE","link":60},{"name":"positive","type":"CONDITIONING","link":61},{"name":"negative","type":"CONDITIONING","link":62},{"name":"samples","type":"LATENT","link":null,"shape":7},{"name":"image_cond_latents","type":"LATENT","link":null,"shape":7},{"name":"context_options","type":"COGCONTEXT","link":null,"shape":7},{"name":"controlnet","type":"COGVIDECONTROLNET","link":null,"shape":7},{"name":"tora_trajectory","type":"TORAFEATURES","link":null,"shape":7},{"name":"fastercache","type":"FASTERCACHEARGS","link":null,"shape":7}],"outputs":[{"name":"cogvideo_pipe","type":"COGVIDEOPIPE","links":[63],"shape":3},{"name":"samples","type":"LATENT","links":[64],"shape":3}],"properties":{"Node name for S&R":"CogVideoSampler"},"widgets_values":[480,720,49,50,6,806286757407563,"fixed","CogVideoXDDIM",1]},{"id":11,"type":"CogVideoDecode","pos":{"0":981,"1":623},"size":{"0":300.396484375,"1":198},"flags":{},"order":5,"mode":0,"inputs":[{"name":"pipeline","type":"COGVIDEOPIPE","link":63},{"name":"samples","type":"LATENT","link":64}],"outputs":[{"name":"images","type":"IMAGE","links":[59],"slot_index":0,"shape":3}],"properties":{"Node name for S&R":"CogVideoDecode"},"widgets_values":[false,120,360,0.2,0.2,true]},{"id":1,"type":"DownloadAndLoadCogVideoModel","pos":{"0":594,"1":42},"size":{"0":315,"1":214},"flags":{},"order":1,"mode":0,"inputs":[{"name":"pab_config","type":"PAB_CONFIG","link":null,"shape":7},{"name":"block_edit","type":"TRANSFORMERBLOCKS","link":null,"shape":7},{"name":"lora","type":"COGLORA","link":null,"shape":7},{"name":"compile_args","type":"COMPILEARGS","link":null,"shape":7}],"outputs":[{"name":"cogvideo_pipe","type":"COGVIDEOPIPE","links":[60],"slot_index":0,"shape":3}],"properties":{"Node name for S&R":"DownloadAndLoadCogVideoModel"},"widgets_values":["THUDM/CogVideoX-2b","fp16","disabled","disabled",false]}],"links":[[54,20,0,30,0,"CLIP"],[56,20,0,31,0,"CLIP"],[59,11,0,33,0,"IMAGE"],[60,1,0,34,0,"COGVIDEOPIPE"],[61,30,0,34,1,"CONDITIONING"],[62,31,0,34,2,"CONDITIONING"],[63,34,0,11,0,"COGVIDEOPIPE"],[64,34,1,11,1,"LATENT"]],"groups":[],"config":{},"extra":{"ds":{"scale":0.6830134553650705,"offset":[-157.53971682235178,-5.75114406010195]}},"version":0.4}