search

2noise / ChatTTS

A generative speech model for daily dialogue.
https://2noise.com
Other
27.43k stars 2.99k forks source link

What are the errors mean，Does my GPU work or not？CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR: #301

Open thomashooo opened 4 weeks ago

thomashooo commented 4 weeks ago

RTX 3080Ti Laptop NVCC -V：12.4 nvdia-smi：12.2

W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_dynamo\bytecode_transformation.py", line 1036, in transform_code_object W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] transformations(instructions, code_options) W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_dynamo\convert_frame.py", line 165, in _fn W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] return fn(*args, kwargs) W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_dynamo\convert_frame.py", line 500, in transform W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] tracer.run() W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_dynamo\symbolic_convert.py", line 2149, in run W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] super().run() W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_dynamo\symbolic_convert.py", line 810, in run W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] and self.step() W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_dynamo\symbolic_convert.py", line 773, in step W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] getattr(self, inst.opname)(inst) W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_dynamo\symbolic_convert.py", line 2268, in RETURN_VALUE W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] self.output.compile_subgraph( W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_dynamo\output_graph.py", line 971, in compile_subgraph W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root) W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\contextlib.py", line 81, in inner W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] return func(*args, *kwds) W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_dynamo\output_graph.py", line 1168, in compile_and_call_fx_graph W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] compiled_fn = self.call_user_compiler(gm) W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_dynamo\utils.py", line 262, in time_wrapper W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] r = func(args, kwargs) W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_dynamo\output_graph.py", line 1241, in call_user_compiler W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] raise BackendCompilerFailed(self.compiler_fn, e).with_traceback( W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_dynamo\output_graph.py", line 1222, in call_user_compiler W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] compiled_fn = compiler_fn(gm, self.example_inputs()) W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_dynamo\repro\after_dynamo.py", line 117, in debug_wrapper W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] compiled_gm = compiler_fn(gm, example_inputs) W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch__init.py", line 1729, in call W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] return compilefx(model, inputs_, config_patches=self.config) W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\contextlib.py", line 81, in inner W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] return func(args, kwds) W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_inductor\compile_fx.py", line 1330, in compile_fx W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] return aot_autograd( W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_dynamo\backends\common.py", line 58, in compiler_fn W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] cg = aot_module_simplified(gm, example_inputs, kwargs) W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_functorch\aot_autograd.py", line 903, in aot_module_simplified W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] compiled_fn = create_aot_dispatcher_function( W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_dynamo\utils.py", line 262, in time_wrapper W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] r = func(args, kwargs) W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_functorch\aot_autograd.py", line 628, in create_aot_dispatcher_function W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config, fw_metadata=fw_metadata) W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_functorch_aot_autograd\runtime_wrappers.py", line 443, in aot_wrapper_dedupe W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] return compiler_fn(flat_fn, leaf_flat_args, aot_config, fw_metadata=fw_metadata) W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_functorch_aot_autograd\runtime_wrappers.py", line 648, in aot_wrapper_synthetic_base W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] return compiler_fn(flat_fn, flat_args, aot_config, fw_metadata=fw_metadata) W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_functorch_aot_autograd\jit_compile_runtime_wrappers.py", line 119, in aot_dispatch_base W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] compiled_fw = compiler(fw_module, updated_flat_args) W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_dynamo\utils.py", line 262, in time_wrapper W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] r = func(*args, *kwargs) W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_inductor\compile_fx.py", line 1257, in fw_compiler_base W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] return inner_compile( W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_dynamo\repro\after_aot.py", line 83, in debug_wrapper W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] inner_compiled_fn = compiler_fn(gm, example_inputs) W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_inductor\debug.py", line 304, in inner W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] return fn(args, kwargs) W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\contextlib.py", line 81, in inner W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] return func(*args, kwds) W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\contextlib.py", line 81, in inner W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] return func(*args, *kwds) W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_dynamo\utils.py", line 262, in time_wrapper W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] r = func(args, kwargs) W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_inductor\compile_fx.py", line 438, in compile_fx_inner W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] compiled_graph = fx_codegen_and_compile( W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_inductor\compile_fx.py", line 714, in fx_codegen_and_compile W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] compiled_fn = graph.compile_to_fn() W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_inductor\graph.py", line 1307, in compile_to_fn W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] return self.compile_to_module().call W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_dynamo\utils.py", line 262, in time_wrapper W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] r = func(*args, *kwargs) W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_inductor\graph.py", line 1250, in compile_to_module W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] self.codegen_with_cpp_wrapper() if self.cpp_wrapper else self.codegen() W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_inductor\graph.py", line 1205, in codegen W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] self.scheduler = Scheduler(self.buffers) W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_dynamo\utils.py", line 262, in time_wrapper W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] r = func(args, **kwargs) W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_inductor\scheduler.py", line 1267, in init W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] self.nodes = [self.create_scheduler_node(n) for n in nodes] W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_inductor\scheduler.py", line 1267, in W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] self.nodes = [self.create_scheduler_node(n) for n in nodes] W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_inductor\scheduler.py", line 1358, in create_scheduler_node W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] return SchedulerNode(self, node) W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_inductor\scheduler.py", line 687, in init__ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] self._compute_attrs() W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_inductor\scheduler.py", line 698, in _compute_attrs W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] group_fn = self.scheduler.get_backend(self.node.get_device()).group_fn W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_inductor\scheduler.py", line 2276, in get_backend W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] self.backends[device] = self.create_backend(device) W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] File "C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch_inductor\scheduler.py", line 2268, in create_backend W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] raise RuntimeError( W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised: W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] RuntimeError: Cannot find a working triton installation. More information on installing Triton can be found at https://github.com/openai/triton W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information W0611 21:28:26.564000 5848 torch_dynamo\convert_frame.py:824] 15%|███████████▋ | 302/2048 [00:41<03:59, 7.30it/s] C:\ProgramData\miniconda3\envs\tts\Lib\site-packages\torch\nn\modules\conv.py:306: UserWarning: Plan failed with a cudnnException: CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR: cudnnFinalize Descriptor Failed cudnn_status: CUDNN_STATUS_NOT_SUPPORTED (Triggered internally at ..\aten\src\ATen\native\cudnn\Conv_v8.cpp:919.) return F.conv1d(input, weight, bias, self.stride,

tankxiaodi commented 2 weeks ago

In gpt.py, include:

import torch._dynamo
torch._dynamo.config.suppress_errors = True