search

SayaSS / vits-finetuning

Fine-Tuning your VITS model using a pre-trained model
MIT License
546 stars 86 forks source link

RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR #5

Closed chiakianyka closed 1 year ago

chiakianyka commented 1 year ago

Try to multiple train and it show

/pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [0,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [1,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [2,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [3,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [4,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [5,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [6,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [7,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [8,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [9,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [10,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [11,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [12,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [13,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [14,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [15,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [16,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [17,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [18,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [19,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [20,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [21,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [22,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [23,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [24,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [25,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [26,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [27,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [28,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [29,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [30,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [31,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [32,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [33,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [34,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [35,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [36,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [37,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [38,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [39,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [40,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [41,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [42,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [43,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [44,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [45,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [46,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [47,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [48,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [49,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [50,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [51,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [52,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [53,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [54,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [55,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [56,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [57,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [58,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [59,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [60,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [61,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [62,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [63,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [64,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [65,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [66,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [67,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [68,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [69,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [70,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [71,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [72,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [73,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [74,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [75,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [76,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [77,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [78,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [79,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [80,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [81,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [82,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [83,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [84,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [85,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [86,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [87,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [88,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [89,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [90,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [91,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [92,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [93,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [94,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [95,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [96,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [97,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [98,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [99,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [100,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [101,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [102,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [103,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [104,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [105,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [106,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [107,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [108,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [109,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [110,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [111,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [112,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [113,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [114,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [115,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [116,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [117,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [118,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [119,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [120,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [121,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [122,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [123,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [124,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [125,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [126,0,0] Assertion srcIndex < srcSelectDimSize failed. /pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [59,0,0], thread: [127,0,0] Assertion srcIndex < srcSelectDimSize failed. NCCL error in: /pytorch/torch/lib/c10d/../c10d/NCCLUtils.hpp:69, unhandled cuda error, NCCL version 2.4.8 Traceback (most recent call last): File "train_ms.py", line 305, in main() File "train_ms.py", line 55, in main mp.spawn(run, nprocs=n_gpus, args=(n_gpus, hps,)) File "/usr/local/lib/python3.8/dist-packages/torch/multiprocessing/spawn.py", line 200, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method='spawn') File "/usr/local/lib/python3.8/dist-packages/torch/multiprocessing/spawn.py", line 158, in start_processes while not context.join(): File "/usr/local/lib/python3.8/dist-packages/torch/multiprocessing/spawn.py", line 119, in join raise Exception(msg) Exception:

-- Process 0 terminated with the following error: Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/torch/multiprocessing/spawn.py", line 20, in _wrap fn(i, args) File "/content/vits-finetuning/train_ms.py", line 123, in run train_and_evaluate(rank, epoch, hps, [net_g, net_d], [optim_g, optim_d], [scheduler_g, scheduler_d], scaler, [train_loader, eval_loader], logger, [writer, writer_eval]) File "/content/vits-finetuning/train_ms.py", line 151, in train_and_evaluate (z, z_p, m_p, logs_p, m_q, logs_q) = net_g(x, x_lengths, spec, spec_lengths, speakers) File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(input, kwargs) File "/usr/local/lib/python3.8/dist-packages/torch/nn/parallel/distributed.py", line 511, in forward output = self.module(*inputs[0], *kwargs[0]) File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(input, kwargs) File "/content/vits-finetuning/models.py", line 467, in forward z, m_q, logs_q, y_mask = self.enc_q(y, y_lengths, g=g) File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(*input, kwargs) File "/content/vits-finetuning/models.py", line 236, in forward x = self.pre(x) x_mask File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(input, kwargs) File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/conv.py", line 256, in forward return F.conv1d(input, self.weight, self.bias, self.stride, RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR

How I can fix it thank!

SayaSS commented 1 year ago

Oh, sorry, if you need to customize the "n_speakers", you'll need to use another pre-trained model. I've updated the readme already. This error is caused by an already existing speaker embedding.

chiakianyka commented 1 year ago

Thank a lot for the replay!!