search

acids-ircam / ddsp_pytorch

Implementation of Differentiable Digital Signal Processing (DDSP) in Pytorch
Apache License 2.0
451 stars 56 forks source link

Runtime error when executing train.py #11

Closed moiseshorta closed 3 years ago

moiseshorta commented 3 years ago

Hi,

I've been now attempting to train a custom model. So far the preprocessing works fine.

It is when I start training that I get this error:

Traceback (most recent call last): File "train.py", line 79, in <module> y = model(p, l).squeeze(-1) File "/media/hexorcismos/Archivo/AI/ddsp_pytorch/ddsp_env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "/media/hexorcismos/Archivo/AI/ddsp_pytorch/ddsp/model.py", line 65, in forward ], -1) RuntimeError: Sizes of tensors must match except in dimension 1. Got 375 and 400 (The offending index is 0)

I also noticed that the arguments to pass into 'train.py' are different than the ones mentioned in the Readme.md. For example, there is no --epoch argument but rather a --steps arg. It maybe a good idea to update that section if it had any changes.

Thanks for your great work.

caillonantoine commented 3 years ago

It might be a bug coming from crepe, be sure to install the latest version. At some point I remember that crepe used to output different shapes depending on if it was using CPU or GPU, so if you have a GPU be sure to use it.

moiseshorta commented 3 years ago

I have the latest version of crepe==0.0.11

It seems when executing the preprocessing.py command, crepe falls back to using CPU, instead of GPU...so I will try and fix my GPU libs and take it from there...

UPDATE: I have tried training on a Google Colab instance with GPU enabled and still am running into the error mentioned above. Is there a specific Tensorflow version you are using? Does the audio which gets preprocessed need to be exactly the same samplerate as the YAML file or does the preprocess.py take care of that?

I am getting these messages when executing preprocess.py from Google Colab:

/content/drive/MyDrive/MelGAN_Datasets/aerofonos/chaclan_0.wav:   0% 0/35 [00:00<?, ?it/s]2021-02-24 12:11:50.249116: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
2021-02-24 12:11:51.556711: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-02-24 12:11:51.557774: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-02-24 12:11:51.615644: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-24 12:11:51.616264: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:00:04.0 name: Tesla P100-PCIE-16GB computeCapability: 6.0
coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 15.90GiB deviceMemoryBandwidth: 681.88GiB/s
2021-02-24 12:11:51.616328: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
2021-02-24 12:11:51.871839: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10
2021-02-24 12:11:51.871997: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.10
2021-02-24 12:11:51.988223: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-02-24 12:11:52.027904: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-02-24 12:11:52.293824: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-02-24 12:11:52.345802: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.10
2021-02-24 12:11:52.863359: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.7
2021-02-24 12:11:52.863645: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-24 12:11:52.864488: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-24 12:11:52.868581: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-02-24 12:11:52.869196: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-02-24 12:11:52.869392: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-24 12:11:52.870049: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:00:04.0 name: Tesla P100-PCIE-16GB computeCapability: 6.0
coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 15.90GiB deviceMemoryBandwidth: 681.88GiB/s
2021-02-24 12:11:52.870135: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
2021-02-24 12:11:52.870189: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10
2021-02-24 12:11:52.870221: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.10
2021-02-24 12:11:52.870252: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-02-24 12:11:52.870324: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-02-24 12:11:52.870362: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-02-24 12:11:52.870392: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.10
2021-02-24 12:11:52.870432: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.7
2021-02-24 12:11:52.870546: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-24 12:11:52.871185: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-24 12:11:52.871745: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-02-24 12:11:52.874884: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
2021-02-24 12:12:06.642757: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-02-24 12:12:06.642815: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2021-02-24 12:12:06.642833: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2021-02-24 12:12:06.643329: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-24 12:12:06.644157: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-24 12:12:06.644858: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-24 12:12:06.645406: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2021-02-24 12:12:06.645456: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14449 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0)
/usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
2021-02-24 12:12:09.497983: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2021-02-24 12:12:09.507228: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2300000000 Hz
2021-02-24 12:12:09.750210: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10
2021-02-24 12:12:11.174750: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.7
188/188 [==============================] - 8s 10ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/chaclan_1.wav:   3% 1/35 [00:33<19:06, 33.73s/it]/usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
51/51 [==============================] - 1s 11ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/chaclan_2.wav:   6% 2/35 [00:36<08:38, 15.70s/it]/usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
101/101 [==============================] - 1s 9ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/chaclan_3.wav:   9% 3/35 [00:42<05:55, 11.11s/it]/usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
51/51 [==============================] - 0s 9ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/chaclan_4.wav:  11% 4/35 [00:45<04:07,  7.99s/it]/usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
138/138 [==============================] - 1s 9ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/chaclan_5.wav:  14% 5/35 [00:53<03:52,  7.77s/it]/usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
63/63 [==============================] - 1s 9ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/chaclan_6.wav:  17% 6/35 [00:56<03:04,  6.35s/it]/usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
126/126 [==============================] - 1s 9ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/chaclan_7.wav:  20% 7/35 [01:03<03:04,  6.60s/it]/usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
113/113 [==============================] - 1s 9ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/chaclan_8.wav:  23% 8/35 [01:10<02:57,  6.57s/it]/usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
51/51 [==============================] - 0s 9ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/chaclan_9.wav:  26% 9/35 [01:13<02:21,  5.45s/it]/usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
126/126 [==============================] - 1s 9ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/chaclan_10.wav:  29% 10/35 [01:20<02:29,  5.96s/it]/usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
38/38 [==============================] - 0s 10ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/chaclan2_0.wav:  31% 11/35 [01:22<01:56,  4.87s/it]/usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
76/76 [==============================] - 1s 9ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/chaclan2_1.wav:  34% 12/35 [01:27<01:49,  4.77s/it]/usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
26/26 [==============================] - 0s 9ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/chaclan2_2.wav:  37% 13/35 [01:29<01:24,  3.86s/it]/usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
38/38 [==============================] - 0s 9ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/chaclan2_3.wav:  40% 14/35 [01:31<01:13,  3.48s/it]/usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
63/63 [==============================] - 1s 9ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/ehecatl_0.wav:  43% 15/35 [01:35<01:11,  3.60s/it] /usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
63/63 [==============================] - 1s 9ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/ehecatl_1.wav:  46% 16/35 [01:39<01:10,  3.70s/it]/usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
126/126 [==============================] - 1s 9ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/ehecatl_2.wav:  49% 17/35 [01:46<01:23,  4.61s/it]/usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
101/101 [==============================] - 1s 9ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/ehecatl_3.wav:  51% 18/35 [01:52<01:24,  4.96s/it]/usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
126/126 [==============================] - 1s 9ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/ehecatl_4.wav:  54% 19/35 [01:58<01:29,  5.57s/it]/usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
138/138 [==============================] - 1s 9ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/ehecatl_5.wav:  57% 20/35 [02:07<01:36,  6.40s/it]/usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
126/126 [==============================] - 1s 9ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/ehecatl_6.wav:  60% 21/35 [02:14<01:32,  6.59s/it]/usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
113/113 [==============================] - 1s 9ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/ehecatl_7.wav:  63% 22/35 [02:20<01:24,  6.52s/it]/usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
151/151 [==============================] - 1s 9ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/ag_0.wav:  66% 23/35 [02:28<01:24,  7.03s/it]     /usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
63/63 [==============================] - 1s 9ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/ag_1.wav:  69% 24/35 [02:32<01:06,  6.06s/it]/usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
88/88 [==============================] - 1s 9ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/ag_2.wav:  71% 25/35 [02:37<00:57,  5.72s/it]/usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
76/76 [==============================] - 1s 9ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/ag_3.wav:  74% 26/35 [02:42<00:48,  5.38s/it]/usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
88/88 [==============================] - 1s 9ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/ag_4.wav:  77% 27/35 [02:47<00:42,  5.31s/it]/usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
126/126 [==============================] - 1s 9ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/ag_5.wav:  80% 28/35 [02:54<00:40,  5.84s/it]/usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
113/113 [==============================] - 1s 9ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/ag_6.wav:  83% 29/35 [03:00<00:35,  5.97s/it]/usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
113/113 [==============================] - 1s 9ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/ag_7.wav:  86% 30/35 [03:07<00:30,  6.11s/it]/usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
126/126 [==============================] - 1s 9ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
126/126 [==============================] - 1s 9ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/ag_9.wav:  91% 32/35 [03:21<00:20,  6.80s/it]/usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
151/151 [==============================] - 1s 9ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/ag_10.wav:  94% 33/35 [03:30<00:14,  7.27s/it]/usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
138/138 [==============================] - 1s 9ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/ag_11.wav:  97% 34/35 [03:39<00:07,  7.73s/it]/usr/local/lib/python3.7/dist-packages/crepe/core.py:209: RuntimeWarning: invalid value encountered in true_divide
  frames /= np.std(frames, axis=1)[:, np.newaxis]
76/76 [==============================] - 1s 10ms/step
/usr/local/lib/python3.7/dist-packages/librosa/core/convert.py:1354: RuntimeWarning: divide by zero encountered in log10
  - 0.5 * np.log10(f_sq + const[3])
/content/drive/MyDrive/MelGAN_Datasets/aerofonos/ag_11.wav: 100% 35/35 [03:43<00:00,  6.39s/it]
caillonantoine commented 3 years ago

Can you provide the yaml configuration file used for this training?

moiseshorta commented 3 years ago

.Sure, this was my config.yaml:

data:
  data_location: "/home/hexorcismos/Desktop/AI/MelGAN-VC/Datasets/ddsp_test"
  extension: "wav"

preprocess:
  sampling_rate: &samplingrate 48000
  signal_length: &signallength 192000
  block_size: &blocksize 512

  oneshot: false # crop every audio file to exactly signal length
  out_dir: ./preprocessed/

model:
  hidden_size: 512
  n_harmonic: 64
  n_bands: 65
  sampling_rate: *samplingrate
  block_size: *blocksize

train:
  scales: [4096,2048, 1024, 512, 256, 128]
  overlap: .75
moiseshorta commented 3 years ago

Seems training is now working for me by changing block_size: 512 to block_size: 192 when at sampling_rate: 48000.