On ColabPro+ I am getting an error when training on a large (3 hour+) dataset. Fairly sure it isn't a RAM/memory issue, but could be wrong. Here is what I get in the console:
Creating dataset...
This usually takes around 2-3 minutes for each minute of audio
(10 minutes of training audio -> 20-30 minutes)
/bin/bash: line 1: 6256 Killed ddsp_prepare_tfrecord --input_audio_filepatterns=audio/* --output_tfrecord_path=data//train.tfrecord --num_shards=10 --sample_rate=16000 --frame_rate=50 --example_secs=4.0 --hop_secs=1.0 --viterbi=True --center=True &> /dev/null
On ColabPro+ I am getting an error when training on a large (3 hour+) dataset. Fairly sure it isn't a RAM/memory issue, but could be wrong. Here is what I get in the console:
Creating dataset... This usually takes around 2-3 minutes for each minute of audio (10 minutes of training audio -> 20-30 minutes) /bin/bash: line 1: 6256 Killed ddsp_prepare_tfrecord --input_audio_filepatterns=audio/* --output_tfrecord_path=data//train.tfrecord --num_shards=10 --sample_rate=16000 --frame_rate=50 --example_secs=4.0 --hop_secs=1.0 --viterbi=True --center=True &> /dev/null
Training... I0613 13:24:12.203999 139645850212224 ddsp_run.py:179] Restore Dir: /content/gdrive/MyDrive/1/ddsp-training-2022-06-13-1303 I0613 13:24:12.204447 139645850212224 ddsp_run.py:180] Save Dir: /content/gdrive/MyDrive/1/ddsp-training-2022-06-13-1303 I0613 13:24:12.244054 139645850212224 resource_reader.py:50] system_path_file_exists:optimization/base.gin E0613 13:24:12.244385 139645850212224 resource_reader.py:55] Path not found: optimization/base.gin I0613 13:24:12.250916 139645850212224 resource_reader.py:50] system_path_file_exists:eval/basic.gin E0613 13:24:12.251169 139645850212224 resource_reader.py:55] Path not found: eval/basic.gin I0613 13:24:12.350420 139645850212224 ddsp_run.py:152] Operative config not found in /content/gdrive/MyDrive/1/ddsp-training-2022-06-13-1303 I0613 13:24:12.350906 139645850212224 resource_reader.py:50] system_path_file_exists:models/vst/vst.gin E0613 13:24:12.351280 139645850212224 resource_reader.py:55] Path not found: models/vst/vst.gin I0613 13:24:12.357154 139645850212224 resource_reader.py:50] system_path_file_exists:datasets/tfrecord.gin E0613 13:24:12.357398 139645850212224 resource_reader.py:55] Path not found: datasets/tfrecord.gin I0613 13:24:12.358336 139645850212224 resource_reader.py:50] system_path_file_exists:datasets/base.gin E0613 13:24:12.358542 139645850212224 resource_reader.py:55] Path not found: datasets/base.gin I0613 13:24:12.367111 139645850212224 ddsp_run.py:184] Operative Gin Config: import ddsp import ddsp.training as ddsp2