Hi there, first of all, I'm thankful for this code.
I'm a beginner and trying to run it. With the parallelization implemented in datasets/preprocessor.py, I'm getting this error:
BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
Can somebody please convert this code to serial implementation:
executor = ProcessPoolExecutor(max_workers=n_jobs)
futures = []
index = 1
for input_dir in input_dirs:
with open(os.path.join(input_dir, 'metadata.csv'), encoding='utf-8') as f:
for line in f:
parts = line.strip().split('|')
basename = parts[0]
wav_path = os.path.join(input_dir, 'wavs', '{}.wav'.format(basename))
text = parts[2]
futures.append(executor.submit(partial(_process_utterance, mel_dir, linear_dir, wav_dir, basename, wav_path, text, hparams)))
index += 1
return [future.result() for future in tqdm(futures) if future.result() is not None]
I understood that i. __process_utterance (out_dir, index, wav_path, text) needs to be called for every input. But I couldn't yet understand how to modify this statement:
return [future.result() for future in tqdm(futures) if future.result() is not None]
79168089
commented
1 year ago
Hi there, first of all, I'm thankful for this code.
I'm a beginner and trying to run it. With the parallelization implemented in datasets/preprocessor.py, I'm getting this error: BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
Can somebody please convert this code to serial implementation:
executor = ProcessPoolExecutor(max_workers=n_jobs) futures = [] index = 1 for input_dir in input_dirs: with open(os.path.join(input_dir, 'metadata.csv'), encoding='utf-8') as f: for line in f: parts = line.strip().split('|') basename = parts[0] wav_path = os.path.join(input_dir, 'wavs', '{}.wav'.format(basename)) text = parts[2] futures.append(executor.submit(partial(_process_utterance, mel_dir, linear_dir, wav_dir, basename, wav_path, text, hparams))) index += 1 return [future.result() for future in tqdm(futures) if future.result() is not None]
I understood that i. __process_utterance (out_dir, index, wav_path, text) needs to be called for every input. But I couldn't yet understand how to modify this statement:
return [future.result() for future in tqdm(futures) if future.result() is not None]