Please check whether this paper is about 'Voice Conversion' or not.
article info.
title: StreamVoice+: Evolving into End-to-end Streaming Zero-shot Voice Conversion
summary: StreamVoice has recently pushed the boundaries of zero-shot voice conversion
(VC) in the streaming domain. It uses a streamable language model (LM) with a
context-aware approach to convert semantic features from automatic speech
recognition (ASR) into acoustic features with the desired speaker timbre.
Despite its innovations, StreamVoice faces challenges due to its dependency on
a streaming ASR within a cascaded framework, which complicates system
deployment and optimization, affects VC system's design and performance based
on the choice of ASR, and struggles with conversion stability when faced with
low-quality semantic inputs. To overcome these limitations, we introduce
StreamVoice+, an enhanced LM-based end-to-end streaming framework that operates
independently of streaming ASR. StreamVoice+ integrates a semantic encoder and
a connector with the original StreamVoice framework, now trained using a
non-streaming ASR. This model undergoes a two-stage training process:
initially, the StreamVoice backbone is pre-trained for voice conversion and the
semantic encoder for robust semantic extraction. Subsequently, the system is
fine-tuned end-to-end, incorporating a LoRA matrix to activate comprehensive
streaming functionality. Furthermore, StreamVoice+ mainly introduces two
strategic enhancements to boost conversion quality: a residual compensation
mechanism in the connector to ensure effective semantic transmission and a
self-refinement strategy that leverages pseudo-parallel speech pairs generated
by the conversion backbone to improve speech decoupling. Experiments
demonstrate that StreamVoice+ not only achieves higher naturalness and speaker
similarity in voice conversion than its predecessor but also provides versatile
support for both streaming and non-streaming conversion scenarios.
Please check whether this paper is about 'Voice Conversion' or not.
article info.
title: StreamVoice+: Evolving into End-to-end Streaming Zero-shot Voice Conversion
summary: StreamVoice has recently pushed the boundaries of zero-shot voice conversion (VC) in the streaming domain. It uses a streamable language model (LM) with a context-aware approach to convert semantic features from automatic speech recognition (ASR) into acoustic features with the desired speaker timbre. Despite its innovations, StreamVoice faces challenges due to its dependency on a streaming ASR within a cascaded framework, which complicates system deployment and optimization, affects VC system's design and performance based on the choice of ASR, and struggles with conversion stability when faced with low-quality semantic inputs. To overcome these limitations, we introduce StreamVoice+, an enhanced LM-based end-to-end streaming framework that operates independently of streaming ASR. StreamVoice+ integrates a semantic encoder and a connector with the original StreamVoice framework, now trained using a non-streaming ASR. This model undergoes a two-stage training process: initially, the StreamVoice backbone is pre-trained for voice conversion and the semantic encoder for robust semantic extraction. Subsequently, the system is fine-tuned end-to-end, incorporating a LoRA matrix to activate comprehensive streaming functionality. Furthermore, StreamVoice+ mainly introduces two strategic enhancements to boost conversion quality: a residual compensation mechanism in the connector to ensure effective semantic transmission and a self-refinement strategy that leverages pseudo-parallel speech pairs generated by the conversion backbone to improve speech decoupling. Experiments demonstrate that StreamVoice+ not only achieves higher naturalness and speaker similarity in voice conversion than its predecessor but also provides versatile support for both streaming and non-streaming conversion scenarios.
id: http://arxiv.org/abs/2408.02178v1
judge
Write [vclab::confirmed] or [vclab::excluded] in comment.