Closed sjoon2455 closed 1 year ago
PAD
@christallire Thanks, do you know where in the code does that?
For ar stage, there is no audio prompt needed. The speech part will be regarded as a Auto-regressive Task and you can refer to paper's description of this design.
In the AR model, we do not explicitly extract an audio clip as the prompt in training. The training process is pure casual language model training. In this way, any prefix sequence c<t,1 is treated as a prompt for the latter part of the sequence c≥t,1.
For nar stage, if you use the default setup with prefix mode == 1, audio prompt will be seleceted from two candidates. One is a audio clip with length sampled from range of 1/4 orignal audio length to 1/2 audio length. Another is 3 sec audio clip from start. The shorter one will be used. You can refer to code here. https://github.com/lifeiteng/vall-e/blob/e38ac9d816412317222b7f0f59198b9124377da2/valle/models/valle.py#L348-L350
@keshawnhsieh Thanks you so much, I really appreciate your explanation.
Hi, I have a question about training stage.
As I understand, Vall-e cuts a given audio for about 3 seconds (if possible) and have it act as an audio prompt (with is aligned phoneme)
What if a given audio is less than 3 seconds long?