Closed shanhaidexiamo closed 1 year ago
@shanhaidexiamo
re: 1 - i actually wasn't too sure what the deletion pre-training task was - is it like you described, or was it to predict the tokens that were missing from the source sequence?
i thought it was, for a sentence like, "cat chase a mouse", "cat mouse" -> "chase a". but you are saying it should be "cat mouse" -> "cat chase a mouse" ?
i thought it was, for a sentence like, "cat chase a mouse", "cat mouse" -> "chase a". but you are saying it should be "cat mouse" -> "cat chase a mouse" ?
Yes. But I'm not sure which method is more useful, it seems that the latter can fill in scattered tokens into complete sentence tokens
@shanhaidexiamo i'm not caught up enough on pretraining literature to know
maybe i can allow for both options?
@shanhaidexiamo ok, just made it an option https://github.com/lucidrains/spear-tts-pytorch/commit/ae9cad1309d698d0f83547ce2fa48a6a7fded1d5
@shanhaidexiamo ok, just made it an option ae9cad1
Actually I learned it from the model figure, the pre-train model use <corrupt(speech), speech> as the input and label, so I think the author just corrupt the input.
yeah they did, but there are many ways of corrupting
they apparently chose the 'deletion' technique with 60% deleted. or rather, said that worked the most effectively
yeah they did, but there are many ways of corrupting
they apparently chose the 'deletion' technique with 60% deleted. or rather, said that worked the most effectively
Alright. I will try both methods, and I am not sure which one works better when it comes to training text to semantic tokens.
yea, we should figure out how the deletion pretraining is done so to test their claim. anyhow, let's start with these two options
Thanks for your contribution to spear-tts. I have some questions about code.
source = rearrange(x[~delete_mask], '(b n) -> b n', b = batch) target = rearrange(x[delete_mask], '(b n) -> b n', b = batch)
to_text_logit.weight = to_text_logit.weight
Thank you