radekosmulski
commented
4 years ago Thank you very much @JoseMMontoro for this, really appreciate your note 🙂
- This is my most general question - I'd like to understand if, at this point, our most immediate goal is to purely reproduce the model that the authors of the paper trained. Is that the case? If so, where are the complications that you have pointed out coming from? Is it because their model architecture, the training steps and the specific training data is not completely defined in the paper? I.e. are there aspects of the training process we're not sure how the authors implemented them?
Our immediate goal is to purely reproduce the results. The difficulties can be due to either there being a bug somewhere in our code or the paper not being thorough in describing the architecture, nor how the model was trained.
Based on literature review, this is the only paper that discusses (and succeeds at) creating semantic embeddings from audio. The results are extremely good and so I would like to do my best to reproduce the results and be able to abstract the method, so that we can start applying it to other datasets.
2. In particular, do we know what the specific 500 hours of audio they use for training are? Which subset of data (if any) did you choose from all Librispeech audio data for training?
From my communication with one of the authors via email, it was shared with us that train-clean-100 and train-clean-360 subsets of Librispeech were used.
3. How is the representation of the audio features created? I feel like that's a crucial part of the process, and there's very little mention of it in the paper. They mentioned they use MFCCs but don't go too deep into how it was implemented, how they evaluated the resulting representation (is it vectors?), etc. What does the sequence of acoustic features used for training look like? Can you expand on this aspect?
I agree this is one of the components of the pipeline that warrants taking a deeper look at. We extract the MFCC features using this off the shelf package. From reading papers in this space, we observed that the authors use consistently the same parameters for generating the representation. Interestingly, this method of generating MFCC features is the default one supported by the referenced package.
Below I am including an example of a representation (first 20 temporal steps). On the vertical axis, we have the 13 MFCC features. On the horizontal axis, we have 20 initial temporal steps (the examples I am training on now have 69 temporal steps max, and if an utterance is shorter, it gets zero padded).
4. How are the different "acoustic realizations" of each word (meaning: different ways of pronouncing a word) averaged? Where in the code does this happen?
The code in this section does the averaging. We first create a dataset with unique utterances in the dataset (any word can have multiple utterances). We then run the encoder on each of the utterances to produce embeddings. We group embeddings by corresponding word and take their mean.
5. Are we able to measure the variance between each realization of the same word present in the dataset, same way the authors did in 3.5?
We don't have it implemented yet. I agree this would be super interesting to do - I didn't work on this as there are other ways we leverage to verify the quality of trained embeddings. Essentially, we run pretrained speech2vec embeddings (that the authors of the paper share on github) along with ours and compare the results.
6. I'm not sure what you refer to by "word pairs" on item 6 in #6. My suggestion was to train embeddings from Librispeech text data, using the word2vec architecture, and compare to the embeddings trained on audio. The authors of the paper did exactly that to compare performance (I was very surprised to see that the embeddings trained on audio perform better on the intrinsic benchmarks! That's very exciting), and they don't mention word pairs as far as I can tell. Can you tell me what you meant by word pairs? Also, could we try to train a vanilla word2vec model just with the full transcriptions of our training data?
The word pairs are implied when they say 'train using the skipgram method' I believe. It is also something that was explained to me through email from one of the authors. Here is a fragment of the communication that speaks to this:
The seq2seq model always takes one segment as input and outputs one segment. Here's an example utterance: "how are you", and assume we are training speech2vec with skipgram with a window size of 2, we first need to generate all (input, output) pairs, which are (how, are), (how, you), (are, how), (are, you), (you, how), and (you, are). In this case the utterance would produce 6 training examples.
I haven't attempted training text embeddings on the librispeech corpus. Seems the authors used the fasttext implementation . I trained very simple text embeddings here. The thought was to verify that the word pairs that we create are good at least to some extent and that the verification method is working. The results of the experiment support both of these notions.
7. Since the authors made the embeddings they trained available here, it could be fun to compare them to some other pre-trained models (trained on text), like the ones by SpaCy. Would this be useful?
Absolutely! The more we understand about embeddings and the LibriSpeech dataset, the better positioned we will be to address training semantic embeddings on this and other datasets.
8. I don't think I saw Teacher Forcing mentioned in the paper, but you mentioned in #6 that we know it was used - did you get that info from the authors? How do you think it affects training?
This was something that was revealed to me via email communication with one of the authors. I believe this to have fundamental importance. I played a little bit with the architecture from the paper as autoencoder and with this small of an embedding size (50 for instance) I suspect the model would lack capacity to do well even on the reconstruction task. Plus here we get to bombard the model with essentially randomly chosen target word for a given source word. For instance, if in the training set we have the following word pairs ('you', 'awesome'), ('you', 'are'), ('you', 'fast'), the model would not have a way of telling which word it needs to predict when presented with 'you' as the source word. Teacher forcing makes a big difference here and I suspect it is the interplay between teacher forcing and the architecture that leads to training semantic embeddings.
9. What's the final word embeddings' dimension? The authors talk about 50-dim being enough, is that what you chose?
Yes, we use embeddings of dimensionality 50.
10. I was recently reading about the Transformer and how attention can outperform RNNs and LSTMs. Do you think it would be worth implementing an architecture similar to that of the Transformer? On the other hand, I think this would be something to explore after we've been able to emulate the results in the speech2vec paper with the same architecture.
Absolutely! 🙂 We also envision the speech2vec architecture to be a starting point. Removing the RNN from the encoder and decoder does sound very tempting.
Ok, I think that's everything! 😅 Thanks for your patience!!
Appreciate the opportunity to discuss this 🙏
JoseMMontoro
ShaunSpinelli
Hi @radekosmulski , I figured I'd open a new issue to discuss the paper itself so we can keep using #6 for your updates only.
Forgive me in advance if some of my questions have already been discussed/explained elsewhere. I hope this discussion is still productive, even if only to clarify some concepts in the paper for future people contributing to the project.
This is my most general question - I'd like to understand if, at this point, our most immediate goal is to purely reproduce the model that the authors of the paper trained. Is that the case? If so, where are the complications that you have pointed out coming from? Is it because their model architecture, the training steps and the specific training data is not completely defined in the paper? I.e. are there aspects of the training process we're not sure how the authors implemented them?
In particular, do we know what the specific 500 hours of audio they use for training are? Which subset of data (if any) did you choose from all Librispeech audio data for training?
How is the representation of the audio features created? I feel like that's a crucial part of the process, and there's very little mention of it in the paper. They mentioned they use MFCCs but don't go too deep into how it was implemented, how they evaluated the resulting representation (is it vectors?), etc. What does the sequence of acoustic features used for training look like? Can you expand on this aspect?
How are the different "acousticc realizations" of each word (meaning: different ways of pronouncing a word) averaged? Where in the code does this happen?
Are we able to measure the variance between each realization of the same word present in the dataset, same way the authors did in 3.5?
I'm not sure what you refer to by "word pairs" on item 6 in #6. My suggestion was to train embeddings from Librispeech text data, using the word2vec architecture, and compare to the embeddings trained on audio. The authors of the paper did exactly that to compare performance (I was very surprised to see that the embeddings trained on audio perform better on the intrinsic benchmarks! That's very exciting), and they don't mention word pairs as far as I can tell. Can you tell me what you meant by word pairs? Also, could we try to train a vanilla word2vec model just with the full transcriptions of our training data?
Since the authors made the embeddings they trained available here, it could be fun to compare them to some other pre-trained models (trained on text), like the ones by SpaCy. Would this be useful?
I don't think I saw Teacher Forcing mentioned in the paper, but you mentioned in #6 that we know it was used - did you get that info from the authors? How do you think it affects training?
What's the final word embeddings' dimension? The authors talk about 50-dim being enough, is that what you chose?
I was recently reading about the Transformer and how attention can outperform RNNs and LSTMs. Do you think it would be worth implementing an architecture similar to that of the Transformer? On the other hand, I think this would be something to explore after we've been able to emulate the results in the speech2vec paper with the same architecture.
Ok, I think that's everything! 😅 Thanks for your patience!!