pzelasko
commented
2 years ago I like your idea!
The Lhotse part can be solved using one of these two approaches:
~1) multiple dataloaders~
EDIT: I realized this won't be viable because of different padding in each sub-batch, go straight to 2)
dloaders = [
DataLoader(DynamicBucketingSampler(libri_cuts, max_duration=150, ...), ...),
DataLoader(DynamicBucketingSampler(giga_cuts, max_duration=150, ...), ...),
DataLoader(DynamicBucketingSampler(aishell_cuts, max_duration=150, ...), ...),
]
for batches in zip(dloaders):
# batches: Tuple[Dict[str, Tensor]]
# combine feature matrices into a single mini-batch
# save info about which sequence idx came from which dloader
# to use for decoder + joiner delegation later
# forward, backward, ...2) zip sampler
sampler = ZipSampler(
DynamicBucketingSampler(libri_cuts, max_duration=150, ...),
DynamicBucketingSampler(giga_cuts, max_duration=150, ...),
DynamicBucketingSampler(aishell_cuts, max_duration=150, ...),
)
dloader = DataLoader(sampler, ...)
for batch in dloader:
# batch: Dict[str, Tensor]
# feature matrices are already merged in this case
# get info about which sequence idx came from which sampler
# using batch["supervisions"]["cut"] and matching some pattern in cut IDs
# (alternatively modify the Dataset class to prepare this info in the worker subprocess)
# forward, backward, ...
danpovey
csukuangfj
ngoel17
ahmedalbahnasawy
Guys, I have been thinking about this for a few days and I think I am getting close to how we can solve this...
We tend to train on only one dataset (e.g. Librispeech); we rarely try to fold in other datasets (e.g. Gigaspeech). I'm thinking that we need to make it easier to train simultaneously on multiple datasets; this is the easiest way to improve our WERs. (For production purposes, you don't really care whether you used extra data, and lots is available). So why do we train on only one dataset? Using multiple datasets is a hassle partly due to domain mismatch. E.g. Gigaspeech is normalized differently from Librispeech, treats punctuations differently, has different kinds of data; so it would either be a lot of work to merge with Librispeech, or it would degrade results when tested on Librispeech.
My proposal is that we get in the habit of using multiple RNN-T predictors/decoders and joiners, one per dataset. That way we don't have to worry about mismatch. In general we can even use different tokenizers per dataset, and diffeerent vocabulary sizes. That way we can even generalize to multiple languages. However, we need to be careful that we always use all of the datasets on each minibatch. If we don't do this it will cause problems with DDP as it requires all parameters to be touched. We could just manually forward with dummy data if we notice a parameter isn't going to be used, so this isn't a 100% hard requirement, but if possible it's better to always have all datasets in a minibatch. This is known to be better for convergence anyway, versus alternating.
Anyway, my proposal is that we figure out at the Lhotse level how to do this. Now, we will require that the different datasets' features eventually be padded to the same length, so the encoder forward can be shared. (Later the core RNN-T computation can be shared too). So this will require some bucketing-sampler type of logic. What I propose is that we have a list of datasets and the user sets data proportions, as in, each minibatch wants x% of the data from each dataset. And we let one of the datasets define the epoch, e.g. the 1st one, and just cycle through the others somehow (might be complex to enable restarting from an epoch though).
I propose that all of the data elements returned by the dataloader be returned as tuples (of size num-datasets). We can manually concatenate tensors in any tuples that we want concatenated, e.g. the features.
I am thinking there can be an option that controls whether it's mandatory to have data from each dataset (note: if this option is set, it will impose a limit on the utterance length, and may also bias the data proportions unless we work around that somehow).
This is a fairly big feature but I think it's the right direction. Sorry that we have not, so far, done the unsupervised multi-dataset experiments with Gigaspeech that I asked for a feature for. We will do them at some point, with the codebook loss. But I think it will be easier to dip our toe into using multiple datasets by having it all-supervised, just with different RNN-T decoders. That should give a super-easy WER improvement without any real need to do research.