This completes the integration of Hanie's sentence corruptor into the pipeline. The code now works: given a set of input sentences, we train a language model, and we use a KB to generate candidate corruptions, then we use the language model to select among the corruptions.
Note that this PR looks like it has more in it than it does, currently, because it builds off of two outstanding PRs. Once those two are merged, it will be a lot easier to see what exactly is in this PR.
This completes the integration of Hanie's sentence corruptor into the pipeline. The code now works: given a set of input sentences, we train a language model, and we use a KB to generate candidate corruptions, then we use the language model to select among the corruptions.
Note that this PR looks like it has more in it than it does, currently, because it builds off of two outstanding PRs. Once those two are merged, it will be a lot easier to see what exactly is in this PR.