Handling larger corpora

[Allegra-Cohen]

There are some hard-coded aspects of the kmeans implementation that only work for tiny corpora around 100 sentences:

1. K-max (the maximum number of clusters tried) and the step size between k's tried (see generate() in soft_kmeans.py)
2. The number of loops allowed if not converged during kmeans
3. The threshold for soft assignment after kmeans

There's probably a smart way of adjusting the above parameters so that quality and speed aren't damaged too badly when the corpora get bigger.

[Allegra-Cohen]

Another possibility here is to preprocess the corpus such that only semantically interesting sentences get kept, and you can scroll infinitely through the context window to read everything else in the document if you want.