I fixed this by moving the downsampling to an earlier place and changing all the later "pruned" to "clean", but some preprocessing results like vocab.pkl, corpus.pkl, and vectors are still not influenced by the downsampling and the further solution (if needed) seems not straightforward.
Just found that in the twenty_newsgroups example, the downsampled word list is never used after calculation: https://github.com/cemoody/lda2vec/blob/b7f4642b750c6e792c07d177bd57ad36e65bb35c/examples/twenty_newsgroups/data/preprocess.py#L46
I fixed this by moving the downsampling to an earlier place and changing all the later "pruned" to "clean", but some preprocessing results like
vocab.pkl,corpus.pkl, andvectorsare still not influenced by the downsampling and the further solution (if needed) seems not straightforward.