Closed ylqfp closed 8 years ago
ping @bhoppi
I think parameters relate to scale are -numPartitions and -numThreads. There is no general best strategy for all corpus, so you need to try. But there are some principles to set these paras: for partitions number: setting it as less as possible, but to insure that blocks must be smaller than 2GB; for threads number: setting it appropriately, not to large or too small. @hucheng Can you share the slides used for Kaiyuanshe about how to set up ZenLDA parameters? I don't have it.
Thanks So Much!!! Closed. @hucheng
Hi, I'm testing LDA for large scale dataset, billion docs*million words. However, spark originated lda always HANG, then i found ZEN. However, I've not found parameter setup guide for lda, except simple description in source code. My question is: