memory requirement - Githubissues

adjidieng / ETM

Topic Modeling in Embedding Spaces

MIT License

549 stars 128 forks source link

memory requirement #12

Open veustp opened 4 years ago

veustp commented 4 years ago

hi,

i was trying to get this running on my own dataset of 40K documents and a vocabulary of 76K. But the computer "says no": need 100Gb RAM...
Am I doing something wrong here or is this really this memory-greedy?

It ran fine with about 1K docs and 5K voc. For 10K docs, it required about 36Gb RAM. Are these normal numbers?

br, Pieter

ydennisy commented 4 years ago

Hi, No this seems very off, much higher than what I needed.

Disclaimer: I am not an author just a random guy using this model