Closed sharthZ23 closed 7 years ago
Yes, it is in the plans (aka minibatch). Is the data sparse or dense? I planned to add sparse features support first.
I have dense data.
This is for my future reference: 500Mx64x4=128GB
Facebook Research has recently released https://github.com/facebookresearch/faiss which is a very fast K-nn CUDA impl. It has K-means and allows to have datasets which do not fit into memory (it splits them and processes chunk by chunk). Faiss is focused on very large scale and is awesome so I will not address this issue.
How about add buffered input for large datasets? ~500M samples in 64 dimension into ~150M clusters for example.