how to use memory-map function

facebookresearch / dlrm

An implementation of a deep learning recommendation model (DLRM)

MIT License

3.71k stars 825 forks source link

Closed gukejun1 closed 11 months ago

gukejun1 commented 1 year ago

How is the memory-map function implemented, how to reduce the memory usage principle, and where is it processed in the dlrm code?

mnaumovfb commented 1 year ago

The --memory-map flag minimizes the usage of RAM, in exchange for larger usage of disc space, see issue https://github.com/facebookresearch/dlrm/issues/201.

It relies on Fisher-Yates-Rao shuffle algorithm, with an example of its application to shuffling large data sets here.