search

rdicosmo / parmap

Parmap is a minimalistic library allowing to exploit multicore architecture for OCaml programs with minimal modifications.
http://rdicosmo.github.io/parmap/
Other
94 stars 20 forks source link

misleading documentation on memory usage #27

Closed william3 closed 9 years ago

william3 commented 10 years ago

the documentation says : "By forking the parent process on a sigle machine, the children get access, for free, to all the data structures already built"

However, forking a process with 4GB of memory with ncores=3, generates 2 additional subprocesses, with 4GB of memory, which leads to actually 12GB used.

The documentation is I think misleading on this matter.

Best regards

rdicosmo commented 10 years ago

Dear William, determining the real memory usage is a tricky issue, so I am very curious to know how you formed the opinion that the documentation is misleading.

Since Parmap is implemented by forking the parent process, immediately after the fork no extra real memory is allocated, despite what tools like ps or top show... for this, I refer you to this discussion on stack overflow, that provides some valuable information http://unix.stackexchange.com/questions/34795/correctly-determining-memory-usage-in-linux

Of course, if you start using the shared data, making copies, writing to it or the like, than the real memory usage will grow (that's copy on write), but this depends on what your program does, and the documentation carefully state you get "access" for free (read access)... write access is not for free.

If you have suggestions on how to make the documentation clearer, I'd be glad to take them into account (a git pull request would be the best way to go)

2014-06-04 11:42 GMT+02:00 william3 notifications@github.com:

the documentation says : "By forking the parent process on a sigle machine, the children get access, for free, to all the data structures already built"

However, forking a process with 4GB of memory with ncores=3, generates 2 additional subprocesses, with 4GB of memory, which leads to actually 12GB used.

The documentation is I think misleading on this matter.

Best regards

— Reply to this email directly or view it on GitHub https://github.com/rdicosmo/parmap/issues/27.

Roberto Di Cosmo

Professeur En delegation a l'INRIA PPS E-mail: roberto@dicosmo.org Universite Paris Diderot WWW : http://www.dicosmo.org Case 7014 Tel : ++33-(0)1-57 27 92 20 5, Rue Thomas Mann F-75205 Paris Cedex 13 Identica: http://identi.ca/rdicosmo

FRANCE. Twitter: http://twitter.com/rdicosmo

Attachments: MIME accepted, Word deprecated

http://www.gnu.org/philosophy/no-word-attachments.html

Office location:

Bureau 320 (3rd floor) Batiment Sophie Germain Avenue de France

Metro Bibliotheque Francois Mitterrand, ligne 14/RER C

GPG fingerprint 2931 20CE 3A5A 5390 98EC 8BFC FCCA C3BE 39CB 12D3

william3 commented 10 years ago

Thanks for your feedback. I used top, which is actually tricky. I have 8GB RAM. My process is actually 1.5GB. with 6 cores, top indicates 5 subprocesses using "1.5GB". Very quickly after subprocesses are launched, my disk starts to swap! While theroretically this should not happen, as 1.5 < 8. I tried to use "Gc.compact ()", because I thought that maybe each subprocesses were using some extra garbage internal memory, but it did not change the problem.

rdicosmo commented 10 years ago

One should be also careful to check what other process is eating up the memory: execute

free

before and after launching the process, to check what goes on.

For example, a web browser with many tabs open can easily eat up more than 1 Gb all by itself.

On Wed, Jun 04, 2014 at 06:27:41AM -0700, william3 wrote:

Thanks for your feedback. I used top, which is actually tricky. I have 8GB RAM. My process is actually 1.5GB. with 6 cores, top indicates 5 subprocesses using "1.5GB". Very quickly after subprocesses are launched, my disk starts to swap! While theroretically this should not happen, as 1.5 < 8. I tried to use "Gc.compact ()", because I thought that maybe each subprocesses were using some extra garbage internal memory, but it did not change the problem.

— Reply to this email directly or view it on GitHub.*

Roberto Di Cosmo

Professeur En delegation a l'INRIA PPS E-mail: roberto@dicosmo.org Universite Paris Diderot WWW : http://www.dicosmo.org Case 7014 Tel : ++33-(0)1-57 27 92 20 5, Rue Thomas Mann
F-75205 Paris Cedex 13 Identica: http://identi.ca/rdicosmo

FRANCE. Twitter: http://twitter.com/rdicosmo

Attachments: MIME accepted, Word deprecated

http://www.gnu.org/philosophy/no-word-attachments.html

Office location:

Bureau 3020 (3rd floor) Batiment Sophie Germain Avenue de France

Metro Bibliotheque Francois Mitterrand, ligne 14/RER C

GPG fingerprint 2931 20CE 3A5A 5390 98EC 8BFC FCCA C3BE 39CB 12D3

william3 commented 10 years ago

ok I have 5GB free, so it is still much biger than 1.5GB.

rdicosmo commented 10 years ago

Ok, now, again, it all depends on what you program do: if you start writing in the data that has been "copied for free" in each process memory, the real memory used will start to grow at the same time (the OS underneath performs a "copy on write" operation) http://en.wikipedia.org/wiki/Copy-on-write

On Wed, Jun 04, 2014 at 6:37:52AM -0700, william3 wrote:

ok I have 5GB free, so it is still much biger than 1.5GB.

— Reply to this email directly or view it on GitHub.*

Roberto Di Cosmo

Professeur En delegation a l'INRIA PPS E-mail: roberto@dicosmo.org Universite Paris Diderot WWW : http://www.dicosmo.org Case 7014 Tel : ++33-(0)1-57 27 92 20 5, Rue Thomas Mann
F-75205 Paris Cedex 13 Identica: http://identi.ca/rdicosmo

FRANCE. Twitter: http://twitter.com/rdicosmo

Attachments: MIME accepted, Word deprecated

http://www.gnu.org/philosophy/no-word-attachments.html

Office location:

Bureau 3020 (3rd floor) Batiment Sophie Germain Avenue de France

Metro Bibliotheque Francois Mitterrand, ligne 14/RER C

GPG fingerprint 2931 20CE 3A5A 5390 98EC 8BFC FCCA C3BE 39CB 12D3

rdicosmo commented 10 years ago

Notice also that your program will need memory space to allocate the result value, which needs to be passed back to the parent process... during the return phase, you may need as much as twice the memory of the result, as explained at the end of the README in https://github.com/rdicosmo/parmap

On Wed, Jun 04, 2014 at 06:37:52AM -0700, william3 wrote:

ok I have 5GB free, so it is still much biger than 1.5GB.

— Reply to this email directly or view it on GitHub.*

Roberto Di Cosmo

Professeur En delegation a l'INRIA PPS E-mail: roberto@dicosmo.org Universite Paris Diderot WWW : http://www.dicosmo.org Case 7014 Tel : ++33-(0)1-57 27 92 20 5, Rue Thomas Mann
F-75205 Paris Cedex 13 Identica: http://identi.ca/rdicosmo

FRANCE. Twitter: http://twitter.com/rdicosmo

Attachments: MIME accepted, Word deprecated

http://www.gnu.org/philosophy/no-word-attachments.html

Office location:

Bureau 3020 (3rd floor) Batiment Sophie Germain Avenue de France

Metro Bibliotheque Francois Mitterrand, ligne 14/RER C

GPG fingerprint 2931 20CE 3A5A 5390 98EC 8BFC FCCA C3BE 39CB 12D3

william3 commented 10 years ago

Parmap.map takes as input thouthands of "big data" (maybe 1MB per data), and as output "modified big data". So maybe the "modify" operation just creates what you call "copy on write". I give up on this, don't have enough time. Thank you very much for your answers

rdicosmo commented 10 years ago

Ok, let me try to explain what I believe is happening.

Imagine you have 1.5G of data at the beginning, and you run List.map on it, and get as a result another 1.5G of modified data. This means that at some point in time you will have 3G of data, as the original list is still there after the computation.

When you use Parmap, you need to count an extra 1.5G for the temporary memory space shared among the subprocesses to compute the result (the reason why this 1.5G cannot be the same 1.5G as the result is technical, but unavoidable).

So in the best case, your program will eat up 4.5G, and probably some more if there is local computation, and this may well explain the trashing you see.

As a rule of thumb, when running Parmap.map on data of size X to get a result of size Y, you need to have at least X+2*Y space.

On Wed, Jun 04, 2014 at 07:13:30AM -0700, william3 wrote:

Parmap.map takes as input thouthands of "big data" (maybe 1MB per data), and as output "modified big data". So maybe the "modify" operation just creates what you call "copy on write". I give up on this, don't have enough time. Thank you very much for your answers

— Reply to this email directly or view it on GitHub.*

Roberto Di Cosmo

Professeur En delegation a l'INRIA PPS E-mail: roberto@dicosmo.org Universite Paris Diderot WWW : http://www.dicosmo.org Case 7014 Tel : ++33-(0)1-57 27 92 20 5, Rue Thomas Mann
F-75205 Paris Cedex 13 Identica: http://identi.ca/rdicosmo

FRANCE. Twitter: http://twitter.com/rdicosmo

Attachments: MIME accepted, Word deprecated

http://www.gnu.org/philosophy/no-word-attachments.html

Office location:

Bureau 3020 (3rd floor) Batiment Sophie Germain Avenue de France

Metro Bibliotheque Francois Mitterrand, ligne 14/RER C

GPG fingerprint 2931 20CE 3A5A 5390 98EC 8BFC FCCA C3BE 39CB 12D3