search

praxes / hexrd

Legacy hexrd repository (python2.7)
http://hexrd.readthedocs.org
14 stars 42 forks source link

Fix scaling problem with multiprocessing in find-orientations/paintGrid #72

Closed mwiebe closed 9 years ago

mwiebe commented 9 years ago

This change should make find-orientations scale well on all platforms, please test! On Windows, however, it will use up a lot more memory because it isn't sharing the common state via a global variable like fork()-based multiprocessing allows.

joelvbernier commented 9 years ago

Will test forthwith!

joelvbernier commented 9 years ago

Question: should we tag the previous commit as windows specific?

mwiebe commented 9 years ago

No, this patch actually improves things on windows as well, you can proceed with a single codebase for all platforms. I would recommend avoiding having separate codebases per platform if at all possible, I find it's far too easy for things to end up out of sync that way.

joelvbernier commented 9 years ago

Agreed! Testing right now -- on linux it will use all available CPU, waiting for my mac to free up to test it there as well. Many thanks!

ddale commented 9 years ago

Also replied on the mailing list, I tested on a mac and things looked good. But I recommend, if possible, that we check how much memory will be required per MP worker, and limit the number of workers based on the available physical memory. (I thought it would not be an issue, assuming memory scaled reasonably with the number of cpus, but then checked my relatively new 4-core laptop and discovered about half of the 8 GB of physical memory was being consumed by Chrome! I only had 500 MB free, perhaps not enough to handle 4 jobs!)

mwiebe commented 9 years ago

What I found on Windows is that there is a spike of memory usage, maybe caused by the depickling process, after which it settles down to something fairly reasonable. Because I disabled swap space (I prefer running out of memory to having my whole system grind to a complete stop when using too much), this spike did cause some out of memory errors. Chrome is truly a memory hog these days, it squeezes my memory like crazy too.

ddale commented 9 years ago

I think this memory spike issue might be mitigated by the approach I took in fit-grains, since it spins up the MP workers sequentially, all are not attempting to depickle state at once.

mwiebe commented 9 years ago

@ddale that sounds right to me