search

Mouse-Imaging-Centre / pydpiper

Python code for flexible pipeline control
Other
24 stars 10 forks source link

Cluster interface: special casing for SGE memory? #416

Open gdevenyi opened 5 years ago

gdevenyi commented 5 years ago

In SGE land, parallel execution environments -pe smp <N> are assigned memory on a per-slot basis. Right now, as the memory requirements increase for a pydpiper executor, more memory is requested for the job (via h_vmem for example). For ANTS (and future ITK-like programs) they are multithreaded and and take advantage of extra CPUs. pydpiper executors could make better use of resources if the memory estimate could be used to scale the PE N request (qbatch --ppj N) instead of increasing h_vmem.

To do this, inputs to executor configuration would need

bcdarwin commented 5 years ago

I don't see a problem with including this functionality but I have no SGE cluster for testing/development.

From: Gabriel A. Devenyi notifications@github.com Sent: January 2, 2019 1:06:59 PM To: Mouse-Imaging-Centre/pydpiper Cc: Subscribed Subject: [Mouse-Imaging-Centre/pydpiper] Cluster interface: special casing for SGE memory? (#416)

In SGE land, parallel execution environments -pe smp are assigned memory on a per-slot basis. Right now, as the memory requirements increase for a pydpiper executor, more memory is requested for the job (via h_vmem for example). For ANTS (and future ITK-like programs) they are multithreaded and and take advantage of extra CPUs. pydpiper executors could make better use of resources if the memory estimate could be used to scale the PE N request (qbatch --ppj N) instead of increasing h_vmem.

To do this, inputs to executor configuration would need

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Mouse-2DImaging-2DCentre_pydpiper_issues_416&d=DwMCaQ&c=Sj806OTFwmuG2UO1EEDr-2uZRzm2EPz39TfVBG2Km-o&r=WbPKw40NU3g_RTKn7pWL3cSAdk6QRKr3kMreWPZzNcg&m=FN98LdsLRiGn5EVJeyTeZqP7i1ZZzuxzmtOeY83sgQo&s=m1NsM7YUTpnwEi7E0MuWIgZ0kwoj1H0upATq8puh0vg&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AAKBNJ8P1QZ9DfWCmiFcW4h2pEdl4LBFks5u-5FPVDgaJpZM4ZnA0-5F&d=DwMCaQ&c=Sj806OTFwmuG2UO1EEDr-2uZRzm2EPz39TfVBG2Km-o&r=WbPKw40NU3g_RTKn7pWL3cSAdk6QRKr3kMreWPZzNcg&m=FN98LdsLRiGn5EVJeyTeZqP7i1ZZzuxzmtOeY83sgQo&s=_XTV_felaCC4ZFxnXPcTbuzXUTQDjCV1p8I55mpv1_w&e=.

This e-mail may contain confidential, personal and/or health information(information which may be subject to legal restrictions on use, retention and/or disclosure) for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete all copies.

gdevenyi commented 5 years ago

Happy to test. I think the logic is pretty simple.

All we need is a flag to switch on this feature which swaps the memory specification for a scaling of the PR N value to be larger than the memory requirement divided by the default memory per slot.