search

Mouse-Imaging-Centre / pydpiper

Python code for flexible pipeline control
Other
25 stars 10 forks source link

more memory cleverness? #294

Open bcdarwin opened 7 years ago

bcdarwin commented 7 years ago

If many stages can run, the server will launch a job array with a vmem request based on the largest amount needed, which is quite wasteful (and in some cases may cause the scheduler to reject the request).

bcdarwin commented 6 years ago

Seems somewhat important to fix (e.g., I have a custom pipeline with 1000+ 1MB stages runnable and 1 80GB stage, and all executors are now running with 80GB requests).