search

insitro / redun

Yet another redundant workflow engine
https://insitro.github.io/redun/
Apache License 2.0
510 stars 43 forks source link

Memory limits for local executor #42

Open jshleap opened 2 years ago

jshleap commented 2 years ago

I have noticed that redun seems to reserve memory even in local executors. Is there a way to tune that? or I am wrong on this assumption?

mattrasmus commented 2 years ago

Hi @jshleap! Thanks for the question. I don't believe we set memory limits on local executors. Are you getting some behavior that suggests otherwise?

reserve memory even in local executors

Unless by "reserve" you mean preallocating memory. redun isn't doing anything unusual in that regard more than what multiprocessing thread and process pools typically do (FWIW, that is how redun implements the local executors). Let me know if that helps, or if I misunderstood the question.

jshleap commented 2 years ago

Thanks for the answer @mattrasmus. I assumed that there is pre-allocation of memory per task as I am running out of memory for a medium sized dataset. I guess I should control the max_workers instead.

mattrasmus commented 2 years ago

max_workers will work to limit parallelism across all tasks. If its just one task that should have parallelism limits, check out the limits config option: https://insitro.github.io/redun/config.html?highlight=limit#limits