search

constantinpape / cluster_tools

Distributed segmentation for bio-image-analysis
MIT License
34 stars 14 forks source link

Tasks do not respect thread limt #3

Closed constantinpape closed 5 years ago

constantinpape commented 5 years ago

Tasks tend to spawn way more threads / processes than specified. This might be related to the luigi scheduler or python subprocess.call?!

constantinpape commented 5 years ago

This does not seem to be a general issue, but rather some workflows which do not respect the thread limit set, or use more than one thread although they should be single threaded. E.g. the watershed workflow.

constantinpape commented 5 years ago

Turns out the most likely culprit is numpy: https://github.com/numpy/numpy/issues/11826 To fix it, either set the corresponding environment variables or use https://github.com/joblib/threadpoolctl.

constantinpape commented 5 years ago

This is indeed the issue. For now, the solution is to use this function and call it before the numpy import, like here.