shane-kercheval / oo-learning

Python machine learning library based on Object Oriented design principles; the goal is to allow users to quickly explore data and search for top machine learning algorithm candidates for a given dataset
MIT License
1 stars 0 forks source link

Upgrade to Python 3.7.5 along with other package updates results in mutliprocessing hanging #47

Open shane-kercheval opened 4 years ago

shane-kercheval commented 4 years ago

These files

oo-learning/oolearning/converters/TwoClassPrecisionRecallOptimizerConverter.py:
oo-learning/oolearning/converters/TwoClassRocOptimizerConverter.py:
oo-learning/oolearning/model_aggregation/ModelAggregator.py:
oo-learning/oolearning/model_processors/GridSearchModelTuner.py:
oo-learning/oolearning/model_processors/GridSearchTransformationTuner.py:
oo-learning/oolearning/model_processors/RepeatedCrossValidationResampler.py:
oo-learning/oolearning/unsupervised/Clustering.py:

with ThreadPool(cores) as pool:

sometimes this hanges

Tried changing to with get_context("spawn").Pool(cores) as pool:

based on https://codewithoutrules.com/2018/09/04/python-multiprocessing/, which seemed to work,

but now i'm getting various warnings, so i'm disabling parallelization for now until i have time to investigate further.

shane-kercheval commented 4 years ago

e.g. commented this out

        if parallelization_cores == 0 or parallelization_cores == 1:
            ppv_tpr_tuple = list(map(get_ppv_tpr, get_ppv_tpr_args))
        else:
            cores = cpu_count() if parallelization_cores == -1 else parallelization_cores
            # with ThreadPool(cores) as pool:
            # https://codewithoutrules.com/2018/09/04/python-multiprocessing/
            with get_context("spawn").Pool(cores) as pool:
                ppv_tpr_tuple = list(pool.map(get_ppv_tpr, get_ppv_tpr_args))

and changed to

ppv_tpr_tuple = list(map(get_ppv_tpr, get_ppv_tpr_args))
shane-kercheval commented 4 years ago

commented def test_Resampler_fold_indexes_parallelized(self): out

shane-kercheval commented 4 years ago

commented def test_resamplers_RandomForest_classification_parallization(self) out

shane-kercheval commented 4 years ago

commented def test_resampling_roc_pr_thresholds_resampler_parallelization(self) out