search

iskandr / fancyimpute

Multivariate imputation and matrix completion algorithms implemented in Python
Apache License 2.0
1.25k stars 178 forks source link

No option to limit the number of threads #40

Closed dburkhardt closed 7 years ago

dburkhardt commented 7 years ago

Thanks for your work putting together all these methods!

I'm using NuclearNormMinimization on a shared computer, and it seems to default to use all of the threads on the machine. Is there any way to limit this? I imagine it has to do with CVXPY, but I read through their documentation and there's no mention of how to do that.

Maybe some of you all know where I would look to limit that? It would be nice if you could pass an n_threads argument to NuclearNormMinimization so my process plays nicely with others.

sergeyf commented 7 years ago

Howdy. Yup we just leave that part at default.

See here: https://github.com/hammerlab/fancyimpute/blob/master/fancyimpute/nuclear_norm_minimization.py#L110

It should be very easy to do a **kwargs type of thing to pass any variables you'd like to the solver. Any interest in making a quick PR? We have lax standards for tests around here =)

dburkhardt commented 7 years ago

I could definitely make the PR, but I'm still not entirely clear on where in CVXPY there's an option of limit the number of threads.

I'm pretty sure the offending line is here: https://github.com/cvxgrp/cvxpy/blob/master/cvxpy/problems/problem.py#L384

But strangely it looks like cvxpy.problem._solve has parallel_solve=False by default so why would it be using so many cores? I'm new to using these packages, so maybe I'm just missing something?

sergeyf commented 7 years ago

Yeah, it doesn't look like cvxpy has any flag to limit CPUs use. I would probably open a PR on the cvxpy repo about the specific line you found, and see if they're interested in adding a flag, and/or if they have ideas about how to globally limit the process to only see n CPUs (where you set n).

Once they have a flag we, can add a feature here to let you specify it and pass it through. I'll close this ticket for now, but feel free to open it again once there's something that can be done on our side.