search

SheffieldML / GPy

Gaussian processes framework in python
BSD 3-Clause "New" or "Revised" License
2.01k stars 558 forks source link

Limit CPU usage of GPy? #670

Closed lisaeisenberg closed 6 years ago

lisaeisenberg commented 6 years ago

Hi,

I am trying to optimize multiple GPRegression models in parallel using multiprocessing. Each process gets a chunk of data, initializes the GPRegression object, calls optimize() on it and saves its parameters. My problem is that each of the processes uses more than 100% CPU, so I always end up using all the CPUs on my machine no matter how many processes I start.

Does GPy use multithreading for optimizing models? And if so, is there some way to control the number of CPUs it can use?

I have a similar problem in a later step, when I use multiple processes to read the optimized models (the way you recommended) and predict for different chunks of new data.

zhenwendai commented 6 years ago

GPy does not do multi-threading, but the underlying numpy does if you install the version with mkl (the version included in Anaconda). The number of used threads can be specified by the environment variable "OMP_NUM_THREADS".

lisaeisenberg commented 6 years ago

Thank you so much, this solved it!