SCS on multiple GPU's

[prateeky2806]

Hi All -- I am solving an SDP program on say a matrix of n x n dimensions. I am using CVXPY with SCS solver and using GPU to solve the problem. Using SCS with GPU is a huge improvement over using CVXOPT with no GPU support. The issue is that for large n, the GPU is maxed out to 100% but there is still memory available in it. I have multiple GPU's but it is using only one of them. See image below

The question is can SCS use multiple GPU's for the same SDP program, if yes how?

Thanks Prateek

[bodono]

Happy to hear that the GPU support is helping!

Unfortunately at the moment SCS can only use one GPU, since it just ships the data over to a single GPU. We could make it able to support multiple GPUs by partitioning the data and sending the different partitions to different GPUs for the matrix multiplies (which is what is getting done on the GPU) and then stitching the result back together, though it would be some work and I'm not sure when I'll have the time to work on it, though anyone else is welcome to give it a shot.

[prateeky2806]

@bodono , thank you for the GPU support but it looks like solving the same problem with GPU and without GPU gives different results, for eg on CPU the status is solved and on GPU the SCS maxes out on the number of iterations and gives a different solution. I have mentioned it just to highlight the problem even though I know it is not valid for this thread and is know already.

[bodono]

Is this something you're seeing for every problem or just one you're trying? Also make sure you're comparing the indirect CPU version (use_indirect=True in the python call).

[prateeky2806]

Hey @bodono Can you please elaborate more on what does the parameter use_indirect=True does?

[sschnug]

@prateeky2806 This is probably the core-parameter to care about SCS and you will get some idea by reading the scientific-paper Conic Optimization via Operator Splitting and Homogeneous Self-Dual Embedding (especially chapter 4!).

For further research you might look into the term matrix-free methods (and simple mentionings are also at wiki).

A good intro into direct-methods can be found here.

A quite complete overview of indirect-methods is maybe Kelley: Iterative Methods for Linear and Nonlinear Equations.

Without not much knowing about SCS's internals, indirect-methods are expected to be less precise / less robust and then it depends on the other components of SCS, what will happen if this precision is quite off (depends on the data). This is probably analyzed in above paper.

As these two approaches are very different, it was recommended to use the same algorithmic approach when benchmarking / comparing CPU vs. GPU.

[prateeky2806]

@sschnug Thank you, I'll go through the references.