search

odow / SDDP.jl

A JuMP extension for Stochastic Dual Dynamic Programming
https://sddp.dev
Other
293 stars 62 forks source link

Multithreading parallelism #599

Closed odow closed 1 month ago

odow commented 1 year ago

The asynchronous solver currently uses multiprocessing. This causes a lot of excess RAM usage because we need N copies of the full problem.

If we introduced a lock to every subproblem, then we could have the asynchronous threads operating in parallel over a single PolicyGraph. We'd just need to be careful to always obtain a lock whenever we modified data inside a node. And potentially avoid some of the plugins that have state, like PSRSamplingScheme.

I don't have plans to implement this, just noting so that I can come back to this in future.

odow commented 1 year ago

This will be easier to test and benchmark on some of the upcoming MSPFormat problems, in which SDDP.jl performs poorly compared with QUASAR. (A Markovian graph with 22,000+ nodes.)

odow commented 1 year ago

One blocker is https://github.com/odow/SDDP.jl/blob/e631d22979163bc321908d97fdab53bcd7cbe26f/src/plugins/duality_handlers.jl#L64-L80. We'd need to change this so that we did it node-by-node, instead of all in one go.

odow commented 1 month ago

I have something in #758 that seems to work well.