Closed johnjasa closed 10 months ago
Thank you @johnjasa ! This fixes the issue I was facing.
Here is a summary of the memory usage for my Dymos+OAS case. The memory usage is shown in % of the total memory I have on my machine (64GB).
Before this fix (Dymos 1.9.0)
n_procs = 1
setup: 12.1%
run_model: 5.5%
compute_totals: 10.6%
n_procs = 2
setup: 20.6%
run_model: 5.2%
compute_totals: 11.0%
n_procs = 4
setup: 37.6%
run_model: 5.6%
compute_totals: 12.0%
n_procs = 8
setup: 72.0%
run_model: 7.2%
compute_totals: 13.6%
n_procs = 16
runs out of memory during setup
And with this fix:
n_procs = 1
setup: 3.8%
run_model: 4.4%
compute_totals: 10.6%
n_procs = 2
setup: 4.2%
run_model: 5.2%
compute_totals: 11.0%
n_procs = 4
setup: 4.8%
run_model: 5.6%
compute_totals: 12.0%
n_procs = 8
setup: 6.3%
run_model: 7.2%
compute_totals: 13.6%
n_procs = 16
setup: 9.6%
run_model: 11.2%
compute_totals: 16.0%
As far as I observe,final_setup()
uses approximately the same amount of memory as run_model
and is not a bottleneck. But I've only monitored memory from top
with 0.1sec frequency, so I might have overlooked if there was any "spike" during final_setup.
Summary
After a good amount of digging into cases that scaled poorly as number of procs increased, I found a
toarray()
call that was turning a sparse array into a dense before going back to sparse. In the case of @kanekosh's run script with relatively highnum_segments
and a memory-expensive parallel ODE, this caused a large increase in memory usage duringsetup()
.This new implementation keeps the jac in sparse format throughout. I've changed it in the two places where it happened -- and maybe those files could be combined into one? Or @robfalck were they two separate files on purpose?
Prior![without_fix](https://github.com/OpenMDAO/dymos/assets/16373529/1869acee-24cc-49bb-b8ee-ee1d5d12292e)
setup()
mem usage:Mem usage for![with_fix](https://github.com/OpenMDAO/dymos/assets/16373529/805fa2fb-6bca-44e9-8a2f-f5b568acdb71)
setup()
with the fix:This PR does not address potentially large memory usage in
final_setup()
as that is a separate but related issue regarding howPETScVectors
are created and used. We should further discuss if we have action items there.Related Issues
Backwards incompatibilities
None
New Dependencies
None