search

UCL-CCS / EasyVVUQ

Python 3 framework to facilitate verification, validation and uncertainty quantification (VVUQ) for a wide variety of simulations.
https://easyvvuq.readthedocs.io/
GNU Lesser General Public License v3.0
85 stars 27 forks source link

Excessive amounts of memory are sometimes used by the PCE analysis phase #404

Closed DavidPCoster closed 9 months ago

DavidPCoster commented 11 months ago

Excessive amounts of memory are sometimes used by the PCE analysis phase

I have traced this down to the production of two outputs which might not be needed: the correlation_matrices (which in one of my cases accounts for 1.490 GB) and the output_distributions (which accounts for 9.780 GB for the case mentioned).

I am in the process of preparing some changes to the code so that the calculation of these outputs can be disabled.

Preliminary results are

                      | default Corr & Dist  | no Corr & Dist | ratio |
----------------------|----------------------|----------------|-------|
Time for analysis (s) |     20600            |    2617.06     | 0.127 |
user time (s)         |     19330.81         |    2563.83     | 0.133 |
sys time (s)          |      1834.21         |      86.13     | 0.047 |
elapsed time (s)      |     20632.66         |    2626.67     | 0.127 |
memory usage (bytes)  |     360 269 376      |    6 823 864   | 0.019 |

Memory usage drops to 2% and the elapsed time to 12.7%.

A pull request should come within the next few days, after some more testing.

DavidPCoster commented 9 months ago

I think we can close this ...