idaholab / moose

Multiphysics Object Oriented Simulation Environment
https://www.mooseframework.org
GNU Lesser General Public License v2.1
1.75k stars 1.05k forks source link

Eliminate tremendous memory requirements of AD "stateful" material properties #16836

Closed lindsayad closed 3 years ago

lindsayad commented 3 years ago

Reason

I recently received an input from @makeclean which had ~800k 3D elements, running AD finite strain calculations. I ran the input through the heap profiler and stopped the run after I got to 100GB of memory required. And guess what, 81 GB of that memory was taken up by the current state of properties in MaterialPropertyStorage. In other words we are currently storing the current state of an AD material property for every element for every quadrature point, and these material properties may be second order or even fourth order tensors. And for every ADReal you get the memory cost of 51 Reals (for the default AD config). This is enormous. And @dschwen has told me that other users have complained about the memory requirements of AD tensor mechanics.

Design

To fix this, I propose that when swapping the material property from MaterialData into MaterialPropertyStorage we do the ADReal to Real conversion right then and there.

Impact

Save orders of magnitude of memory usage and make things like AD tensor mechanics available to users with machines that don't have tremendous repositories of memory.

FWIW here is the memory profile. fixed-temp

fdkong commented 3 years ago

@lindsayad Do you think it is a good idea to add a few memory benchmarks?

lindsayad commented 3 years ago

I think it is a very good idea in theory. The implementation per all of our discussions is difficult. I think it would be a good thing for an intern (probably not doctoral level) to work on.