Design / prototype vectorized material property evaluation framework

[dschwen]

Reason

Expensive material models provided by external codes make use of vectorized evaluation on accelerator devices (e.g. GPUs). For these models to operate with optimal efficiency we need to provide large batches of quadrature point data in one go (as opposed to consecutive calls of the material model for each quadrature point).

Design

An initial proposed design would be an ElementUserObject to gather required data for all quadrature points. This UO would then call the external vectorized property computation during finalize. The resulting data would be made available to a proxy material class that would provide a MOOSE way of accessing the computed properties.

Impact

Added capability. This feature has been requested by @reverendbedford for a future vectorization of NEML (or rather its successor library) as well as my me for fast evaluation of neural network based thermodynamic free energies (potentially using the now available torch library, or some custom CUDA code).

[reverendbedford]

As @dschwen notes I strongly support this. I think it could significantly speed up some of our "complicated material" models, where the stress update is actually a significant portion of the overall simulation time.

[hugary1995]

I put up a basic prototype in hugary1995/moose/batch_material, with which we can test the performance of vectorized material.

[dschwen]

I was planning to work on it this week. I'll test my phase field use case.

[hugary1995]

It's not quite ready yet. I still need to implement a parallel gather in finalize. It'll only work in serial until I add that.

[dschwen]

Would be nice if it generic programming could be used to avoid passing these type strings in

[hugary1995]

Okay, this should be usable now.

Basically, replace the original material

# Regular version
# [Materials]
#   [compute_stress]
#     type = ComputeLagrangianLinearElasticStress
#     large_kinematics = true
#   []
# []

with the vectorized version

# Vectorized version
[UserObjects]
  [compute_stress_vectorized]
    type = VectorizedMaterialFake
    material = compute_stress
    execute_on = 'INITIAL LINEAR'
  []
[]
[Materials]
  [compute_stress]
    type = ComputeLagrangianLinearElasticStressVectorized
    large_kinematics = true
    vectorized_material = compute_stress_vectorized
  []
[]

In this example, faked GPU calls are made in VectorizedMaterialFake. Then ComputeLagrangianLinearElasticStressVectorized is essentially the same class as ComputeLagrangianLinearElasticStress but all actual computations are out-sourced to VectorizedMaterialFake which utilizes GPUs.

Have fun testing guys.

[hugary1995]

I made a few strong assumptions in this prototype:

• The mesh has a single block. (I actually don't know what'll happen if there are multiple blocks, it may work, or not...)
• The mesh is replicated on all processors, and element ids are continuous, and ranges from 0 to _mesh.maxElemId().
• All material properties are either Real, RankTwoTensor, or RankFourTensor. Although extending the support should be trivial.
• Regarding GPU: 1. There is only 1 compute node. 2. There is only 1 GPU on this node. Therefore, I gather all input vectors onto rank 0, then let rank 0 handle the external GPU calls, and finally broadcast the output vector.
• No threads.
• No meshChanged().