search

KratosMultiphysics / Kratos

Kratos Multiphysics (A.K.A Kratos) is a framework for building parallel multi-disciplinary simulation software. Modularity, extensibility and HPC are the main objectives. Kratos has BSD license and is written in C++ with extensive Python interface.
https://kratosmultiphysics.github.io/Kratos/
Other
1.03k stars 245 forks source link

Nodal NORMAL Shape Derivatives in MPI #7329

Closed sunethwarna closed 4 years ago

sunethwarna commented 4 years ago

Description I want to calculate nodal normal shape derivatives to make adjoint bossak scheme to work with slip conditions (in the SensitivityBuilder). Currently, nodal normals for nodes with SLIP flag is calculated as follows.

  1. Calculate normals on conditions with SLIP flag is true.
  2. Distribute condition normals to nodes where SLIP on nodes are true.

When shape derivatives are calculated for nodal normals, first I will calculate shape derivatives for condition normals. Then I need to transfer these condition normal shape sensitivities to nodes. Lets look at the following example in MPI. (All of them are surface triangles with SLIP true) Capture

The solid black ones are local nodes for that rank. Grey one is a ghost node. Red and Green ones belong to different ranks than the current rank. If I want to calculate nodal norm shape derivative for middle most node, it will have shape derivatives w.r.t. all of the outer nodes as well because of step 2 described above. I will have to calculate normal shape derivative contributions from each condition and put them in a vector variable in nodes and then assemble them. In order to do proper assembly, I need to have a pre-defined order of normal shape derivatives (according to nodes).

Up to now, I have investigated and I see three potential ways of achieveing this.

  1. Using RetrieveGlobalIndexedPointersMap (as in FindGlobalNodalNeighboursProcess and doing the assembly of nodal normal shape derivatives by my-self
  2. Adding all of the "Red" and "Green" nodes also as ghost nodes to current rank (on the fly), and using communicator assemble methods. ( This way no mdpa modifications required to run adjoints, so same distributed mdpas used in primal can be used for adjoints )
  3. Adding all of the "Red" and "Green" nodes as ghost nodes at mdpa creation level inside metis partitioner. So no on the fly modifications will be required (with an option like what we do to add conditions to the same rank as its parent element). This will add additional overhead for non-SLIP based problems, or SLIP based problems where SHAPE_SENSITIVITIES are not required and same mdpa which is used for primal cannot be used for adjoints if the primal does not have this modification.

I am not sure whether 2nd or 3rd way is the best.. However, if I am to implement 2nd approach, I have following concerns/questions.

  1. How should I add ghost nodes to model part (the information about these neighbours can be retrieved by using RetrieveGlobalIndexedPointersMap as in FindGlobalNodalNeighboursProcess, It is a GlobalPointer, what is the proper way to convert them to ghost nodes?)
  2. What extra steps do I need to consider when modifying ghost mesh with additional nodes?
  3. What is an interface mesh used in parallel fill utilities.
  4. Can we have another variable in core to store condition neighbors like NODAL_CONDITION_NEIGHBOURS? otherwise for some reason if some one runs the current default find global nodal neighbors process, it will overwrite condition neighbors with element nodal neighbors.
  5. Any other concerns?

I would like to have opinions/solutions for this problem. Thanks all in advance :).

sunethwarna commented 4 years ago

@RiccardoRossi @roigcarlo @philbucher @adityaghantasala I am not sure who should I be tagging about this issue. I tagged only few I know of.

sunethwarna commented 4 years ago

I found a better way to do this without having to introduce additional ghost nodes, hence closing :)