FSI: Duplicate Force contibutions in parallel simulations

[MakisH]

As observed by @derekrisseeuw, if the interface patch is split between two processes, there are duplicate contributions to the force on those cells (see figure).

This is relevant to #7.

[derekrisseeuw]

Thank you for making this an issue Makis.

I have some notes on this:

• The problem is not in the force object on the openfoam side. I tried unidirectional coupling of only the force for both in serie and in parallel. In both cases the force is identical.
• The error in the force is not necessary on the boundary between processors.
• If the patch of interest is located on one processor, no problems occur.

[MakisH]

The problem is not in the force object on the openfoam side. I tried unidirectional coupling of only the force for both in serie and in parallel. In both cases the force is identical.

I am not sure I understand this. Why does the uni-directional coupling of the force giving the same error prove that it is not a problem in Force.C?

The error in the force is not necessary on the boundary between processors.

Where else have you seen it? Still near the parallel boundary? Does it also appear in serial simulations then? (I guess you say this in bullet 1)

[derekrisseeuw]

Where else have you seen it? Still near the parallel boundary? Does it also appear in serial simulations then? (I guess you say this in bullet 1)

The problem with the malfunctioning forces only occurs when running the fluid solver in parallel, and the boundary is broken up in several domains.

For the coupling, so far I only tried serial coupling, so I cannot comment on parallel coupling. (although I think this is an issue independent of the type of coupling).

[derekrisseeuw]

I also found a workaround.

In the decomposeParDict there is an option to force a faceSet + connecting cells on one core. This is called: singleProcessorFaceSets

The procure then becomes: Create a faceSet with topoSet:

    {
        name    fsiFaces;
        type    faceSet;
        action  new;
        source  patchToFace;
        sourceInfo
        {
          name fsiPatch;
        }
    }

Decompose with the additional constraint:

    constraints
    {
        singleProcessorFaceSets
        {
           //- Keep all of faceSet on a single processor. This puts all cells
           //  connected with a point, edge or face on the same processor.
           //  (just having face connected cells might not guarantee a balanced
           //  decomposition)
           // The processor can be -1 (the decompositionMethod chooses the
           // processor for a good load balance) or explicitly provided (upsets
           // balance)
           type    singleProcessorFaceSets;
           singleProcessorFaceSets ((fsiFaces -1));
       }
    }

The decomposition with additional constraint can however create a mesh which is highly suboptimal in terms of performance, because the first layer wrt to the FSI surface is added. A high amount of information then has to be communicated between cores.

[MakisH]

I cannot reproduce this anymore with our updated tutorial cases. I am trying with OpenFOAM v1806 and CalculiX 2.13 (the same as before).

For our cylinderFlap case, I have changed the decomposition method to hierarchical and the interface is scattered to different processes.

Additionally, with our latest setup, all the forces are smaller and inwards, contrary to the first picture. I am only plotting the Forces that the adapter writes, I have removed the Forces function object. I use glyphs in ParaView, with Forces as vector and none for scalar. The mesh is also coarser now.

I have currently only looked at the first coupling time steps, but this should be enough if this is an implementation error.

[derekrisseeuw]

As far as I see now you are evaluating the openfoam force object. I encountered this problem after the mapping from openfoam to the structural mesh. The time should not matter for this problem.

Could you try exporting the vtk files for calculix and obtain the force on the points there?

[MakisH]

I exported vtk/vtu files from both participants and they all look smooth.

Here are the Calculix_Mesh-Fluid files for every rank and for the first time step. I have removed the time from the name, so that they are not grouped strangely in ParaView. I don't see any duplicate non-zero contribution.

Are these the files you are refering to?

[derekrisseeuw]

I've only checked for vtk, but this should't make any difference.

Could you check the Calculix_Mesh-Calculix? I saw it for these vtk files. Otherwise I don't have a clue what is causing the difference in behaviour.

[MakisH]

This also looks fine:

Here are all the vtk/vtu files: vtk-all.tar.gz

I am closing this for now, please reopen if you find any hints that this still holds.

[precice-bot]

This issue has been mentioned on preCICE Forum on Discourse. There might be relevant details there:

https://precice.discourse.group/t/mapping-problem-openfoam-fenics-on-cluster/1344/2