SimVascular / svFSIplus

svFSIplus is an open-source, parallel, finite element multi-physics solver.
https://simvascular.github.io/documentation/svfsi.html
Other
10 stars 26 forks source link

Using pre-computed velocity solutions in one-way coupled physics #202

Open zasexton opened 9 months ago

zasexton commented 9 months ago

Problem

The general problem: We wish to use the (potentially time-varying) velocity solutions obtained from a previous simulation within a subsequent physics (namely, scalar advection-diffusion).

Solution

This has been done previously in svFSI on side-branches where residence time calculations have been desired or where thrombosis modeling has been done. See Gutierrez et al 2021.

Although it is not clear if this exact framework would work well for a more object-oriented svFSIplus the following scheme is generally implemented in svFSI:

  1. True/False flag is selected for Use pre-computed velocity
  2. If True the following subroutine is called in svFSI to read in the velocity solution vector fields across some set of time points. This subroutine can be found in the VTKXML.f file of the attached svFSI source code. Reminder this will not be found on the main branches of svFSI as this functionality was never formally included.

    !**********************************************************************
    
      SUBROUTINE READVELOCITYVTU(fName)
    
      USE COMMOD
      USE LISTMOD
      USE ALLFUN
      USE vtkXMLMod
    
      IMPLICIT NONE
    
      CHARACTER(LEN=STDL) :: fName, sName
    
      TYPE(vtkXMLType) :: vtu
    
      INTEGER :: iStat, iEq, iOut, iM, l, s, e, a, b, Ac, nNo, oGrp
      INTEGER :: i, j, iTS, iVel, zp
      CHARACTER(LEN=stdL) :: varName, velName
      REAL(KIND=8), ALLOCATABLE :: tmpS(:,:), tmpGS(:,:)
      i = (lStep - fStep)/iStep + 1
      ALLOCATE (allU(nsd,gtnNo,i))
      iStat = 0
      CALL loadVTK(vtu, fName, iStat)
      IF (iStat .LT. 0) err = "VTU file read error (init)"
    
      CALL getVTK_numPoints(vtu, nNo, iStat)
      IF (iStat .LT. 0) err = "VTU file read error (num points)"
      IF (nNo .NE. SUM(msh(:)%gnNo)) 
     2   err = "Incompatible mesh and "//TRIM(fName)
      j=0
      ALLOCATE(tmpGS(nsd,gtnNo))
    
      iStat = -1
      iVel = 0
      zp = 0
    
      IF (fStep .GE. 1000) THEN
         sName = STR(fStep)
      ELSE
         WRITE(sName,'(I3.3)') fStep
      END IF
    
      DO WHILE(iStat .LT. 0)
         iVel = iVel + 1
         iStat = 0
         SELECT CASE (iVel)
         CASE(1)
            velName = "velocity"
         CASE(2)
            velName = "NS_Velocity"
         CASE(3)
            velName = "FS_Velocity"
         CASE(4)
            zp = 1
            WRITE(sName,'(I4.4)') fStep
            velName = "velocity"
         CASE(5)
            err ="VTU file read error (point data)"
         END SELECT
         varName = TRIM(velName)//"_0"//TRIM(ADJUSTL(sName))
         IF (iVel .EQ. 4) THEN
            PRINT *, "Try format <"//TRIM(velName)//"_00>"
         ELSE
            PRINT *, "Try format <"//TRIM(velName)//"_0>"
         END IF
         CALL getVTK_pointData(vtu, TRIM(varName), tmpGS, iStat)
      END DO
    
      DO iTS=fStep, lStep, iStep
         IF (iTS .GE. 1000) THEN
            sName = STR(iTS)
         ELSEIF (zp .EQ. 1) THEN
            WRITE(sName,'(I4.4)') iTS
         ELSE
            WRITE(sName,'(I3.3)') iTS
         END IF
    
         varName = TRIM(velName)//"_0"//TRIM(ADJUSTL(sName))
         CALL getVTK_pointData(vtu, TRIM(varName), tmpGS, iStat)
         IF (iStat .LT. 0) err ="VTU file read error (point data)"
    
         j=j+1
         allU(:,:,j) = tmpGS
    
      END DO
      CALL flushVTK(vtu)
    
      RETURN
      END SUBROUTINE READVELOCITYVTU
    !**********************************************************************
  3. The read velocity solutions are distributed across the processors. This code snippet is taken from the DISTRIBUTE subroutine in DISTRIBUTE.f
    
    !     Distributing allU to processors
      IF (velFileFlag) THEN
         IF (cm%mas()) THEN
            ALLOCATE(tmpU(nsd,gtnNo,ntpoints))
            tmpU = allU
            DEALLOCATE(allU)
         ELSE
            ALLOCATE(tmpU(0,0,0))
         END IF
         ALLOCATE(allU(nsd,tnNo, ntpoints))
         allU = LOCAL(tmpU)
         DEALLOCATE(tmpU)
      END IF

! Distributing tagFile node info to processors flag = (ALLOCATED(tagRT)) CALL cm%bcast(flag) IF (flag) THEN
IF (cm%mas()) THEN ALLOCATE(tmpRT(gtnNo)) tmpRT = tagRT DEALLOCATE(tagRT) ELSE ALLOCATE(tmpRT(0)) END IF ALLOCATE(tagRT(tnNo)) tagRT = LOCAL(tmpRT) DEALLOCATE(tmpRT) END IF

4. In the predictor step of the time iteration loop the new solution values are obtained from the `allU` solutions
```fortran
!     Predictor step
         CALL PICP
         CALL SETBCDIR(An, Yn, Dn)
         IF (velFileFlag) THEN
             n = MOD(cTS,ntpoints*iStep)

             n1 = n/iStep + 1
             n2 = n1 + 1

             i = (lStep - fStep)/iStep + 2
             IF(n2 .EQ. i) n2 = 1

             alpha = REAL(MOD(n,iStep),8)/REAL(iStep,8)
             Yn(1:nsd,:) = allU(:,:,n1)*(1D0 - alpha) + 
     2          allU(:,:,n2)*alpha
         END IF 
  1. The physics problem is solved at the current iteration and continues to iterate until the number of time steps is achieved

Attached files include the source code and example input files for residence time calculations. svFSI_input.txt tagFile.txt Code.tar.gz

Additional context

No response

Code of Conduct

zasexton commented 9 months ago

The following function is how I propose to read in time-varying solution field data across a given vtu mesh for svFSIplus. This would replicate behavior of the subroutine READVELOCITYVTU called in svFSI. However, there are some notable differences.

  1. the field_name such as "Velocity", "Pressure", etc must be supplied by the user and will not be hard-coded into the function.
  2. the time step of the field data is assumed to come at the end of the vtkDataArray name. This conforms to how such time-varying velocity and pressure data are written out to vtk files from svPost. However, because we do not hard-code the array name format we needed to define a robust method for extracting the time step information.
  3. Last, this function assumes that the mshType objects are equipped with an additional Array3<double> (called Ys) attribute capable of storing the solution field of interest.

    void load_time_varying_field_vtu(const std::string file_name, const std::string field_name, mshType& mesh) {
    
    #define n_debug_load_vtu
    #ifdef debug_load_vtu
    std::cout << "[load_vtu] " << std::endl;
    std::cout << "[load_vtu] ===== vtk_xml_parser::load_time_varying_field_vtu ===== " << std::endl;
    std::cout << "[load_vtu] file_name: " << file_name << std::endl;
    #endif
    
    auto reader = vtkSmartPointer<vtkXMLUnstructuredGridReader>::New();
    reader->SetFileName(file_name.c_str());
    reader->Update();
    vtkSmartPointer<vtkUnstructuredGrid> vtk_ugrid = reader->GetOutput();
    vtkIdType num_nodes = vtk_ugrid->GetNumberOfPoints();
    int array_count = 0;
    std::vector<str::pair<std::string, int>> array_names;
    
    if (num_nodes == 0) {
        throw std::runtime_error("Failed reading the VTK file '" + file_name + "'.");
    }
    // Store all array names
    for (int i = 0; i < vtk_ugrid->GetPointData()->GetNumberOfArrays(); i++) {
        std::string array_name = vtk_ugrid->GetPointData()->GetArrayName(i);
        size_t pos = array_name.find(field_name.c_str());
        if (pos != std::string::npos) {
            auto not_digit = [](char c) { return !std::isdigit(c); };
            auto it = std::find_if(array_name.rbegin(), array_name.rend(), not_digit);
            std::string time_step = std::string(it.base(), array_name.end());
            array_count++;
            if (!time_step.empty()) {
                array_names.push_back({array_name, std::stoi(time_step)});
            } else {
                array_names.push_back({array_name, 0});
            }
        }
    }
    
    if (array_count == 0) {
        throw std::runtime_error("No '" + field_name + "' data found in the VTK file '" + file_name + "'.");
    }
    
    // Order all array names
    std::sort(array_names.begin(), array_names.end(), [](const std::pair<std::string, int>& a, const std::pair<std::string, int>& b) {
        return a.second < b.second;
    });
    
    int num_components = array->GetNumberOfComponents();
    mesh.Ys.resize(num_components, num_nodes, array_count);
    
    for (int i = 0; i < array_count; i++) {
        auto array = vtk_ugrid->GetPointData()->GetArray(array_names[i].first.c_str());
        if (array == nullptr) {
            throw std::runtime_error("No '" + array_names[i].first + "' data found in the VTK file '" + file_name + "'.");
        }
        for (int j = 0; j < num_nodes; j++) {
            for (int k = 0; k < num_components; k++) {
                mesh.Ys(k, j, i) = array->GetComponent(j, k);
            }
        }
    }
    }
zasexton commented 9 months ago

For partitioned multiphysics, I believe this will be a powerful and flexible approach to incorporate previous solution data into new physics simulations. Because we do not assume a particular structure to the solution field we can potentially use this same functionality for scalar or tensor fields (should a partitioned scheme become relevant for such cases).

zasexton commented 9 months ago

The branch above implements functionality for reading in specified solution fields but does not yet distribute the global data across processors (necessary for any parallel simulation). This will be added next and seek to replace the functionality of the added code in svFSI DISTRIBUTE.f and should be necessary for partitioned multi-physics schemes (agnostic of the physics).

zasexton commented 9 months ago

The following function is how I propose to distribute precomputed state variables. Similar to the all_fun::local for Vector<double> and Array<double>, this function allows for the distribution of Array3<double>.

Array3<double>
local(const ComMod& com_mod, const CmMod& cm_mod, const cmType& cm, Array3<double>& U){
  if (com_mod.ltg.size() == 0) {
    throw std::runtime_error("ltg is not set yet");
  }

  Array3<double> local_array;
  int m;
  int n;
  int r;

  if (cm.mas(cm_mod)) {
    m = U.nrows();
    r = U.ncols();
    n = U.nslices();
    if (U.ncols() != com_mod.gtnNo) {
      throw std::runtime_error("local_rv is only specified for vector with size gtnNo");
    }
  }

  if (cm.seq()) {
    local_array.resize(m, com_mod.gtnNo, U.nslices());
    local_array = U;
    return local_array;
  }

  cm.bcast(cm_mod, &m);
  cm.bcast(cm_mod, &n);
  cm.bcast(cm_mod, &r);

  local_array.resize(m, com_mod.tnNo, r);
  Vector<double> tmpU(m * com_mod.gtnNo * r);

  if (cm.mas(cm_mod)) {
    for (int a = 0; a < com_mod.gtnNo; a++) {
      int s = m * a;
      for (int i = 0; i < U.nslices(); i++) {
        int e = i * m * (com_mod.gtnNo + 1);
        for (int j = 0; j < U.ncols(); j++) {
          tmpU(j+s+e) = U(j, a, i);
        }
      }
    }
  }

  cm.bcast(cm_mod, tmpU);

  for (int a = 0; a < com_mod.tnNo; a++) {
    int Ac = com_mod.ltg[a];
    int s = m * Ac;
    for (int i = 0; i < n; i++) {
      int e = i * m * (r + 1);
      for (int j = 0; j < m; j++) {
        local_array(j, a, i) = tmpU(j+s+e);
      }
    }
  }

  return local_array;
}
ktbolt commented 8 months ago

@zasexton Good to show the proposed code!

Some comments

zasexton commented 8 months ago

@ktbolt regarding your comments.

zasexton commented 8 months ago

The final component necessary to integrate precomputed state variables for partitioned physics allows for time-varying solutions. In general we may not assume that the time step size from a previous physics simulation will be the same as current physics being computed (the reasons for this are practical and numerous, but often partitioned multi-physics schemes may seek to bridge different time scales explicitly, for example). Instead we assume that pre-computed solutions can be linearly interpolated to establish temporal agreement. This seems to be a valid assumption since any well-resolved solution from a previous simulation should be well-approximated with temporal linearity. In the special cases where the governing equations of the previous simulation are linear (e.g. diffusion, steady-porous media flow), of course, linear interpolation is mathematically well-justified.

To do this we must add the following code to the main iteration loop of the main.cpp file (sorry @ktbolt for adding code here). Specifically, we must inject the interpolated, precomputed time-varying solutions we need into the next new state-variable array for the predictor/multi-corrector algorithm within the outer loop. This will be accomplished via the following:

    if (com_mod.usePrecomp) {
        #ifdef debug_iterate_solution
        dmsg << "Use precomputed values ..." << std::endl;
        #endif
        // This loop is used to interpolate between known time values of the precomputed
        // state-variable solution
        for (int l = 0; l < com_mod.nMsh; l++) {
            auto lM = com_mod.msh[l];
            if (lM.Ys.nslices() == 1) {
                // If there is only one temporal slice, then the solution is assumed constant
                // in time and no interpolation is performed
                continue;
            } else {
                // If there are multiple temporal slices, then the solution is linearly interpolated
                // between the known time values and the current time.
                double precompDt = com_mod.precompDt;
                double preTT = precompDt * (lM.Ys.nslices() - 1);
                double cT = cTS * dt;
                double rT = std::fmod(cT, preTT);
                int n1 = static_cast<int>(rT / precompDt);
                double alpha = std::fmod(rT, precompDt);
                int n2 = n1 + 1;
                for (int i = 0; i < nsd; i++) {
                    for (int j = 0; j < tnNo; j++) {
                        if (alpha == 0.0) {
                            Yn(i, j) = lM.Ys(i, j, n2);
                        } else {
                            Yn(i, j) = (1 - alpha) * lM.Ys(i, j, n1) + alpha * lM.Ys(i, j, n2);
                        }
                    }
                }
            }
        }
    }
zasexton commented 8 months ago

I will now work on adding necessary error checking and create a test case against the default scalar advection test within svFSIplus-Tests.

zasexton commented 8 months ago

When testing this scheme I am noticing differences between the concentration values obtained for time step 1 of simulations when comparing test case 02-dye-AD against the same problem which takes the velocity field from 02-dye-AD as an input. While I obtain a qualitatively similar answer. The concentration values are only similar for an absolute tolerance less than 1E-02. I would think that we should obtain closer numerical agreement. I have attached my test case folder here. result_001_check.vtu is the solution file that I am checking against for spatial agreement in concentration values obtained from the 8-procs/result_001.vtu.

Any suggestions on why I might be noticing such differences would be incredibly helpful!

08-AD-precompute.zip

zasexton commented 8 months ago

Perhaps there are contributions from the acceleration matrix Ag of the state variables that I am neglecting... I will check this now.

zasexton commented 8 months ago

When evaluating the updates to the Yo, Yn, Yg, Ao, An, and Ag matrices within the inner loop of the test problem 02-dye_AD we get the following for node 0:

---------------------------------------------------------------------
 Eq     N-i     T       dB  Ri/R1   Ri/R0    R/Ri     lsIt   dB  %t
Processor: 0 Testing velocity solution 1 time: 1.0000000000000000e-02
Yo(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00
Yn(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00
Yg(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00
Ao(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00
An(0:4, 0) = -0.0000000000000000e+00 -0.0000000000000000e+00 -0.0000000000000000e+00 -0.0000000000000000e+00
Ag(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00
---------------------------------------------------------------------
 NS 1-1  1.050e+01  [0 1.000e+00 1.000e+00 8.198e-05]  [6 -16 89]
Processor: 0 Testing velocity solution 1 time: 1.0000000000000000e-02
Yo(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00
Yn(0:4, 0) = -5.3276918290568094e-04 -1.1692563817828404e-03 -1.3421719058673989e-01 -1.5960282417374954e+03
Yg(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00
Ao(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00
An(0:4, 0) = -7.9915377435852114e-02 -1.7538845726742600e-01 -2.0132578588010976e+01 -2.3940423626062422e+05
Ag(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00
 NS 1-2  1.851e+01  [-57 1.371e-03 1.371e-03 7.028e-05]  [4 -16 85]
Processor: 0 Testing velocity solution 1 time: 1.0000000000000000e-02
Yo(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00
Yn(0:4, 0) = -5.3360839660324664e-04 -1.1668867423981941e-03 -1.3422206319926949e-01 -1.5960337788708462e+03
Yg(0:4, 0) = -3.5517945527045394e-04 -7.7950425452189356e-04 -8.9478127057826595e-02 -1.0640188278249968e+03
Ao(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00
An(0:4, 0) = -8.0041259490486974e-02 -1.7503301135972907e-01 -2.0133309479890414e+01 -2.3940506683062686e+05
Ag(0:4, 0) = -6.6596147863210095e-02 -1.4615704772285501e-01 -1.6777148823342479e+01 -1.9950353021718687e+05
 NS 1-3  2.755e+01  [-127 4.033e-07 4.033e-07 2.165e-05]  [5 -17 87]
Processor: 0 Testing velocity solution 1 time: 1.0000000000000000e-02
Yo(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00
Yn(0:4, 0) = -5.3360849099892975e-04 -1.1668874366192548e-03 -1.3422206473249659e-01 -1.5960337954684771e+03
Yg(0:4, 0) = -3.5573893106883106e-04 -7.7792449493212939e-04 -8.9481375466179652e-02 -1.0640225192472308e+03
Ao(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00
An(0:4, 0) = -8.0041273649839442e-02 -1.7503311549288816e-01 -2.0133309709874482e+01 -2.3940506932027149e+05
Ag(0:4, 0) = -6.6701049575405819e-02 -1.4586084279977424e-01 -1.6777757899908678e+01 -1.9950422235885574e+05
 HF 1-1  2.781e+01  [0 1.000e+00 1.000e+00 9.319e-07]  [14 -139 9]
Processor: 0 Testing velocity solution 1 time: 1.0000000000000000e-02
Yo(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00
Yn(0:4, 0) = -5.3360849099892975e-04 -1.1668874366192548e-03 -1.3422206473249659e-01 -1.5960337954684771e+03
Yg(0:4, 0) = -3.5573899399928650e-04 -7.7792495774616985e-04 -8.9481376488331058e-02 -1.0640225303123179e+03
Ao(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00
An(0:4, 0) = -8.0041273649839442e-02 -1.7503311549288816e-01 -2.0133309709874482e+01 -2.3940506932027149e+05
Ag(0:4, 0) = -6.6701061374866200e-02 -1.4586092957740682e-01 -1.6777758091562070e+01 -1.9950422443355958e+05
 HF 1-2  2.810e+01  [-7 4.259e-01 4.259e-01 7.688e-07]  [20 -141 14]
Processor: 0 Testing velocity solution 1 time: 1.0000000000000000e-02
Yo(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00
Yn(0:4, 0) = -5.3360849099892975e-04 -1.1668874366192548e-03 -1.3422206473249659e-01 -1.5960337954684771e+03
Yg(0:4, 0) = -3.5573899399928650e-04 -7.7792495774616985e-04 -8.9481376488331058e-02 -1.0640225303123179e+03
Ao(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00
An(0:4, 0) = -8.0041273649839442e-02 -1.7503311549288816e-01 -2.0133309709874482e+01 -2.3940506932027149e+05
Ag(0:4, 0) = -6.6701061374866200e-02 -1.4586092957740682e-01 -1.6777758091562070e+01 -1.9950422443355958e+05
 HF 1-3  2.839e+01  [-19 1.116e-01 1.116e-01 5.772e-07]  [24 -144 19]
Processor: 0 Testing velocity solution 1 time: 1.0000000000000000e-02
Yo(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00
Yn(0:4, 0) = -5.3360849099892975e-04 -1.1668874366192548e-03 -1.3422206473249659e-01 -1.5960337954684771e+03
Yg(0:4, 0) = -3.5573899399928650e-04 -7.7792495774616985e-04 -8.9481376488331058e-02 -1.0640225303123179e+03
Ao(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00
An(0:4, 0) = -8.0041273649839442e-02 -1.7503311549288816e-01 -2.0133309709874482e+01 -2.3940506932027149e+05
Ag(0:4, 0) = -6.6701061374866200e-02 -1.4586092957740682e-01 -1.6777758091562070e+01 -1.9950422443355958e+05
 HF 1-4  2.870e+01  [-26 4.801e-02 4.801e-02 7.866e-07]  [25 -141 20]
Processor: 0 Testing velocity solution 1 time: 1.0000000000000000e-02
Yo(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00
Yn(0:4, 0) = -5.3360849099892975e-04 -1.1668874366192548e-03 -1.3422206473249659e-01 -1.5960337954684771e+03
Yg(0:4, 0) = -3.5573899399928650e-04 -7.7792495774616985e-04 -8.9481376488331058e-02 -1.0640225303123179e+03
Ao(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00
An(0:4, 0) = -8.0041273649839442e-02 -1.7503311549288816e-01 -2.0133309709874482e+01 -2.3940506932027149e+05
Ag(0:4, 0) = -6.6701061374866200e-02 -1.4586092957740682e-01 -1.6777758091562070e+01 -1.9950422443355958e+05
 HF 1-5s 2.904e+01  [-32 2.367e-02 2.367e-02 9.940e-07]  [25 -138 18]
Processor: 0 Testing velocity solution 2 time: 2.0000000000000000e-02
Yo(0:4, 0) = -5.3360849099892975e-04 -1.1668874366192548e-03 -1.3422206473249659e-01 -1.5960337954684771e+03
Yn(0:4, 0) = -5.3360849099892975e-04 -1.1668874366192548e-03 -1.3422206473249659e-01 -1.5960337954684771e+03
Yg(0:4, 0) = -3.5573899399928650e-04 -7.7792495774616985e-04 -8.9481376488331058e-02 -1.0640225303123179e+03
Ao(0:4, 0) = -8.0041273649839442e-02 -1.7503311549288816e-01 -2.0133309709874482e+01 -2.3940506932027149e+05
An(0:4, 0) = 4.0020636824919693e-02 8.7516557746444013e-02 1.0066654854937234e+01 1.1970253466013566e+05
Ag(0:4, 0) = -6.6701061374866200e-02 -1.4586092957740682e-01 -1.6777758091562070e+01 -1.9950422443355958e+05
---------------------------------------------------------------------
 NS 2-1  3.636e+01  [0 1.000e+00 1.839e+02 3.944e-05]  [3 -16 71]
Processor: 0 Testing velocity solution 2 time: 2.0000000000000000e-02
Yo(0:4, 0) = -5.3360849099892975e-04 -1.1668874366192548e-03 -1.3422206473249659e-01 -1.5960337954684771e+03
Yn(0:4, 0) = -1.3252451822083765e-03 3.7517246374009733e-04 -4.3286150942518128e-02 -1.0326527324563585e+01
Yg(0:4, 0) = -5.3360849099892975e-04 -1.1668874366192548e-03 -1.3422206473249659e-01 -1.5960337954684769e+03
Ao(0:4, 0) = -8.0041273649839442e-02 -1.7503311549288816e-01 -2.0133309709874482e+01 -2.3940506932027149e+05
An(0:4, 0) = -7.8724866856497266e-02 3.1882554280034675e-01 2.3707041923433998e+01 3.5755862488172262e+05
Ag(0:4, 0) = 2.0010318412459843e-02 4.3758278873221999e-02 5.0333274274686151e+00 5.9851267330067807e+04
 NS 2-2  4.579e+01  [-87 4.347e-05 7.994e-03 5.968e-05]  [5 -16 87]
Processor: 0 Testing velocity solution 2 time: 2.0000000000000000e-02
Yo(0:4, 0) = -5.3360849099892975e-04 -1.1668874366192548e-03 -1.3422206473249659e-01 -1.5960337954684771e+03
Yn(0:4, 0) = -1.2384220200150923e-03 3.8510504958807226e-04 -4.2575470341120109e-02 -7.8751555937798114e+00
Yg(0:4, 0) = -1.0613662851385610e-03 -1.3884750304635343e-04 -7.3598122205844288e-02 -5.3889561670586818e+02
Ao(0:4, 0) = -8.0041273649839442e-02 -1.7503311549288816e-01 -2.0133309709874482e+01 -2.3940506932027149e+05
An(0:4, 0) = -6.5701392527504648e-02 3.2031543067754298e-01 2.3813644013643700e+01 3.5792633064134017e+05
Ag(0:4, 0) = -7.8944267988720965e-02 2.3651576641814095e-01 1.6400316651215917e+01 2.5806467584805697e+05
 NS 2-3  5.541e+01  [-149 3.164e-08 5.819e-06 2.762e-05]  [5 -16 87]
Processor: 0 Testing velocity solution 2 time: 2.0000000000000000e-02
Yo(0:4, 0) = -5.3360849099892975e-04 -1.1668874366192548e-03 -1.3422206473249659e-01 -1.5960337954684771e+03
Yn(0:4, 0) = -1.2384151328604723e-03 3.8509879095267039e-04 -4.2575630754292104e-02 -7.8750636174432405e+00
Yg(0:4, 0) = -1.0034841770097049e-03 -1.3222577914770347e-04 -7.3124335138245608e-02 -5.3726136888534563e+02
Ao(0:4, 0) = -8.0041273649839442e-02 -1.7503311549288816e-01 -2.0133309709874482e+01 -2.3940506932027149e+05
An(0:4, 0) = -6.5700359454311644e-02 3.2031449188223271e-01 2.3813619951667899e+01 3.5792634443779069e+05
Ag(0:4, 0) = -6.8091372714560450e-02 2.3775733964913784e-01 1.6489151726390674e+01 2.5837109731440491e+05
 HF 2-1  5.570e+01  [0 1.000e+00 9.756e-01 7.971e-07]  [17 -140 12]
Processor: 0 Testing velocity solution 2 time: 2.0000000000000000e-02
Yo(0:4, 0) = -5.3360849099892975e-04 -1.1668874366192548e-03 -1.3422206473249659e-01 -1.5960337954684771e+03
Yn(0:4, 0) = -1.2384151328604723e-03 3.8509879095267039e-04 -4.2575630754292104e-02 -7.8750636174432405e+00
Yg(0:4, 0) = -1.0034795855732915e-03 -1.3222995157130475e-04 -7.3124442080360272e-02 -5.3726130756778787e+02
Ao(0:4, 0) = -8.0041273649839442e-02 -1.7503311549288816e-01 -2.0133309709874482e+01 -2.3940506932027149e+05
An(0:4, 0) = -6.5700359454311644e-02 3.2031449188223271e-01 2.3813619951667899e+01 3.5792634443779069e+05
Ag(0:4, 0) = -6.8090511820232943e-02 2.3775655731971260e-01 1.6489131674744172e+01 2.5837110881144699e+05
 HF 2-2  5.604e+01  [-7 4.065e-01 3.966e-01 7.464e-07]  [19 -141 16]
Processor: 0 Testing velocity solution 2 time: 2.0000000000000000e-02
Yo(0:4, 0) = -5.3360849099892975e-04 -1.1668874366192548e-03 -1.3422206473249659e-01 -1.5960337954684771e+03
Yn(0:4, 0) = -1.2384151328604723e-03 3.8509879095267039e-04 -4.2575630754292104e-02 -7.8750636174432405e+00
Yg(0:4, 0) = -1.0034795855732915e-03 -1.3222995157130475e-04 -7.3124442080360272e-02 -5.3726130756778787e+02
Ao(0:4, 0) = -8.0041273649839442e-02 -1.7503311549288816e-01 -2.0133309709874482e+01 -2.3940506932027149e+05
An(0:4, 0) = -6.5700359454311644e-02 3.2031449188223271e-01 2.3813619951667899e+01 3.5792634443779069e+05
Ag(0:4, 0) = -6.8090511820232943e-02 2.3775655731971260e-01 1.6489131674744172e+01 2.5837110881144699e+05
 HF 2-3  5.637e+01  [-15 1.678e-01 1.637e-01 8.330e-07]  [20 -140 12]
Processor: 0 Testing velocity solution 2 time: 2.0000000000000000e-02
Yo(0:4, 0) = -5.3360849099892975e-04 -1.1668874366192548e-03 -1.3422206473249659e-01 -1.5960337954684771e+03
Yn(0:4, 0) = -1.2384151328604723e-03 3.8509879095267039e-04 -4.2575630754292104e-02 -7.8750636174432405e+00
Yg(0:4, 0) = -1.0034795855732915e-03 -1.3222995157130475e-04 -7.3124442080360272e-02 -5.3726130756778787e+02
Ao(0:4, 0) = -8.0041273649839442e-02 -1.7503311549288816e-01 -2.0133309709874482e+01 -2.3940506932027149e+05
An(0:4, 0) = -6.5700359454311644e-02 3.2031449188223271e-01 2.3813619951667899e+01 3.5792634443779069e+05
Ag(0:4, 0) = -6.8090511820232943e-02 2.3775655731971260e-01 1.6489131674744172e+01 2.5837110881144699e+05
 HF 2-4  5.668e+01  [-22 7.136e-02 6.962e-02 9.638e-07]  [22 -139 19]
Processor: 0 Testing velocity solution 2 time: 2.0000000000000000e-02
Yo(0:4, 0) = -5.3360849099892975e-04 -1.1668874366192548e-03 -1.3422206473249659e-01 -1.5960337954684771e+03
Yn(0:4, 0) = -1.2384151328604723e-03 3.8509879095267039e-04 -4.2575630754292104e-02 -7.8750636174432405e+00
Yg(0:4, 0) = -1.0034795855732915e-03 -1.3222995157130475e-04 -7.3124442080360272e-02 -5.3726130756778787e+02
Ao(0:4, 0) = -8.0041273649839442e-02 -1.7503311549288816e-01 -2.0133309709874482e+01 -2.3940506932027149e+05
An(0:4, 0) = -6.5700359454311644e-02 3.2031449188223271e-01 2.3813619951667899e+01 3.5792634443779069e+05
Ag(0:4, 0) = -6.8090511820232943e-02 2.3775655731971260e-01 1.6489131674744172e+01 2.5837110881144699e+05
 HF 2-5s 5.704e+01  [-30 2.958e-02 2.886e-02 8.107e-07]  [19 -140 11]

From this readout I am a bit confused why the 4th degree of freedom corresponding to the scalar field is being updated during the fluid physics?

ktbolt commented 8 months ago

@zasexton This may be an indexing bug.

zasexton commented 8 months ago

@ktbolt this was a zack bug. the 4th degree of freedom is pressure so I'll need to add another print statement to check again

zasexton commented 8 months ago

When rechecking the updates to the Yo, Yn, Yg, Ao, An, and Ag matrices within the inner loop of the test problem 02-dye_AD we get the following for node 0:

---------------------------------------------------------------------
 Eq     N-i     T       dB  Ri/R1   Ri/R0    R/Ri     lsIt   dB  %t
---------------------------------------------------------------------
Processor: 0 Testing velocity solution 1 time: 1.0000000000000000e-02
Yo(0:4, 0) = -5.3360849097713642e-04 -1.1668874366280032e-03 -1.3422206473249587e-01 0.0000000000000000e+008.8645060761327133e-04
Yn(0:4, 0) = -5.3360849097713642e-04 -1.1668874366280032e-03 -1.3422206473249587e-01 0.0000000000000000e+008.8645060761327133e-04
Yg(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+000.0000000000000000e+00
Ao(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+000.0000000000000000e+00
An(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 -0.0000000000000000e+000.0000000000000000e+00
Ag(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+000.0000000000000000e+00
 HF 1-1  2.670e-01  [0 1.000e+00 1.000e+00 3.747e-07]  [15 -148 10]
Processor: 0 Testing velocity solution 1 time: 1.0000000000000000e-02
Yo(0:4, 0) = -5.3360849097713642e-04 -1.1668874366280032e-03 -1.3422206473249587e-01 0.0000000000000000e+008.8645060761327133e-04
Yn(0:4, 0) = -5.3360849097713642e-04 -1.1668874366280032e-03 -1.3422206473249587e-01 -9.3323161016075558e-068.8645060761327133e-04
Yg(0:4, 0) = -5.3360849097713642e-04 -1.1668874366280032e-03 -1.3422206473249587e-01 0.0000000000000000e+008.8645060761327133e-04
Ao(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+000.0000000000000000e+00
An(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 -1.3998474152411330e-030.0000000000000000e+00
Ag(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+000.0000000000000000e+00
 HF 1-2  5.480e-01  [-6 4.499e-01 4.499e-01 7.123e-07]  [20 -142 15]
Processor: 0 Testing velocity solution 1 time: 1.0000000000000000e-02
Yo(0:4, 0) = -5.3360849097713642e-04 -1.1668874366280032e-03 -1.3422206473249587e-01 0.0000000000000000e+008.8645060761327133e-04
Yn(0:4, 0) = -5.3360849097713642e-04 -1.1668874366280032e-03 -1.3422206473249587e-01 -4.7320534883353537e-068.8645060761327133e-04
Yg(0:4, 0) = -5.3360849097713642e-04 -1.1668874366280032e-03 -1.3422206473249587e-01 -6.2215440677383700e-068.8645060761327133e-04
Ao(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+000.0000000000000000e+00
An(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 -7.0980802325030286e-040.0000000000000000e+00
Ag(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 -1.1665395127009443e-030.0000000000000000e+00
 HF 1-3  8.810e-01  [-18 1.158e-01 1.158e-01 6.809e-07]  [24 -142 17]
Processor: 0 Testing velocity solution 1 time: 1.0000000000000000e-02
Yo(0:4, 0) = -5.3360849097713642e-04 -1.1668874366280032e-03 -1.3422206473249587e-01 0.0000000000000000e+008.8645060761327133e-04
Yn(0:4, 0) = -5.3360849097713642e-04 -1.1668874366280032e-03 -1.3422206473249587e-01 -8.7446218908725520e-078.8645060761327133e-04
Yg(0:4, 0) = -5.3360849097713642e-04 -1.1668874366280032e-03 -1.3422206473249587e-01 -3.1547023255569025e-068.8645060761327133e-04
Ao(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+000.0000000000000000e+00
An(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 -1.3116932836308834e-040.0000000000000000e+00
Ag(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 -5.9150668604191913e-040.0000000000000000e+00
 HF 1-4  1.202e+00  [-26 4.892e-02 4.892e-02 6.112e-07]  [26 -143 20]
Processor: 0 Testing velocity solution 1 time: 1.0000000000000000e-02
Yo(0:4, 0) = -5.3360849097713642e-04 -1.1668874366280032e-03 -1.3422206473249587e-01 0.0000000000000000e+008.8645060761327133e-04
Yn(0:4, 0) = -5.3360849097713642e-04 -1.1668874366280032e-03 -1.3422206473249587e-01 -1.8497082606738441e-078.8645060761327133e-04
Yg(0:4, 0) = -5.3360849097713642e-04 -1.1668874366280032e-03 -1.3422206473249587e-01 -5.8297479272483680e-078.8645060761327133e-04
Ao(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+000.0000000000000000e+00
An(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 -2.7745623910107754e-050.0000000000000000e+00
Ag(0:4, 0) = 0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00 -1.0930777363590695e-040.0000000000000000e+00
 HF 1-5s 1.531e+00  [-32 2.386e-02 2.386e-02 9.198e-07]  [26 -139 20]

We do have an indexing issue for the update to acceleration Ag. The acceleration updates are accumulating in the 4th degree of freedom (pressure) instead of the scalar field (5th degree of freedom). This is most likely do to the mis-match in physics equations actually being called in the problem (i.e. we do not actually add the fluid physics equation to the simulation) but the degrees of freedom for the simulation remain the same since we inject the precomputed solution. I will resolve this now.

zasexton commented 8 months ago

It seems that the numerical differences are from the Dirichlet velocity boundary condition imposed on the fluid physics where the value for Yg is prescribed as Yn . Of course, if scalar advection is computed without prescribing an accompanying fluid simulation and instead we have only the velocity field solutions across time we will not guarantee this strict satisfaction (nor would we necessarily want to!). To recover the solution of the svFSIplus-Test/04-fluid/02-dye-AD in the first time step we can thus impose velocity at the fluid Dirichlet boundary in the precomputed velocity files to ensure Yg agreement. When we do this we recover the same solutions at the first time step regardless of using the partitioned sequential physics (as is traditionally been done) or using a precomputed velocity solution. See below:

precomputed_concentration_error

The maximum absolute node-wise error is 6.44449771e-16 and the mean absolute node-wise error is 2.45571103e-19. This makes sense and agrees with our intuition of numerical precision that we would expect of this simulation.

zasexton commented 8 months ago

Attached is the relevant test case for verifying agreement of the scalar-advection equation using a steady, precomputed velocity field solution. The two solutions, for the scalar field demonstrate parity to desired floating point precision. Unfortunately such a test case for time-varying inlet flow rates is not as easy to construct due to assumptions about Yo, Yn, and Yg between the two simulation schemes. For now, this implementation of the code seems sufficient and rational for the applications it would be called.

I will begin a draft merge request now...

02-precompute-velocities.zip