Missing Ifpack2 settings in Teko

[MalachiTimothyPhillips]

@trilinos/Teko @cgcgcg

It seems as if there are several Ifpack2 settings that are missing from Teko.

For example, schwarz: num iterations is missing.

In other words, if I use the following settings:

<ParameterList name="Teko-Tpetra">
  <ParameterList name="Teko-parameters">
    <ParameterList name="my_block_solver">
      <Parameter name="Type" type="string" value="Block Gauss-Seidel"/>
      <Parameter name="Inverse Type 1"  type="string" value="schwarz"/>
      <Parameter name="Inverse Type 2"  type="string" value="direct"/>
      <Parameter name="Inverse Type 3"  type="string" value="direct"/>
      <Parameter name="Use Upper Triangle" type="bool" value="true"/>
    </ParameterList>
    <ParameterList name="direct">
     <Parameter name="Type" type="string" value="Amesos2"/>
    </ParameterList>
    <ParameterList name="schwarz">
     <Parameter name="Type" type="string" value="Ifpack2"/>
     <Parameter name="Prec Type" type="string" value="SCHWARZ"/>
      <ParameterList name="Ifpack2 Settings">
        <Parameter name="schwarz: use reordering" type="bool" value="true"/>
        <Parameter name="schwarz: overlap level" type="int" value="0"/>
        <Parameter name="schwarz: num iterations" type="int" value="2"/>
      </ParameterList>
    </ParameterList>
  </ParameterList>
</ParameterList>

I encounter this error:

Error, the parameter {name="schwarz: num iterations",type="int",value="2"}
in the parameter (sub)list "ANONYMOUS->Teko-Tpetra->Teko-parameters->schwarz->Ifpack2 Settings"
was not found in the list of valid parameters!

The valid parameters and types are:
  {
    "fact: iluk level-of-fill" : int = 1
    "fact: iluk level-of-overlap" : int = 0
    "chebyshev: max eigenvalue" : double = nan
    "chebyshev: ratio eigenvalue" : double = nan
    "chebyshev: min eigenvalue" : double = 30
    "chebyshev: degree" : int = 1
    "chebyshev: eigenvalue max iterations" : int = 10
    "chebyshev: eigenvalue relative tolerance" : double = 0
    "chebyshev: eigenvalue keep vector" : bool = 0
    "chebyshev: assume matrix does not change" : bool = 0
    "chebyshev: min diagonal value" : double = 2.22045e-16
    "chebyshev: zero starting solution" : bool = 1
    "chebyshev: use native spmv" : bool = 0
    "amesos: solver type" : string = Amesos_Klu
    "fact: level-of-fill" : int = 0
    "fact: absolute threshold" : double = 0
    "fact: relative threshold" : double = 1
    "fact: drop tolerance" : double = 0
    "fact: ict level-of-fill" : double = 1
    "fact: relax value" : double = 0
    "fact: ilut level-of-fill" : double = 1
    "fact: type" : string = serial
    "parallel ILUT options" : ParameterList = ...
    "trisolver: type" : string = Internal
    "trisolver: block size" : int = 1
    "trisolver: reverse U" : bool = 0
    "partitioner: local parts" : int = 1
    "partitioner: overlap" : int = 0
    "partitioner: print level" : int = 0
    "relaxation: container" : string = TriDi
    "relaxation: type" : string = Jacobi
    "relaxation: sweeps" : int = 1
    "relaxation: direction" : string = forward
    "relaxation: damping factor" : double = 1
    "relaxation: min diagonal value" : double = 1
    "relaxation: zero starting solution" : bool = 1
    "relaxation: backward mode" : bool = 0
    "relaxation: use l1" : bool = 0
    "relaxation: l1 eta" : double = 1.5
    "relaxation: banded container superdiagonals" : int = -1
    "relaxation: banded container subdiagonals" : int = -1
    "relaxation: mtgs cluster size" : int = 1
    "relaxation: mtgs coloring algorithm" : string = Default
    "relaxation: long row threshold" : int = 0
    "fact: sparskit: lfil" : int = 0
    "fact: sparskit: tol" : double = 0
    "fact: sparskit: droptol" : double = 0
    "fact: sparskit: permtol" : double = 0.1
    "fact: sparskit: alph" : double = 0
    "fact: sparskit: mbloc" : int = -1
    "fact: sparskit: type" : string = ILUT
    "schwarz: compute condest" : bool = 0
    "schwarz: combine mode" : string = ZERO
    "schwarz: use reordering" : bool = 1
    "schwarz: filter singletons" : bool = 0
    "schwarz: overlap level" : int = 0
    "partitioner: type" : string = greedy
    "partitioner: parts" : Array(Teuchos::ArrayRCP<int>) = {}
    "partitioner: maintain sparsity" : bool = 0
    "block relaxation: decouple dofs" : bool = 0
    "partitioner: use symmetric graph" : bool = 1
    "Amesos2" : ParameterList = ...
    "Amesos2 solver name" : string = KLU2
    "partitioner: map" : Teuchos::ArrayRCP<int> = Teuchos::ArrayRCP<int>{ptr=0,lowerOffset=0,upperOffset=-1,size=0,node=0,strong_count=0,weak_count=0}
    "partitioner: line detection threshold" : double = 0
    "partitioner: PDE equations" : int = 1
    "partitioner: coordinates" : Teuchos::RCP<Tpetra::MultiVector<double, int, long, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > > = Teuchos::RCP<Tpetra::MultiVector<double, int, long, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > >{ptr=0,node=0,strong_count=0,weak_count=0}
    "hypre: Solver" : string = PCG
    "hypre: Preconditioner" : string = Euclid
    "hypre: SolveOrPrecondition" : string = Solver
    "hypre: Solver functions" : ParameterList = ...
    "hypre: Preconditioner functions" : ParameterList = ...
    "Operators" : ParameterList = ...
    "Coordinates" : ParameterList = ...
    "hypre: Dump" : bool = 0
    "hypre: SetPreconditioner" : bool = 0
    "hypre: NumFunctions" : int = 0
  }

I am not sure why I am seeing this issue -- at first glance, it looks like Teko should be getting the settings from the Ifpack2 adapter https://github.com/trilinos/Trilinos/blob/9665b8ef28ce53e3a47ad2243fdcb33b73ff49f4/packages/ifpack2/adapters/thyra/Thyra_Ifpack2PreconditionerFactory_def.hpp#L321.

[github-actions[bot]]

Automatic mention of the @trilinos/ifpack2 team

[cgcgcg]

@MalachiTimothyPhillips Can you check your version of Trilinos? I think I fixed that here: https://github.com/trilinos/Trilinos/commit/523e2a54d75dad2eacfc836a67e6a41742e84ae5

[MalachiTimothyPhillips]

@MalachiTimothyPhillips Can you check your version of Trilinos? I think I fixed that here: 523e2a5

I do not have a specific SHA to point to since we build from a tarball "snapshot" of master via spack, but it does seem those changes are present.

packages/ifpack2/src/Ifpack2_Parameters.cpp:

  // Additive Schwarz preconditioner
  params.set("schwarz: compute condest", false); // mfh 24 Mar 2015: for backwards compatibility ONLY
  params.set("schwarz: combine mode", "ZERO"); // use string mode for this
  params.set("schwarz: use reordering", true);
  params.set("schwarz: filter singletons", false);
  params.set("schwarz: overlap level", 0);
  params.set("schwarz: num iterations", 1);
  params.set("subdomain solver name", "");
  Teuchos::ParameterList dummyListSubdomain;
  params.set("subdomain solver parameters",dummyListSubdomain);
  params.sublist("subdomain solver parameters").disableRecursiveValidation();

If it helps, the app code usage resembles:

  Teuchos::RCP<Stratimikos::DefaultLinearSolverBuilder> strat =
      Teuchos::rcp(new Stratimikos::DefaultLinearSolverBuilder("",
          "",
          "",
          "linear-solver-params-file",
          "extra-linear-solver-params",
          "linear-solver-params-used-file"));
  Stratimikos::enableMueLu<double, LocalOrdinal, GlobalOrdinal, Node>(*strat);
  Teko::addToStratimikosBuilder(strat);

  // preconditioner_parameters is read from the XML file
  Teuchos::RCP<Teko::InverseLibrary> invLib =
  Teko::InverseLibrary::buildFromParameterList(preconditioner_parameters, strat);

  // throws in this call
  Teuchos::RCP<Teko::InverseFactory> inverse = invLib->getInverseFactory(myInverseFactory);

[cgcgcg]

Ok, lemme see if I can reproduce this.

[MalachiTimothyPhillips]

Ok, lemme see if I can reproduce this.

If it helps, I did manage to trace the SHA we are using to de68716d971f08d995548a6bdc37f205dbf2ba50. Let me know if you need me to create a reproducer on my end -- this is currently routed through SIERRA.

[MalachiTimothyPhillips]

Marking as closed. Next time I should do a clean build to verify 🤦 .

[MalachiTimothyPhillips]

Thanks, @cgcgcg!

[cgcgcg]

Nice! Issue fixed without any code changes :-P

[jhux2]

@MalachiTimothyPhillips This was through the SIERRA build process?

[github-actions[bot]]

Automatic mention of the @trilinos/ifpack2 team

[MalachiTimothyPhillips]

@MalachiTimothyPhillips This was through the SIERRA build process?

It is -- the first build was before a Trilinos integration including the changes in 523e2a5. The "regular" build process did not seem to catch the change. Altering the compiler flags or forcing a rebuild resolved the issue.