Synchronize operator

[abarret]

@bindi-nagda take a look at this branch. It introduces a couple of options to synchronize different parts of the Krylov operator. I can get the Krylov solver to converge to a residual of 1e-15 in almost all cases with these options. Running the four-roll-mill target gives a non-converging solver sometimes, I'm unsure why. Check and make sure that you can too, and then see what order of accuracy you get with the solver. I'm also unsure if these are exactly the correct steps to take, I'm still exploring what exactly these changes do.

Note this requires the most recent IBAMR main commit, so you will probably have to pull in the IBAMR build.

[bindi-nagda]

Okay great, I will test this.

[bindi-nagda]

Running target fluid_multigrid without preconditioning, I also get convergence to 1e-15 with rectangular and L-shaped refinement. With preconditioning turned on, the convergence is very slow. Do you also see this?

[abarret]

Running target fluid_multigrid without preconditioning, I also get convergence to 1e-15 with rectangular and L-shaped refinement. With preconditioning turned on, the convergence is very slow. Do you also see this?

No. I see convergence with the preconditioner in ~50 iterations in L-shaped domains. Can you send me the input file you are using?

[bindi-nagda]

The solver converges in about 39-78 iterations for most test problems with L-shaped refined regions. There is stalling observed at a relative residual of 1e-04 for input2d.var_theta.trans_vel.

So far, I also get convergence for the four roll mill problem. The solver takes ~30 iterations to achieve a relative residual of 1e-15 in each time step.

[bindi-nagda]

In particular, these three lines control which changes are used:

* `USE_DIV` synchronizes the side centered values before computing the divergence of the velocities.

* `USE_SYNCHED_INTERP` uses a synchronized node and side interpolated version of the volume fraction

* `POST_SYNCH` performs an extra synchronization step at the end of the `apply()` function.

You can switch back to the old behavior by commenting out one of these options and rebuilding.

I suppose next we want to toggle between these options to identify which ones are helping with the convergence?

[abarret]

I suppose next we want to toggle between these options to identify which ones are helping with the convergence?

I expect USE_SYNCHED_INTERP is not necessary, but the others should be. It would be good to try all combinations on an L-shaped domain + variable theta.

[abarret]

I'm going to merge this patch as is so I can add more improvements. I've highlighted my concerns in #46.