Fix broken civet next tests

[miaoyinb]

closes #27628

Reason

Design

Impact

[miaoyinb]

The broken tests are caused by randomness in triangulation. When using different numbers of cores, the generated TRI elements are slightly different. I tried to use average elem quality and size with a reasonable custom tolerances to cover all the cases (using -p 1 as the gold). I did not try -p 5 or -p 7, which failed in this case.

To fix this, I did a comprehensive test using different -p numbers and adopt the mean values instead of the -p 1 values as the gold.

[miaoyinb]

@GiudGiud @loganharbour FYI.

[loganharbour]

Do we not just want to limit the triangulation to a fixed set of cores to run on?

[miaoyinb]

You meant just using the same number in max_parallel and min_parallel to limit this? (and maybe have multiple tests with different parallel numbers?)

[GiudGiud]

You meant just using the same number in max_parallel and min_parallel to limit this? (and maybe have multiple tests with different parallel numbers?)

this can be done. Or simply a max_parallel = 3 if for example the results for p = 1, 2, 3 are consistent enough to pass the gold

[GiudGiud]

I m fine with this averaging that you have now. Time will tell if it's robust enough though

[miaoyinb]

I think the motivation of fixing p is to remove the randomness so that Exodiff can be used instead. In that case, p = 1,2,3.. would all give different exo files.

I think the current average solution is better than a series of Exodiff tests.

[miaoyinb]

Also, my observation is once p>9, things are converged for this specific case and area/quality pp values no longer change.

[loganharbour]

I think the current average solution is better than a series of Exodiff tests.

If this works, I agree.

[moosebuild]

Job Documentation on ccfec93 wanted to post the following:

View the site here

This comment will be updated on new commits.

[GiudGiud]

Test failures are unrelated

[moosebuild]

Job Coverage on ccfec93 wanted to post the following:

Framework coverage

	dc8a1f	#27629 ccfec9
	Total	Total	+/-	New
Rate	85.09%	85.09%	+0.00%	-
Hits	103153	103154	+1	0
Misses	18077	18076	-1	0

Diff coverage report

Full coverage report

Modules coverage

Coverage did not change

Full coverage reports

Reports

- [`framework`](https://mooseframework.inl.gov/docs/PRs/27629/coverage/framework) - [`chemical_reactions`](https://mooseframework.inl.gov/docs/PRs/27629/coverage/chemical_reactions) - [`combined`](https://mooseframework.inl.gov/docs/PRs/27629/coverage/combined) - [`contact`](https://mooseframework.inl.gov/docs/PRs/27629/coverage/contact) - [`electromagnetics`](https://mooseframework.inl.gov/docs/PRs/27629/coverage/electromagnetics) - [`external_petsc_solver`](https://mooseframework.inl.gov/docs/PRs/27629/coverage/external_petsc_solver) - [`fluid_properties`](https://mooseframework.inl.gov/docs/PRs/27629/coverage/fluid_properties) - [`fsi`](https://mooseframework.inl.gov/docs/PRs/27629/coverage/fsi) - [`functional_expansion_tools`](https://mooseframework.inl.gov/docs/PRs/27629/coverage/functional_expansion_tools) - [`geochemistry`](https://mooseframework.inl.gov/docs/PRs/27629/coverage/geochemistry) - [`heat_transfer`](https://mooseframework.inl.gov/docs/PRs/27629/coverage/heat_transfer) - [`level_set`](https://mooseframework.inl.gov/docs/PRs/27629/coverage/level_set) - [`misc`](https://mooseframework.inl.gov/docs/PRs/27629/coverage/misc) - [`navier_stokes`](https://mooseframework.inl.gov/docs/PRs/27629/coverage/navier_stokes) - [`optimization`](https://mooseframework.inl.gov/docs/PRs/27629/coverage/optimization) - [`peridynamics`](https://mooseframework.inl.gov/docs/PRs/27629/coverage/peridynamics) - [`phase_field`](https://mooseframework.inl.gov/docs/PRs/27629/coverage/phase_field) - [`porous_flow`](https://mooseframework.inl.gov/docs/PRs/27629/coverage/porous_flow) - [`ray_tracing`](https://mooseframework.inl.gov/docs/PRs/27629/coverage/ray_tracing) - [`rdg`](https://mooseframework.inl.gov/docs/PRs/27629/coverage/rdg) - [`reactor`](https://mooseframework.inl.gov/docs/PRs/27629/coverage/reactor) - [`richards`](https://mooseframework.inl.gov/docs/PRs/27629/coverage/richards) - [`scalar_transport`](https://mooseframework.inl.gov/docs/PRs/27629/coverage/scalar_transport) - [`solid_mechanics`](https://mooseframework.inl.gov/docs/PRs/27629/coverage/solid_mechanics) - [`solid_properties`](https://mooseframework.inl.gov/docs/PRs/27629/coverage/solid_properties) - [`stochastic_tools`](https://mooseframework.inl.gov/docs/PRs/27629/coverage/stochastic_tools) - [`thermal_hydraulics`](https://mooseframework.inl.gov/docs/PRs/27629/coverage/thermal_hydraulics) - [`xfem`](https://mooseframework.inl.gov/docs/PRs/27629/coverage/xfem)

This comment will be updated on new commits.