smoke_test_injection_3d fails health check before first step

[matthiasdiener]

smoke_test_injection_3d fails the health check before the first step:

Pressure range violation (p_min=2552.188953324666, p_max=274274.3979008766)
Fluid solution failed health check.
Errors detected; attempting graceful exit.
Traceback (most recent call last):

The error happens for single-rank eager and lazy, with the current main branches (except mirgecom, which is on the production branch).

Pressure (and temperature) eventually become negative, after about 8 steps.

To reproduce:

$ git clone git@github.com:illinois-ceesd/emirge etest
$ cd etest
$ ./install.sh --branch=production --env-name=etest
$ source config/activate_env.sh
$ vim meshmode/meshmode/mesh/__init__.py  # disable node vertex consistency test in line 1200
$ git clone git@github.com:illinois-ceesd/drivers_y2-isolator
$ cd drivers_y2-isolator/smoke_test_injection_3d
$ python -m mpi4py isolator_injection_init.py -i run_params.yaml
$ python -m mpi4py isolator_injection_run.py -i run_params.yaml -r restart_data/isolator_init-000000

smoke_test_3d is also affected.

[MTCam]

✅ Reproduced with a weeks-old SingleGridWorkBalancing array context. Init seems to be working OK, but after the first step:

Pressure range violation (p_min=2528.236407762848, p_max=274270.23911909416)
health_pres_min: 2.65231e3
health_pres_max: 2.77231e5

I think we should just ignore this until @anderson2981 takes a look. In the meantime, as a workaround, the following changes to run_params.yaml will make the tests step without running into health issues:

-current_dt: 1.0e-9
-t_final: 2.e-8
+current_dt: 1.0e-11
+t_final: 2.e-10
-health_pres_min: 2.65231e3
+health_pres_min: 2.3e3