Add tracer particles to phoebus

[AstroBarker]

I believe that the tracer treatment is now mature enough to have more eyes on it. It's a pretty big PR and has been awhile coming, so interact in whatever level you feel.

Summary

Here I add tracer particle support in phoebus. We create a tracer package that, if enabled, registers the appropriate swarm variables. See src/tracers/tracers.cpp. At present, the tracers are coupled in first order operator split, though in the future I'd like to explore improving this, after some phoebus task list refactoring. Tracers are advected in AdvectTracers(...), interpolating the appropriate quantities to the tracer positions and then pushing (with RK2). The "filling" of the tracers is handled in FillTracers registered with UserWorkBeforeOutput. To enable this, I added phoebus::UserWorkBeforeOutput to the driver. It's a lot of quantities.. and may desire to shorten later, or otherwise make what is stored runtime modular?

To deal with black hole accretion, when FMKS geometry is enabled, we use PurgeParticles in src/fixup/fixup_particles.cpp If FMKS is not enabled, the constexpr if resolves to a no op. In principle, here, future "purge particles if.." conditions can be supplied.

At present, I have implemented tracers in the torus and advection pgens. The advection test yields relative errors on tracer positions of ~ 1e-14 after one period. Tracer orbits in a non-magnetized torus work as desired.

To run with tracers (when the pgen supports it), simply

<physics>
tracers = true

<tracers>
num_tracers = ...

TODO:

• [x] Add tracers into advection, at least. Good to have a simpler-than-torus test problem.
• [x] Do 3D, non-magnetized torus orbit test.
• [ ] Unit & regression test.

cc: @bprather

[AstroBarker]

re: tests failing, I had some old testing code in radiation_advection and radiation_equilibration problems that didn't work with tracers disabled. I've just reverted those files to pre-tracers, and will put tracers in the fluid advection problem.

[AstroBarker]

Some updates:

• I have refactored the torus pgen to use rejection sampling. It is cleaner now, and doesn't fail on GPU. As implemented, it should be uniform in volume.
• I have split the tracer rhs calculation into a separate function
• Tracer integration is now RK2, as in nubhlight
• Added Bernoulli parameter - u^0 h - 1 to tracer output.
• Tracers are implemented into the advection pgen
  • Implementation here could be improved. As is, we get nblocks * num_tracers particles. Should just move to post init...

In the advection test, relative errors on tracer positions was ~ 1e-14 or lower after one period. I also ran a 3D, non-magnetized torus to see tracer orbits (notes: below only goes to t=500.. longer is running. And this was before the upgrade to RK2)

[Yurlungur]

Looks awesome! When do you think you'll be ready to remove the WIP markings? :)

[AstroBarker]

One last change: I've moved the FillTracers call from PostFillDerivedBlock into UserWorkBeforeOutput. To accommodate this, I created phoebus::UserWorkBeforeOutput in the driver and registered it in main. At present it just unpacks MeshBlockData and calls FillTracers, but it could accommodate extension in the future. Should cut down on unnecessary work. Let me know how this bit looks @Yurlungur.

(Also fixed something I forgot to commit in the torus init..).

[Yurlungur]

Actually @AstroBarker before merge, please correct the missing metric factor in the volumes in the torus calculation.

[Yurlungur]

Since @brryan never had a chance to look. I'm going to click the button. Ben whenever you get the chance, I'd still like your thoughts.

[Yurlungur]

Also thanks for all your hard work on this @AstroBarker !