sxs-collaboration / spectre

SpECTRE is a code for multi-scale, multi-physics problems in astrophysics and gravitational physics.
https://spectre-code.org
Other
158 stars 187 forks source link

Make horizon finder faster for GPUs #6200

Open geoffrey4444 opened 1 month ago

geoffrey4444 commented 1 month ago

Geoffrey's raw notes from the discussion.

08/07/24

Horizon finding chat at SXS Con 2024

Harald whiteboard talk

Shape map + control systems

Worldtube method: position evolving based on self force effects: $\ddot{\mathbf{x}}p = \frac{1}{m} \mathbf{F}{GSF}$. (GSF=gravitational self force) Implemented: he chose the charge at the center of the sphere... charge is at the center. Mapped with control system with rotation and scaling to charge moving in inertial coordinates.

Niko has removed the control system, implemented ODEs for the mapping parameters

\left(\begin{matrix}\ddot{\phi} \\\\ \ddot{a}\end{matrix}\right) = f(\mathbf{F}_{GSF}).

Idea: define approximate position of horizon as grid coordinate sphere, radius $R$. Map via $r_{\ell m}$ to the inertial shape of the horizon. Idea is to determine

\dot{r}_{\ell m} = - (f\theta)_{\ell m}

from flow equation of apparent horizon. If this converges quickly enough, using real time instead of artificial time should still keep the surface tracking AH, at least approximately... you can put an extra timescale in, to make the tracking more accurate... add a constant to RHS $\ref{eq:blah}$

$$ \dot{r}_{\ell m} = - (f \theta)^{\ell m} $$

... no separate AH finders anymore for control systems... initial guess for approximate horizon finder...

Interpolation always happens at the same grid points.

Nils: I'm not worried so much about interpolation; 1 interpolation/timestep is fine

![[Pasted image 20240807162235.png]] Mark: is this too restrictive, having horizon and excision surface both coordinate spheres? You still have size control

Mark: in flow equation, there's also a 00 equation... because then you can't change $R$ ... Harald: yes you can

Nils: for gpus, we need to find the horizon faster...that's it

H.: can you do punctures in GH? Nils Deppe: You must rederive if system is still hyperbolic, and there might be an issue with punctures being in the domain with spectral methods Do you have to to the conformal decomposition? Wolfgang Tichy+ have tried GH conformal decomposition to do punctures, and their code is currently blowing up, not sure if it's at the DG FD boundaries because their DG+FD scheme is unstable, or if the equations just can't be solved... now implementing in CarpetX to see if it's the equations or their numerical scheme

Mark: spin 0.99 on both holes aligned, equal mass, average of 5-6 iterations on AhA / AhB every time in SpEC... for q=4, spin=0, at most 2 iterations

Larry: current tolerance on the AH finder? Number of iterations in AH finder is a discrete number...

Nils remarks

We need...can we get these easier?

Nils: Stateless size!! this is going to limit our convergence at some point

Harald: I worry that we are tying spectre to one specific way of doing things; reduces flexibility... ties AH finder and control systems together on a deep level

Harald: Different modes converge at different rates; $\beta$ controls how much you overrelax the normals.

Discussion

Larry: another speedup that helps both: current algorithm has L representation and L to which you interpolate in order to solve the problem... currently Lmesh is 3/2 LAH. You can make that L+2 or +4 instead of 3/2 LAH? Harald: I remember having lots of AH failures with little difference in resolution

The issue for GPU code?

Now need to find the horizons and update the control systems... I need to do horizon find faster than GPU takes to take a timestep...

Nils Deppe: this might be hard, but I think it would solve this

Larry: multiple AH finding methods; FastFlow was added to SpEC in 2006, because it beat any finder known then...but it's limited in that it's adapted to spherically shaped horizons... and the expansion sign has to change correctly... there are other algorithms that if you have infinite speed on GPUs that might work better... usually they are global, return a scalar function and solve an elliptic equation for the function; this function's zeros are the horizon... after solving it everywhere, you have to find the zeros. The challenge was the elliptic solve.

GL: Elliptic solves on GPUs? Yes, that's machine learning Parabolic solutions? CFL condition much worse... First thing is how slow is the AH find...

Larry: if you choose $\beta$ wrong, it diverges, so if you catch this you can just lower $\beta$ and try again

Larry: This won't work for AhC

Overall options:

geoffrey4444 commented 1 month ago

image