TODO 09/24

[apatlpo]

keeping track of email exchange entitled "Re: drifter inference - update"

[apatlpo]

Andrew

• So fixing the mooring locations with seed didn’t seem to reduce that noise on the MAP? correct

• I started looking here from the bottom (from small n to large n), because we should be very confident that we can’t say anything about the spatial scale for one and two moorings (need min three moorings because there is an implicit lambda_y)? we should indeed even though the geometry of the mooring relative to the spatial scale may modulate the performance I assume that the posterior dist’ seen here on lambda_x is just the prior? And the overestimate in \eta is due to us only sampling a small portion of this 500 km eddy (i.e. the full unsampled eddy would have a greater total variance than the small portion we are observing)? There must be some trade off as to which of lambda_x and eta are jammed towards their prior, and which one compensates for the error.

• Do you mind sharing a histogram of the priors? I had a look at your notebook but I think they are buried in /home/datawork-lops-osi/aponte/nwa/drifter_stats/3D_matern32_iso_matern12_pp_r0.0_u0.2_flow.zarr. done

• Either way, as we increase the number of n the eta and lambda_x posteriors do seem to converge faster around alpha=2. Supports your idea that for ‘large’ n, we don’t see the alpha sensitivity as it has already converged. yep

• Regarding convergence - is there a way to normalise the eta posterior to remove the imposed trend and thus show this convergence more clearly? Using true eta(alpha) as the offset is sensible, but I’m not sure what to use for the ‘width’. yes we could, I have not done it yet as wonder if we shouldn't. Let's just remember it for now, I'm not sure it's critical at this stage

• Thinking about the relative ability to resolve lambda_x and lambda_t for n=2 in the drifter case - do we also need a way to characterise the density of the drifters/moorings with respect to lambda_x? yes I do believe this is an important point, in general: how do we quantify the number of data points available when drifters may generally distributed over space and time

[apatlpo]

Lachy

• I’ve just found this (https://arxiv.org/pdf/2302.10364.pdf) which is either problematic or useful. Essentially, they beat us to writing down a lot of the math, but they’ve missed a lot of the oceanography and don’t have any of the spectral links. The silver lining is it will allow us to cite out some of the math for the GP and then focus on the oceanography. (The majority of them are GP kids who I think are doing this for fun, I doubt they have plans to seriously look at the applied science.) I agree in general but am surprised the little attention is paid to the inference part of the problem

The statistical noise in the fits can be to a few things:

1. Sometimes the true parameters, and the effective parameter from the sample are different. So, you set, say, lambax = 2 and sample a field. For a small amount of data sometimes the genuine best fit is not at lambax = 2. The way you can see if this is a problem is run repeated experiments with different samples of the field. understood, this would unfortunately increase the computational burden if we had to go around this
2. I’m increasingly uneasy that emcee is just doing a bad job at fitting. I’m in Lancaster at the moment and I’ve got Dodd to get the generalised Matern into GPJax. We could try and write the inference in GPJax to see if emcee is giving us the right answers or not. (I could also just knock up a quick fit for purpose MH sampler if we’re set on the number of parameters we’re using.) to be discussed along with that I may be keen in general to hand over the inference part to an expert

Are the mooring and drifter locations the same between each of the three batches of experiments? If not, this would contribute to the noise. I think they are different between batches (they necessarily are to some extent given their inequal length) but would have to double check

I think that out of these things 2) is the problem. The main thing that suggests this to me is the times where the MAP is outside of the predictive interval. I’ve never seen this happen in any sort of real statistical problem and so it suggests to me that the sampler is cooked.