Closed AlexLipp closed 1 year ago
If we decide to pursue this in we should change obs_mean
to be the geometric mean of the data not the arithmetic mean as it currently is.
I'm okay going with this for now as I don't have better ideas yet of how to combine log-ratio regularization with the log-ratio primary terms. If we decide to stick with it for the long-term, I should probably go back and remove the sample adjacency/border length code.
Agreed on this. The specific attachment to smoothness as opposed to model size is a legacy from previous ways of solving in any case.
@AlexLipp : Could you also rebase to fix the merge conflict?
@r-barnes I've incorporated the suggested changes and rebased to main
We currently regularise by penalising the roughness of the proposed solution, considering adjacent nodes. However, this considers only absolute difference between nodes when relative deviations are desired. This current method produces results which thus allow for significant overfitting of low-magnitude observations (i.e., orders of magnitude below one) which is undesirable for fitting (for example) trace element concentrations. An example of these undesirable outputs are shown here:![old_regularization](https://user-images.githubusercontent.com/10188895/205090502-b5c34f28-09b3-4fdc-8aa7-649ebccb4865.png)
In this PR, I instead penalise relative deviations of each variable from the mean of all observations. This permits slightly 'rougher' solutions, but inspecting the results, the results are elsewise much better conditioned. Specifically, no longer do we see outputs spuriously spanning many orders of magnitude below, as shown in the example below which is same as example above but with new regularization method:
This PR removes the need for an adjacency matrix which also simplifies the code.