Locally time-invariant distance metric for Bayesian Model Averaging

[rolyp]

Used in climate model evaluation. Potentially interesting because it (non-monotonically) aligns data points from different time series, and so could be a good example for linked inputs. Could contrast with other “monotonic” approaches such as Dynamic Time Warping.

See the paper.

• [ ] [Roly] Read paper
• [x] [Joe] Read paper
• [x] [Roly] Email Mala Virdee for link to repo
• [ ] Identify steps involved and what we might be able to show/reveal using linked inputs/outputs (see below)

Details of the algorithm

• Uses a bipartite matching algorithm to calculate $\mathsf{arg\ min}_{\pi_w}$
• In turn might require representation of the problem as a cost matrix (similar to adjacency matrix)
• BMA step is separate, not sure what implementation would be required there
• Preprocessing steps:
  • “mean-shift” bias correction
  • projecting out time series for grid cells corresponding to specific locales

Thoughts on what we might be able to actually show

• Can we do much more than just show how big the “permutation window” is?

[rolyp]

Some initial notes on algorithm. Emailed first author for link to repo.

[rolyp]

Added notes on couple of the preprocessing steps (that we could either ignore by only considering the output of those steps, or include in order to illustrate something else).

[rolyp]

Now we have the repo, I guess one thing to try to answer soon is how much of the BMA algorithm we can likely implement in Fluid (and if the answer is “not much”, what the alternative is).

@JosephBond It might be useful to start enumerating the key components (e.g. scipy.optimize.linear_sum_assignment) and thinking about what it would be realistic to implement.

[JosephBond]

Currently, it doesn't seem like there will be any inherent issues with implementing this in Fluid, with the caveat that we may need to introduce a method to draw random numbers in the language. This would be a good first effect to add to Fluid if we need to, as it doesn't in theory add any complications relative to slicing (as we slice against a given execution). I will mark the subtasks we will need in terms of understanding, as well as in terms of implementation here.

Understanding

• [ ] Clearly define separated components of the metric + BMA computation
  • [ ] metric.py
    • [x] root mean squared error - easy
    • [ ] band matrix - matrix with 1's on the diagonal (and offset from diagonal), Infty everywhere else (use an adt: IntInfty = Int x | Infinity
    • [ ] threshold_cost - easy, just filter list of datapoints down according to threshold, then apply RMSE
    • [ ] reorder - most complicated, due to matrix ops (computation of cost matrix seems odd), and the use of a scipy library method (linear sum assignment). This requires most investigation
    • [ ] Linear sum assignment is the place we need to perform the bipartite matching. Could be computed via a foreign function, but I would like to try and figure out how to do it in a functional language, the scipy version implements the algorithm described here
• [ ] Identify the aspects of torch we can easily reproduce
  • [ ] 3 types of distribution involved
  • [ ] identify aspects of neural network that we can elide
• [ ] Can we directly implement the optimization scheme (EM algorithm)?

  Implementation

• [ ] Prototype in purescript
  • [x] RMSE
  • [ ] Band Matrix
  • [ ] Threshold_cost
  • [ ] Reorder
  • [ ] LSA
• [ ] re-implement prototype in Fluid

[rolyp]

Dropped for now (see #756).