The EnKF is a bit like particle filtering, and takes $O(P S)$ time,
where $S$ is number of samples and $P$ is number of latent states (parameters),
but it uses Kalman updates to update the posterior mean samples instead of importance sampling.
For details, see sec 8.5.4 of my book 2.
EnKF can be applied to DNN training, as shown in the paper below.
This is an interesting alternative to LoFi since it is derivative free, but cost is still linear in $P$.
However it will be higher variance, since it is a stochastic method and ignores gradient info, and will therefore
likely need larger $S$ to get good results.
The EnKF is a bit like particle filtering, and takes $O(P S)$ time, where $S$ is number of samples and $P$ is number of latent states (parameters), but it uses Kalman updates to update the posterior mean samples instead of importance sampling. For details, see sec 8.5.4 of my book 2.
EnKF can be applied to DNN training, as shown in the paper below. This is an interesting alternative to LoFi since it is derivative free, but cost is still linear in $P$. However it will be higher variance, since it is a stochastic method and ignores gradient info, and will therefore likely need larger $S$ to get good results.
A. Yegenoglu, K. Krajsek, S. D. Pier, and M. Herty, “Ensemble Kalman Filter Optimizing Deep Neural Networks: An Alternative Approach to Non-performing Gradient Descent,” in Machine Learning, Optimization, and Data Science, 2020, pp. 78–92, doi: 10.1007/978-3-030-64580-9_7 [Online]. Available: https://juser.fz-juelich.de/record/889208/files/main.pdf https://github.com/alperyeg/enkf-dnn-lod2020.