Open danjenson opened 1 year ago
The log-likelihood computation raises the same exact memory issues than the Deterministic you had in the other issue.
It's going to be a chains draws len(dataset) array of float64 numbers. That's exactly as large as the Deterministic that was consuming the whole RAM.
This is not uncommon and it's one of the reasons we don't compute it hy default. I've had to resort to libraries like dask to compute this quantity without running out of RAM (as well as the subsequent model comparison statistics in arviz).
There's a related discussion here: https://github.com/pymc-devs/pymc/discussions/5371
Ok, so is there a workaround? I see the discussion but it seems inconclusive. It seems like the primary solution is to cultivate a much smaller dataset so I don't run out of memory?
There is no workaround yet. The long-term solution is to allow the log-likelihood to be computed in dask (pymc-side), as well as the model comparison statistics (arviz-side), so that you can perform model comparison on large datasets. CC @OriolAbril
For now you can try to work with a smaller dataset or less chains/draws if the subset converges equally well as the default number of chains/draws
if your likelihood is easy-ish to compute by hand using distributions, you can also use xarray-einstats to help the process which is already compatible with dask. The process should be similar to https://oriolabrilpla.cat/en/blog/posts/2022/einstats-hmm-cmdstanpy.html but using logpdf
method instead of rvs
.
There is also a PR open in ArviZ to compute loo/waic using dask, but it needs testing and don't have much time to dedicate to it now. If anyone can try it out it would be extremely helpful. https://github.com/arviz-devs/arviz/pull/2205
Describe the issue:
Process memory grows steadily while computing log likelihood until it consumes all available memory (and swap). Replicated on linux and M1 Mac.
PYMC version: 5.7.2
Linux system:
Void Linux Kernel 6.3.12_1 64 GB DDR5 RAM (64 GB SWAP) 24 GB RTX 4090 GPU AMD Ryzen 9 7950X 16 core, 32 threads
Mac System:
16 GB memory 8 Cores
Dataset: ~161 mb total.
Reproducible code example:
Error message:
PyMC version information:
PYMC version: 5.7.2
Context for the issue:
Trying to use this with
arviz.compare(...)