search

zenna / Omega.jl

Causal, Higher-Order, Probabilistic Programming
MIT License
164 stars 17 forks source link

Parallel Replica Exchange #220

Closed SamWitty closed 2 years ago

SamWitty commented 2 years ago

Added Julia multiprocessing and more extensive testing.

SamWitty commented 2 years ago

Thanks for the feedback/requests.

Re: tests being slow, I think this is something we'll need to think a bit more about when testing approximate inference methods. The current test suite includes some pretty forgiving assessments of inference "correctness": (i) it only tests a single inference run for each of the two target densities, as opposed to the many many runs required by methods like SBC and (ii) it only requires that a handful of summary statistics (mean, variance, proportion of samples in positive quadrant) approximately match the truth. Even so, I'm finding it very difficult to get the tests to pass even these permissive assessments with much fewer iterations than are currently in the test suite.

As far as I see it we have a few different options:

  1. Remove "inference correctness" tests altogether, and simply test whether the inference method returns the right kinds of objects with the right kinds of inputs.
  2. Keep the test suite as-is, or even it slower by making the tests more principled.
  3. Stage the tests so that "inference correctness" tests run less often and standard "does it run" tests run on every commit to master.

My preference is 3, but I'm not sure how involved it is to change Omega's CI/CD workflows to make that happen.

Thoughts?

SamWitty commented 2 years ago

@zenna. I've now addressed your requests, although it's not clear that I've set up the @ref reference in the documentation correctly. Also, all of the tests regarding inference performance have been commented out, as they all fail when using a small number of iterations. These tests pass when using a large number of iterations, but as you mentioned earlier, this bogs down the CI pipeline. See my earlier comment for more discussion and suggestions.

I will reach out separately to schedule a time for us to synchronously meet and discuss the right path here.

zenna commented 2 years ago

Nice work