v1.2

[joshspeagle]

The recent set of PRs introduces enough performance changes/improvements that it's worth looking into another stable version release, along with some updates to the documentation. I should look into doing that in the near-term future.

[segasai]

Yes, I was about to suggest the same. Hopefully with the final slice sampling fix there should be significant improvements over 1.1.

[joshspeagle]

My thoughts exactly!

[segasai]

on the topic of the release, (as I think we are mostly ready), I think it would be nice to put a new one on zenodo as well (so the new release could be specifically referred to.

[joshspeagle]

Given the number of improvements that you've made since v1.1 (plus the bugfix), I'm wondering if you think things are stable enough to cut a v1.2 release soon-ish.

[segasai]

I think they are. I've been running some of my science processing through it (using rwalk in particular) and it seemed to be working fine. I've just started another run of 100 tests in parallel, to see how we do there. I also started the high-dim tests, like this import test_highdim as T; T.do_gaussians(sample="rwalk")' >highdim_rwalk.log to see that there is no regression (that's how I discovered that rwalk behaves like rstagger -- because the curve of bias/sigma was identical for them) I think it's worth writing somewhere a list of checks to be done before a release (that can include things that need to be run longer), as the test suite is still pretty limited.

I've had another change I wanted to introduce https://github.com/joshspeagle/dynesty/compare/master...segasai:results_branch?expand=1 changing the results object so it'd be a proper object (rather than just a dictionary), so one could for example determine whether it was a dynamic run/static run without resorting to trying to check for the existance of attributes. But we can probably discuss/postpone that till after 1.2.

[segasai]

It's also probably worth to put an editable draft release notes listing all the changes, as there was a lot. And also usually I haven't touched the docs much, but I think its' worth to go over items in the notes and check there are no contradictions. One thing I've doing periodically is making sure the jupyter notebooks run (it's probably worth adding to the release checklist)

[joshspeagle]

I've had another change I wanted to introduce...changing the results object so it'd be a proper object (rather than just a dictionary)...But we can probably discuss/postpone that till after 1.2.

That sounds like a good change but I agree that should be postponed for the future.

It's also probably worth to put an editable draft release notes listing all the changes, as there was a lot.

Good idea. I'll list a summary here before updating the documentation.

I think its' worth to go over items in the notes [docs] and check there are no contradictions.

That's on my to-do list, yes.

One thing I've doing periodically is making sure the jupyter notebooks run (it's probably worth adding to the release checklist)

Those are quite out of date now yea, so I should rerun them and update the files both on Github and in the docs as needed.

[segasai]

Here is the quick (possibly incomplete) scan of the git logs with at least some noteworthy points for the release

• More testing + validation
• Saving likelihood
• Many speedups (ellipsoid bounds, bootstrap, jitter_run)
• Pointvol fixes
• Internal refactoring reducing duplication (saved_run, integral calculations, different samplers etc)
• Make dynesty fully deterministic if random state is provided (also migrate to new numpy random generator funcitonality)
• Fix bugs in dynamic sampling that leads to sampler not finding points in the interval ( #244)
• Improve bounding ellipsoids algorithms #264, #268
• Major refactor of rslice/slice sampling increasing its stability
• disable ncdim when slice sampling
• change the defaults for slices/walks/bootstrap (vs ndim)
• change default samplers (vs ndim, i.e. use rslice for highdim)
• remove rstagger
• fix rwalk. Before the chains were not markovian
• change adaptation of rwalk and rslice
• get rid of vol_dec parameter
• more stable multi-ellipsoidal splitting using BIC
• make uncertainties smaller (more correct) (remove factor of two)
• fix update_interval
• exception if unknown arguments in nestedsampler/dynamic
• fix use_pool:loglikelihood
• get rid of kl_divergence f-n
• refactor the addition of batches in dynamic sampling, preventing possibly infinite loops

[segasai]

Just bumping this. I think it would be good to move forward with this, as I do believe that the main branch is significantly improved compared to 1.1. Since I don't have commit rights, I can't really work on this on my own efficiently (also the semester started, so I have less time ).

[joshspeagle]

(also the semester started, so I have less time ).

Same here -- this is the main reason for the delay on my end. Rest assured I do have this on my near-term to-do list!

Since I don't have commit rights

I could probably give these to you going forward if that would make things easier, especially given the size/scope of your commits (which have been super!). Would that be helpful?

[segasai]

Same here -- this is the main reason for the delay on my end. Rest assured I do have this on my near-term to-do list!

Sure

Since I don't have commit rights

I could probably give these to you going forward if that would make things easier, especially given the size/scope of your commits (which have been super!). Would that be helpful?

I think it would be helfpul. I'll aim to directly commit only small fixes/changes, while anything substantial would go through PRs.

[wenhaoxuan]

Yes, I was about to suggest the same. Hopefully with the final slice sampling fix there should be significant improvements over 1.1.

Does the upcoming 1.2 version fix the issue #344? I just posted that, but wonder if you are referring to a similar issue.

[segasai]

@joshspeagle Okay, I propose to merge the docfix branch and not delay the release before more documentation edits, as the benefits of releasing significantly improved release (i.e. on issues with rwalk,rstagger and rslice) outweigh the cons of somewhat outdated discussion of the uncertainties in the docs.

IMO the things to be done then

• Write release notes (see post above), additionally highlighting the user-facing differences. (I can do that)
• git tag the release
• Create a github release
• Upload the version on zenodo and the citation info to the documentation
• upload version to pip. maybe updates to AUTHORS.md

[avivajpeyi]

Hi! when is this planned to be released? Id really like to use the fixed rstatefeature

[segasai]

I'm afraid we are waiting for @joshspeagle - I can't upload to pypi & stuff. otherwise you just need to use the master branch which is stable (that's what I use in production)

[avivajpeyi]

Hey @joshspeagle -- would it be possible to get 1.2 out soon? Happy to try and help if you need some more hands on deck :)

We (Bilby + Parallel Bilby) would really like to use dynesty for LIGO GW analysis in the next observing run -- but will need some time to update from our current pinned version to 1.2

[segasai]

Hi @avivajpeyi

I didn't manage to get any reply from Josh recently, I tried again a couple of days ago. I think I have the right permissions to branch a github release, create a tag, but I can't release on pypi.

S

[asmasca]

Hi, I'm just commenting to say that I recently updated to the github master (coming from v1.0.1) and now rslice sampling seems much more stable. I'm also seeing a higher CPU usage (on a 32 threads CPU) and less instances of most workers idling while they wait for 1-2 workers to finish. In general I'm seeing a 3-4x time reduction in my basic test case (with equivalent results).

Thanks for all the work in this fantastic code!

[segasai]

The version 1.2.0 has been finally released! It is available on pypi and on github. See the changelog for the detailed list of changes.