Release 2.30 - Githubissues

rok-cesnovar commented 2 years ago

Feature freeze (21st of June):

[x] Ensure all expiring language deprecations have been removed or given new removal dates.
[x] Create and merge version updating pull requests in Math/Stan/Cmdstan. These should be the last PRs accepted before the freeze.
[x] Create Math/Stan RC releases.
[x] Create Stanc3 RC binary.
[x] Create a release candidate tarball for x86. Make sure RC tarballs include stanc3 binaries.
[x] Check external links in docs (i.e. TBB docs link) - use https://github.com/tcort/markdown-link-check
[x] Create a release candidate feature/bugfix list (major features/bugfixes that need testing, link to new docs in Github)
[x] Run CmdStanR tests with the RC tarball.
[x] Run CmdStanPy tests with the RC tarball.
[x] Make a Discourse RC post.
[x] Post a tweet with a link to the Discourse RC post.

Release (10 days after the RC):

[x] Create the Math Release notes.
[x] Create the Stan Release notes.
[x] Create the Cmdstan Release notes.
[x] Create the Stanc3 Release notes, include new deprecations and removals.
[x] Rebuild and publish docs for the new version.
[x] Check that docs for the previous release links correctly to the newest docs.
[x] Create the Math release.
[x] Create the Stan release.
[x] Create the Stanc3 release.
[x] Create x86 CmdStan tarballs (check version, check that the extracted folder is in the cmdstan-version format).
[x] Create non-x86 CmdStan tarballs.
[x] Create conda CmdStan tarballs.
[x] Run CmdStanR tests with the release tarball.
[x] Make a Stan blog release announcement post (thank the sponsors and all contributors, mention new devs).
[x] Link to the blog post in a Discourse thread.
[x] Make a Twitter announcement.
[x] Update https://github.com/stan-dev/stan/wiki/Coding-Style-and-Idioms#supported-cpp-versions-and-compilers

Things to highlight in the release notes:

[x] The Windows RTools35 toolchain will no longer be supported

rok-cesnovar commented 2 years ago

The RC is currently set to this coming Thursday, but if it helps anyone we can move it to Monday, which is our typical "release" day. Let me know.

WardBrian commented 2 years ago

It would be great to get any of https://github.com/stan-dev/math/issues/2699 addressed before release.

I think we will need to add a message to the release notes about “Please let us know if functions you expect to work with complex matrices are not yet supported so we can prioritize what to work on” or something like that anyway, but those should be fairly obvious ones

WardBrian commented 2 years ago

I also think we should look at https://github.com/stan-dev/cmdstan/issues/1057 - @rok-cesnovar how difficult would it be to improve the error messages for this?

rok-cesnovar commented 2 years ago

On it! Been promising that one since January :)

WardBrian commented 2 years ago

I think that one is as simple as

else {
        throw std::invalid_argument(
            "num_chains can currently only be used for HMC with adaptation engaged");
      }

following the if (is_hmc && adapt_engaged) {

rok-cesnovar commented 2 years ago

lol yeah

rok-cesnovar commented 2 years ago

Barring anything unexpected, the RC will be released on Tuesday. The only thing left open currently is the std_normal_qf I believe.

rok-cesnovar commented 2 years ago

@serban-nicusor-toptal the current develop (master for stanc3) is what we want to release for the 2.30 RC. Whenever you are free tomorrow, feel free to start with the process. No rush.

I will make an RC announcement with some highlights until sometime afternoon CET when the US folks get up.

rok-cesnovar commented 2 years ago

The draft of the RC announcement is here: https://gist.github.com/rok-cesnovar/3a1419b665f4b09c26f2e56995938c81#file-stan-2-30-rc-announcement

Would appreciate any comments.

serban-nicusor-toptal commented 2 years ago

Hey @rok-cesnovar I've created the PRs to update the versions:

Would you mind taking a look and approve? Should we also skip waiting for CI and merge straight away so I can do the RC ?

rok-cesnovar commented 2 years ago

Thanks! Go ahead and merge them.

serban-nicusor-toptal commented 2 years ago

I am now starting with the RC, most of it shouldn't take long but we will need to wait a bit for stanc3 binaries to be built. Will reply here when all done.

rok-cesnovar commented 2 years ago

We are not under any time pressure, so whenever they are built its fine.

serban-nicusor-toptal commented 2 years ago

Release candidates are up now!

rok-cesnovar commented 2 years ago

Thanks. I am seeing a cmdstan-2.30.0-rc1.tgz and .tar.gz file (we typically only have the tar.gz one). And it seems as if they dont have the Stan sources inside?

WardBrian commented 2 years ago

The RC was not tagged as a pre-release so things like cmdstanpy which look for the latest version picked it up (and, due to the issue @rok-cesnovar notes, immediately failed)

serban-nicusor-toptal commented 2 years ago

Unintended mistake, I forgot to checkout submodules in cmdstan. Everything should be fine now, updated release candidates as pre-release, removed the .tgz and re-uploaded the .tar.gz

rok-cesnovar commented 2 years ago

Thanks, this seems to work. The RC announcement was posted on Discourse and Twitter. I am also running CmdStanR tests. I will check for broken links in the next few days.

WardBrian commented 2 years ago

Ran cmdstanpy tests here: https://github.com/stan-dev/cmdstanpy/actions/runs/2557546817

rok-cesnovar commented 2 years ago

Thanks! I also checked the links. All of them work, some just require an unnecessary redirect, that is addressed here: https://github.com/stan-dev/docs/pull/552

rok-cesnovar commented 2 years ago

@serban-nicusor-toptal you can start the release whenever you have a moment. There were a couple of fixes merged, so please release the current develop (Math/Stan/CmdStan) and master (stanc3) branches as the official release.

Release notes:

-----------
Math
-----------

Complex:
 - Added fast Fourier transform (FFT) implementations of 1D and 2D discrete Fourier transform and their inverses. (#2686, #2750)
 - Added type traits for detecting whether a type has a value type that is complex. (#2694)
 - Added vectorized versions of get_real and get_imag. (#2700)
 - Vectorized complex `abs()`, `to_complex`. (#2737, #2749)

New functions:
 - Added the incomplete beta function inverse. (#2637)
 - Added the `wishart_cholesky_lpdf`, which is the Cholesky parameterization of the Wishart distribution of both the input matrix and the scale matrix. (#2710)
 - Added the  inverse Wishart Cholesky parameterization `lpdf\lupdf` is available along with `inv_wishart_cholesky_rng`. (#2713)
 - Added the Cholesky parameterization of the multivariate Student's T distribution. (#2715)
 - Added the quantile function of the standard normal density function with log input. (#2744)

Misc:
 - Upgraded Sundials to version 6.1.1. (#2692)
 - Updated Boost to 1.78.0. (#2725)
 - Added namespace qualifiers to `size` and `apply` calls for C++17 compatibility. (#2693)
 - Enabled address sanitizer and fix it when running inside Docker in Flatiron CI. (#2702)
 - Made `arena_allocator` usable for all std containers. (#2708)
 - Added gradient calculation for the a2 input in stan-math for the `grad_2F1` function and fixes incorrect return values when `z` is between -1 and 0. (#2721)
 - Updated the signatures of `eigenvalues` and `eigenvectors` to use our pseudo-concept require templates.(#2728)
 - `norm1` and `norm2` were extended with the `apply_vector_unary` to accept general std::vectors as well as the Eigen vectors previously supported. (#2742)
 - Cleaned up compiler warnings for unused variables and non-consistent use of class and struct for ops_partials_edge. (#2757)
 - Fixed the check for `y` in `gamma_lpdf` so that it now errors if y is not a positive finite value. (#2758)
 - Improved numerical stability of gamma_lcdf gradients (#2767)

-----------
Stan
-----------

- Added `const noexcept` modifiers to the three accessors related to step-size (nominal, current, and jitter). (3124)
- Removed unused variable `num_params`. (#3126)
- Code cleanup - consistent set of includes for the set of services in stan/services/sample. (#3111)

-----------
CmdStan
-----------

- Fixed makefile for use of USER_HEADER with the bernoulli example model. (#1104)
- Added an error if `num_chains` is supplied for unsupported algorithms and methods. (#1102)
- Fixed handling of size-zero JSON arrays as input. (#1093)

-----------
stanc3
-----------

Complex:
 - Added complex containers: `complex_matrix`, `complex_vector`, and `complex_row_vector` with basic arithmetic support. (#1133)
 - Fixed indexing of a complex container to return the correct type. (#1158)
 - Added many common linear algebra functions and container operations for complex types. (#1166)
 - Fixed an issue reading in `complex_matrix` in the data block. (#1168)
 - Added vectorized signatures for `get_real` and `get_imag`, e.g. `get_real(complex_matrix) => matrix`. (#1169)
 - Added `fft`, `inv_fft`, `fft2`, and `inv_fft2` to the language. (#1170)
 - `pow` (and subsequently the `^` operator) now have overloads for complex containers. (#1186)
 - `to_complex` is now vectorized. This means signatures like `to_complex(matrix, matrix) => complex_matrix` are now available. (#1208)
 - Added signatures for addition, subtraction and division which mix scalars and complex vector types, broadcasting the scalar as appropriate. (#1212)
 - Added `svd_U`, `svd_V`, `singular_values`, `eigenvalues_sym`, and `eigenvectors_sym` for complex inputs. (#1212)

New functions:
 - Added `L1` and `L2` norms. (#1140)
 - Added `log_determinant_spd` which is faster when the input matrix is known to be symmetric and positive definite. (#1178)
 - Added `wishart_cholesky_lpdf` and `wishart_cholesky_rng` distribution functions. (#1188)
 - Added `multi_student_t_cholesky_lpdf` and `multi_student_t_cholesky_rng` distribution functions. (#1197)
 - Added `inv_wishart_cholesky_lpdf` and `inv_wishart_cholesky_rng` distribution functions. (#1202)
 - Exposes inv_inc_beta. (#1211)

Misc: 
 - Re-enabled function inlining optimizations at O1. (#1097)
 - Modified the optimizer to turn statements like `3 + 4i` into `to_complex(3,4)` directly. (#1112)
 - Added `cumulative_sum(Vector/RowVector)` for `var<Matrix>` types. (#1152)
 - Improved compile times by accumulating all template requirements into a single `require_all_t` on user defined functions. (#1157)
 - The `--info` and `--debug-generate-data` commands now use the Yojson library for it's output. (#1161, #1175)
 - Bugfixed for output values in `write_array` that should be `NaN`. (#1165)
 - `--dump-stan-math-signatures` now prints an entire signature on one line, no matter how long it is. (#1167)
 - Fix --debug-generate-data for ordered vector data types. (#1173)
 - Exposed an overload for `cumulative_sum` which takes and returns an array of integers. (#1174)
 - The javascript interface (stancjs) now generates errors which look more like stanc3's by pointing out where in the supplied code the error occurred. (#1177)
 - Cleaned up the internal structuring of modules in the compiler. (#1179)
 - Cleaned up code for pretty printing the Middle Intermediate Representation (MIR). (#1180)
 - Added `--debug-mem-patterns` flag, which will print out the memory type of variables (either Array of Structs or the more efficient Struct of Arrays pattern) when optimization is enabled. (#1185)
 - Allowed the adding of new distributions without requiring them to support deprecated suffixes. (#1191)
 - Added `eigenvectors` and `eigenvalues` functions which return the complex-valued eigenvectors or eigenvalues of a real (asymmetrix) matrix. (#1192)
 - `abs` is now vectorized for container types. `fabs` is now deprecated, as all usages can be replaced with `abs` thanks to array promotion rules. (#1195)
 - Made installing the developer tools for the Stanc repository an interactive script with error handling. (#1200)
 - Internally clean up how variable declarations are represented. (#1203)
 - Simplified internal representations in the compiler. (#1204)
 - The pretty-printer now preserves multiple declarations in one line like `int a, b;` rather than splitting them into two declarations. (#1207)
 - Declarations are now allowed in places they previously would not parse, such as the only statement in the body of an if statement. (#1207)
 - `./`, `/` `.*`, and `*` now have consistent behavior with scalar arguments. (#1209)

WardBrian commented 2 years ago

Removed for loop trick for inlining functions without multiple return statements. (#1097)

This title should probably be something more like "Re-enabled function inlining optimizations at O1", and I think the abs/fabs change should be higher up. Otherwise the notes look good to me!

rok-cesnovar commented 2 years ago

Thanks. Updated my post, also added the line for the Math bugfix of gamma_lcdf.

serban-nicusor-toptal commented 2 years ago

Hey @rok-cesnovar I'll try to do it later today or early tomorrow

rok-cesnovar commented 2 years ago

Ok, cool.

serban-nicusor-toptal commented 2 years ago

Hey @rok-cesnovar everything should be ready except docs:

2.30 documentation is currently 404 until the following PRs are approved & merged:

Please let me know if there's anything else I can do to help!

rok-cesnovar commented 2 years ago

Thanks. The tarball looks good on first look, locally everything seems to work fine. Running CmdStanR tests now.

WardBrian commented 2 years ago

Cmdstanpy tests running here: https://github.com/stan-dev/cmdstanpy/runs/7152748217?check_suite_focus=true

WardBrian commented 2 years ago

Docs look good to me

rok-cesnovar commented 2 years ago

There is some formatting issue here: https://mc-stan.org/docs/functions-reference/array-reductions.html#norms I am not sure exactly what is up. It's a minor issue though. Not something we need to delay anything for.

WardBrian commented 2 years ago

Looks alright to me? What am I missing?

rok-cesnovar commented 2 years ago

On refresh it did show the equations, seems just a temporary browser issue. Sorry for the false alarm.

rok-cesnovar commented 2 years ago

All tasks are done so I am going to close this. Thanks all.

stan-dev / cmdstan

Release 2.30 #1101