search

stan-dev / stan

Stan development repository. The master branch contains the current release. The develop branch contains the latest stable development. See the Developer Process Wiki for details.
https://mc-stan.org
BSD 3-Clause "New" or "Revised" License
2.6k stars 370 forks source link

Consolidate and refactor output from algorithms #2534

Open martinmodrak opened 6 years ago

martinmodrak commented 6 years ago

Summary:

This issues is to track progress on refactoring outputs. The design and other issues are discussed at forums: http://discourse.mc-stan.org/t/proposal-for-consolidated-output/4263/19

Description:

This will be detailed once the design is settled, currently the issue is useful for me to be able to relate it to other already filed issues

martinmodrak commented 6 years ago

I scanned the stan-dev/stan repo for open issues possibly related to output refactoring so that we keep an eye on them, see the use cases and hopefully resolve as much as possible while refactoring the code.

https://github.com/stan-dev/stan/issues/2474 (additional ADVI diagnostics) https://github.com/stan-dev/stan/issues/2473 (Switch to store Jacobian for optimization) https://github.com/stan-dev/stan/issues/2459, https://github.com/stan-dev/stan/issues/1641, https://github.com/stan-dev/stan/issues/2355 (the writers should let us detect when gen quants are actually output somewhere - especially to check against warmup/sampling) https://github.com/stan-dev/stan/issues/2515 (precision of CSV output) https://github.com/stan-dev/stan/issues/2432 (standalone execution of gen quants) https://github.com/stan-dev/stan/issues/2283, https://github.com/stan-dev/stan/issues/637 (writing NaNs/inf) https://github.com/stan-dev/stan/issues/2202 (extending status messages) https://github.com/stan-dev/stan/issues/2197 (reporting the result of optimization) https://github.com/stan-dev/stan/issues/2198 (reporting errors/warnings in machine-readable form) https://github.com/stan-dev/stan/issues/2184 (a probably conflicting proposal for minor writer refactor) https://github.com/stan-dev/stan/issues/2165 (extracting transformed data from model - also related to https://github.com/stan-dev/stan/issues/2360) https://github.com/stan-dev/stan/issues/2070 (different rules for status reporting frequency) https://github.com/stan-dev/stan/issues/1663 (output from ADVI upon interrupt) https://github.com/stan-dev/stan/issues/1661, https://github.com/stan-dev/stan/issues/1786 (extended ADVI outputs) https://github.com/stan-dev/stan/issues/1580 (exception reporting) https://github.com/stan-dev/stan/issues/1321 (more discussion of problems with output) https://github.com/stan-dev/stan/issues/1264 (buffering output) https://github.com/stan-dev/stan/issues/519 (allow compressing output)

martinmodrak commented 6 years ago

Here is a list of relevant open issues in RStan https://github.com/stan-dev/rstan/issues/536 (changing the way samples are stored) https://github.com/stan-dev/rstan/issues/156 (discusses output options) https://github.com/stan-dev/rstan/issues/49 (suppress output from RStan) https://github.com/stan-dev/rstan/issues/41 (consolidating between sampling and optimizing output, probably a bit outdated) https://github.com/stan-dev/rstan/issues/519 (large gen quants can be slow to transfer) https://github.com/stan-dev/rstan/issues/515 (handling inits of vb) https://github.com/stan-dev/rstan/issues/438 (storing warnings in stanfit object) https://github.com/stan-dev/rstan/issues/272 (outputting ELBO in ADVI) https://github.com/stan-dev/rstan/issues/161 (access properties of a model even when Stan run fails).

martinmodrak commented 6 years ago

And here is a list for other interfaces (0 relevant issues were found in StataStan, MatlabStan, MathematicaStan, httpstan and Stan.jl) https://github.com/stan-dev/pystan/issues/418 (getting more info out of optimizing calls) https://github.com/stan-dev/cmdstan/issues/619 (filtering params to report by prefix/regex) https://github.com/stan-dev/cmdstan/issues/547 (report inits) https://github.com/stan-dev/cmdstan/issues/511 (allow output as RDump)

bob-carpenter commented 6 years ago

@martinmodrak Thanks so much for putting this list together.

@syclik and @seantalts: Is this an OK use of issues? I've been trying to restrict them to just actionable, well-defined issues, not general discussion, but there's no good way to track multiple issues.

So I'll leave this here for now pending advice on how to deal with this kind of thing going forward.

syclik commented 6 years ago

Thanks!

I think it’s fine for now. There are less active issues that we can cull before this one. This one is active and serves a purpose, so I’d prefer keeping it until we have explicit process.