Funding Stan features

[spinkney]

An issue arose where a user offered to pay for a bounty to help get the feature included in Stan. We do not have a procedure for paying for features. This topic is to discuss how Stan should handle these requests and to make a public document on how a corporation or person can pay for feature development.

A few non-exhaustive options:

• Github bounties. This would require tracking bounties, typically by adding a 3rd party integration into Github such as https://issuehunt.io/. The benefit is that we may get more developers willing to work on the issue, the drawback is that the quality of the PR may come in to question resulting in a dispute over whether the contributor should be given the bounty.
• Paying directly to Stan for the feature. We would have a track which estimates the development cost and then pays a known Stan developer for their time. I have many questions on this route, like how does the estimation work? What if the feature is much more complex than originally thought? What happens if the feature does not make it in? I think many of these issues can be solved by explicit language in a work contract. Can we talk to Numfocus lawyers about getting a standard work-order for open-source software development?

[rok-cesnovar]

I don't see a problem if someone wants to fund a feature, but it has to be explicitly stated and in "all caps" that we will in no way guarantee a feature will be added just because someone funds it.

[mitzimorris]

I would say that bounties only make sense for features that would be implemented with or without a bounty. if a bounty involves lawyers, it's not worth the trouble.

there is one precedent: standalone generated quantities. we had a good relationship with the FB Prophet team. in 2017 or so they offered $25K bounty to Andrew Gelman at Columbia for this feature. it was a feature that many folks had asked for and was deemed generally useful. FB paid the $25K bounty to Columbia when the first version of this feature was implemented (by me), although that version was less than useful because it required input draws on the unconstrained scale. I reimplemented it so that it was properly useful. there was no contract - just some emails. subsequently, the FB Prophet team arranged a second gift of $100K, this time to the Stan NumFOCUS project.

[bob-carpenter]

To second what @rok-cesnovar and @mitzimorris said, the main bottleneck is that we can't circumvent our PR, code review, and voting process for software changes just because there's some money behind something.

I think there are a lot of features we'd implement eventually, but aren't at the top of anyone's to-do list that we could be incentivized to reprioritize. (Not me personally---I can't take grant money at Flatiron.)

Writing fixed-price contracts for features (rather than for hourly work) is always tricky, because it always comes down to a judgment call w.r.t. whether the work is done according to spec or not.

(@mitzimorris) ... subsequently, the FB Prophet team arranged a second gift of $100K, this time to the Stan NumFOCUS project.

Cool! I wondered where the additional funding had come from.

[yizhang-yiz]

Agree. Feature specification is not easy to navigate and we need to be prudent otherwise sooner or later we'll run into "customers" who are not satisfied with what's delivered and Stan gets a bad vibe.

[ryan-richt]

An alternative way to think about this: Perhaps, even considering PR quality, the requirement of the standard Numfocus contract, all other sources of priorities, that the developers who would otherwise be doing the work will be doing the work, etc ... there exist many potential priority orderings of the same underlying set of features, to which the Stan team would be indifferent.

In this case, feature bounties just provide a tie-breaking scheme to let the community have a stronger role in shaping priorities. The Stan team can choose to steer through the priority paths that claim them or not, or of course choose to steer through a "richer vein" of related feature bounties that maybe are individually smaller but result in a larger total for the same total effort than some bigger bounty... They just add another dimension of value to the objective function, but you can traverse it as you will!

[bob-carpenter]

@ryan-richt I'd go one further and say it doesn't really matter about ordering. What we want to do is add features that are good. So even if the consensus is that feature A is better than feature B, if someone wants to pay for feature B and feature B is good, I don't see a problem adding it first. That's how things work now,---decided by dev interests rather than some consensus roadmap.

I think the issue people are worried about is the SGB having to negotiate contracts around delivering something not really under their control. Features get accepted based on voting, not SGB decree.

There's also the issue of having to write a software contract, which is tricky business if it's not hourly.

[WardBrian]

I should note that some of the platforms which sit on top of Github and provide bounties are set up in such a way that they're entirely external to the project. For example, https://www.bountysource.com/ (NB: I have never used this site nor can I at all vouch for it) allows you to set up bounties for arbitrary Github issues, in such a way that the repository owner may not even be aware.

For example, you can find existing issues in stanc3 on this site and add a bounty to them if you desire: https://app.bountysource.com/issues/104450374-unexpected-error-compiling-something-to-do-with-data-arguments. There is nothing preventing someone from doing this right now, and it's not immediately clear to me that we would even be aware of it.

For example, here is an issue with a bounty placed by IBM: https://app.bountysource.com/issues/38507358-implement-the-loopfilter-support

If you click through to the underlying github issue, there is no comment or tag which indicates it has a bounty associated with it.

[yizhang-yiz]

No matter how the bounty is issued eventually it has to go through PR review process, possibly even design doc (as the DAE issue where this discussion was initiated). So it is going to involve more than just bounty hunter's effort, and it's going to involve Stan team one way or the other.

[WardBrian]

That is correct. My point was simply to point out that it’s possible the Stan team member doing the review could be totally unaware that there is money changing hands behind the scenes, which is something we should keep in mind in the discussion of this

[bob-carpenter]

there is money changing hands behind the scenes,

I don't think this is relevant information for code review. Many of us are getting paid to work on Stan.

No matter how the bounty is issued eventually it has to go through PR review process, possibly even design doc

This is the relevant part and why it's hard to promise someone a feature will get added.

[WardBrian]

I think the appeal of the bounty-based system is that the promise goes in the other direction. The promise is not from the developer saying "give me $X and feature y will be added", but rather from the user saying "If feature y is added/bug z is resolved, I will give you $X". Moderating this is not itself easy (most services which do this - for obvious reasons - do not charge the user until it is actually completed, so it's possible for them to not keep up this promise), but it seems more tractable than the other way around