proper-testing / proper

PropEr: a QuickCheck-inspired property-based testing tool for Erlang
http://proper-testing.github.io
GNU General Public License v3.0
880 stars 167 forks source link

Consider making PropEr a community managed project. #30

Closed blt closed 12 years ago

blt commented 12 years ago

At the time of this writing PropEr has two open pull requests and 12 open issues, many of those having attached code and ready contributors.

It would seem, from comments left in other issues, that the original authors are short on time. Please consider creating a GitHub Organization for proper and adding interested parties as managers. I am interested.

PropEr is a well-crafted tool. It should not rot.

kostis commented 12 years ago

We have absolutely no intention of allowing PropEr to rot, be sure about it.

There are good reasons why the two pull requests are not merged. For example, the memory consumption fix pull request is actually not a fix, but something to bypass a current limitation in the Erlang/OTP VM which we are working on fixing at the VM level (where it should really be fixed). Hiding problems below the carpet is not the PropEr approach to software development.

In any case, we can't make you a manager if we don't test you first. As a first test, explain to us what the issue is with the memory consumption fix and why it should (or should not) be merged.

yrashk commented 12 years ago

@kostis, #19 is not trying to hide the problem under the carpet, it just steers clear from that buggy part of the VM while gaining some additional minor memory usage improvement (assuming the VM will have the original bug fixed). I thought we sort of came to a common understanding on this back at the farewell drinks :)

blt commented 12 years ago

We have absolutely no intention of allowing PropEr to rot, be sure about it.

I'm glad to hear it.

There are good reasons why the two pull requests are not merged.

Excellent. Be so kind as to respond to the Pull Requests and close them out if you don't want them?

For example, the memory consumption fix pull request is actually not a fix, but something to bypass a current limitation in the Erlang/OTP VM which we are working on fixing at the VM level (where it should really be fixed). Hiding problems below the carpet is not the PropEr approach to software development.

That's a hell of an assertion. If PropEr is meant to be a pragmatic testing tool--meaning it aids developers in crafting more correct code and is usable in existing environments--then a patch that corrects a nasty behavior manifest in PropEr is a valuable one, no matter if it's really the VM's fault. I agree, the VM should be fixed, but updating one's VM to catch a fix for a testing tool is not always feasible. A VM fix in the future is all well and good, but @yrashk hit a bug in PropEr, was good enough to fix it by providing patches and was left in an ambiguous state as to whether or not his bug fix would be accepted.

Now, if PropEr is meant to be primarily a research vehicle all that's okay: PropEr's development obviously needs to be oriented to the production of new papers. If that is the case, you might be kind enough to note it in the README and close out Issues / Pull Requests that violate your need to work in a small-ish group.

In any case, we can't make you a manager if we don't test you first.

The thing is, I'm not setting out to be a manager just to do it. I'm willing to because PropEr looks dead. Here's why:

If you made me a manager, I'd close out all the Issues that don't matter, merge the Pull Requests that don't break the test suite, open the Wiki for business and rewrite the README so that it's useful to any Random J. Hacker that stumbles across this and thinks "Oh, that looks useful to me. How do I use it?" If this is a pragmatic project, that's your audience: all those Random J. Hackers. Right now PropEr does a terrible job of serving them, which means it's less used than it could be.

That's a damn shame because PropEr's a great tool that you folks have obviously poured a lot of effort into. I understand that you want control over PropEr--it is, at least, partially a research project--but you're strangling it to death by keeping your control tight and--at least public--activity low.

As a first test, explain to us what the issue is with the memory consumption fix and why it should (or should not) be merged.

I'm moving this week, then starting a new job next week and learning a couple alien code-bases; I won't have time to study the patches or the problem reported. My primary concern is not @yrashk's particular fix, but the way in which PropEr is being managed. If you want PropEr to be a pragmatic tool, make its development more Bazaar and less Cathedral. Use your community, don't ignore it.

If you're really interested in bringing me onto the team please email a list of questions to brian@troutwine.us and I'll get back to you in mid-May. I'd do pretty much everything I listed above, and probably open-source the website besides. Open up a project-specific mailing list too, I imagine.

Erlang infrastructure needs better tools, but those tools are going to need people that are excited to help people learn and adapt those tools. Look at Ruby; it's a crummy language with a great tool infrastructure and lots of interesting projects because they're running around in bazaars, sharing ideas and inventing.

That said, the patches should be merged if they fix an nasty issue in some supported VMs and a fix to the underlying VM should also be fixed. If, that is, PropEr aims to be a pragmatic tool.

kostis commented 12 years ago

I am sitting at an airport and I am about to board a long flight, so I need to keep this short. But more generally, I do not enjoy long mails on github.

If it makes any difference, here is what happens in a test case that I've made from the information that @yrashk has sent us. The following shows R15B and the upcoming R15B01 (which will be released on 4/4/2012, i.e. in two days).

Erlang R15B (erts-5.9) [source] [64-bit] [smp:8:8] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.9  (abort with ^G)
1> c(proper_gen_size_problem). 
{ok,proper_gen_size_problem}
2> erts_debug:size(proper_gen_size_problem:email()).
211322572
Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:8:8] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.9.1  (abort with ^G)
1> c(proper_gen_size_problem). 
{ok,proper_gen_size_problem}
2> erts_debug:size(proper_gen_size_problem:email()).
6567

This is PropEr without @yrashk's patch. Unfortunately, the changes in R15B01 solve half the problem that Yurii has noticed. We are also working on a fix to the VM that takes care of the remaining issues, but these will most probably appear in R15B02. So rest assured we are working actively (actually really hard) on this. It's just that I prefer to attack the problems in their root.

That said, Yurii is right in his comment and we have agreed to review his patch soon and possibly include it in PropEr's code base, after making sure that there are no negative consequences anywhere. This is exactly the reason why this issue is still open: we have not dismissed the patch, it's just that I am a bit skeptical that this really fixes the problem.

Over and out on this. Following your suggestion about having too many open issues, I am closing it.

blt commented 12 years ago

Thank you for your time.