Tournament Proposal

[leoj3n]

A tournament is a competition involving a relatively large number of competitors, all participating in a sport or game. More specifically, the term may be used in either of two overlapping senses: One or more competitions held at a single venue and concentrated into a relatively short time interval. (Wikipedia)

This is a proposal for a way of ranking that involves comparing apps against each other directly, instead of indirectly. Whether this is superior is up for debate, and perhaps a middle ground or combination can be reached where this is used in addition to the current method of ranking.

Please excuse my terrible drawing skills. What is happening in the images is an example of 14 "apps", from A to N. Over the course of 25 comparisons, apps are pitted against each other. In this demonstration the app to the LEFT always wins to keep things easy to understand.

Each "row" is a round, where the app to the left challenges the app to the right. If there is no app to the right in the case of an odd number, a "bye" is issued...

Bye: Automatic advancement of a player to the next round of a tournament without facing an opponent. (Wikipedia)

A line through the letter in that round indicates a loss or "L", and wins are alternatively filed under "W". A square around the "app" letter indicates a loser at the end of the tree.

Next, we calculate the the number of wins each "app" letter obtained throughout the tournament process. If a letter had a bye, it will be retroactively matched against another letter that had a bye in the next closest tree, the winner of which will be awarded the extra win. The number of times you lost doesn't count against you except that you don't gain a win from that match.

For instance, in the first case, the distribution of wins results in the following ranking:

A I E C B M J F D K G N L H

For these 14 apps, 25 reviewers/reviews would have been necessary (at a minimum; theoretically you could increase the number of reviewers per match to 2, or 3 for tie breaks). Obviously this one round of review matches isn't robust enough to approximate a correct ranking of the apps. For instance, Let's say theoretically app A is better than B which is better than C. Currently, C is ranking higher than B which is not accurate or fair to B. So, if another round of ranking were to take place with another set of 25 reviewers to further compare apps against each other as in the second image, B and C might have a chance to compete with each other and we derive a better approximation of their respective relative ranks.

After another round with the arbitrary rule for this demonstration of always having the app to the left win, tallying the number of wins after bye matches results in the following ranking:

A D B E I L K M C G J H N F

The more times this process is repeated the closer we get to the "true" or "perfect" ranking of these apps. Of course, this is some kind of logarithmic growth that may not be sustainable for larger numbers of competitors, so it probably makes sense to cut it off after a certain point as we approach a good enough approximation.

For newcomers to this system, it would probably be most fair to start them out exactly in middle of the existing pack from the last month. If this system were adopted as is, replacing the current indirect system of ranking, you could either completely randomize the current pool of apps for a fresh start, or you could adopt the current rankings and start from there.

The rankings should stabilize and become more accurate as time goes on and months go by. This is all assuming my assumptions and calculations are correct. It would be really cool if they were, but of course this is a nascent untested idea that may be very flawed. Perhaps a game theorist can explain any mathematical fallacies I have failed to see? I'm hopeful something like this would work, but perhaps that is just wishful thinking. Obviously this is a first crack at attempting to develop a direct comparison system, and maybe it's not feasible to scale or something.

Regardless, I just wanted to put this idea out there because I feel the current indirect system leaves a lot to be desired. This would be much more exciting and the things we learn from directly comparing apps are totally lost when just competing indirectly via checkboxes, which doesn't excite me at all really. Does the current system have flaws? Yes, definitely. Could this tournament system be a better replacement? If it works, I think it could be.

Anyways, would like to hear some of your thoughts. I won't be ideologically defending this if you found something wrong with my proposal. Perhaps we can work on it together to make something like this actually work... Who knows? I do have a feeling it may be unacceptable to replace the current system because of the existing technical debt already incurred, but perhaps some more direct comparison of apps can be weaved in? Thanks!

[friedger]

I think a direct comparison is important. I am not sure how feasible this is in the app mining program.

I think this works well in categories: compare blog apps, compare notes apps, compare files apps, compare chat apps,... My suggestion would be that we as a community organize such a competition. Maybe we can even raise funds for a reward. @joshthegreatavenue is doing this already in a more open way on Telegram from time to time.

[stackatron]

Thanks for the extra details. What entity is running this test and who are the testers? Do the testers need to try the app? If so, how much? How would that get enforced or tracked?

[leoj3n]

@jeffdomke If it were me, I would eliminate randomly chosen uneducated TryMyUI testers and have a prerequisite to participation in App Mining be that you, as a fellow app developer participating in mining, directly compare three sets of apps made by other app app developers every month you wish your app(s) to be eligible to participate.

For instance, if you have submitted two Dapps to app mining, and you wish both to be considered for payouts, you must submit comparison results for six sets of randomly chosen apps that you yourself did not develop (that would be three sets per app you are trying to promote through app mining, and in this case a total of 12 comparisons). This will have the side effect of limiting how many redundant single-purpose apps developers feel the need to submit to app mining every month, so now no one individual or entity can dominate these early times by flooding the rankings with multiple single-purpose apps that will remain dormant.

For instance, the app I submitted to an early hackathon had been receiving payments even without me even realizing it until many months later. If the emails I was receiving were about the requirement of me submitting comparison results on three sets of other apps, I may have been more inclined to take action and see what it was all about; i.e. if I wanted to participate that month or not. I still may not have participated or pushed any updates to my app, but this would have certainly made sense. People are skeptical of free money, so requiring this small contribution would help people get involved and encourage them to understand that app rankings is not the same thing as app mining; another point entirely that I may expound upon in another issue (I think the two would make more sense disambiguated, as it was an initial source of confusion for me as a n00b).

Anyways, I would imagine submitting an Dapp to the app store called something like "App Ranking" which keeps track of this data and user results is a good way to go about dogfooding this idea. App developers wanting to participate in Mining will be required to first participate in the Ranking, by clicking the link in the email to be taken to the "App Ranking" app, where they are given a set of three apps to rank under their username, which rewards them a token they can apply to submitting one app of their own to the actual mining payouts.

Maybe this thinking is too radical of a departure from your current modus operandi, but I think it would be something to try.