improvements to the algorithm

[jlodahl]

first of all: thank you for great work! Pairwise partial comparison is a cool way to do ranking.

The sorting is not very efficient though. A simple example: I made 9 ideas, labeled p1, p2, ...., p9. When I vote, I simply voted for the highest number.

With 9 ideas, 9 * 8 /2 = 36 possible pairs exist. A significant ranking should be possible with much less votes, otherwise the method is too inefficient or unreliable. Even with 36 votes in my case, the ranking was far from correct.

some ideas to improve this:

1. Instead of voting either / or, make a scale: I prefer much more, a little more ... e.g. move a slider or give points on a 1-5 scale. this way you will get information like alternatives p9 and p1 are "far" but p3 and p4 are "close". (this could be mathematical formulated of course).
2. Present pairs which will contribute to significance og the ranking, most likely alternatives that are "close" or alternatives that have not been compared yet and which cannot be ranked by use of other information (and transitivity assumption). For the user this implies, that if you vote several times, its getting more and more difficult to decide, becourse you present alternatives which is har to distinct.

it should then be possible to indicate when you have votes enough to establish a significant rankíng which would - in some cases - be as sginal to the user, that no more votes are needed.

Of course transitivity is not allways the case, but its possible to calculate a consistency index.

I think this can make the solution even more useful.

Last but not least: Keep up the good work.

Regards

Jørn

[allourideas]

Hi Jørn,

Thanks for the feedback. These are both issues that we've thought about. I don't think we will make the voting more complicated for the user (e.g., slider, 5 point scale) even if this would give us more information because we are very committed to the current simplicity of the voting.

However, as for the second idea (showing the voters the most important pairs), that is something we are definitely working on.

Here's a paper that explains a bit more about our methodology: http://arxiv.org/pdf/1202.0500v1

Thanks, Matt

[jlodahl]

Thank you for your fast reply. I have used the holiday to read your paper and thought more one what you are doing. I find your considerations very interesting and definitely a promising way to develop surveys etc in a combined quantitative and qualitative way. I also understand your desire to keep user interface simple. Still I am sure you can benefit from improving your algorithm by using cardinal inputs and have given it some more thoughts which comes here as well as a few comments to your paper.

A more efficient algorithm is needed, since the user as well as the surveyour expects a valid result. It can be done without much effort and will strengthen your approach. One thing is to choose the “right” pair to prompt the user for a response to. You allready have focus on this and choose the pair which will bring most information is definately a big improvement. I will not dig further into this.

Another thing is to request the user to quantity his preferences, e.g. “like a little more” “like much more” or similar. A simple scale with 5 or 7 possible answers will utilize if a respondent wishes to give more information and corresponds very well to your “greediness” criteria.

It will make the UI more complicated, I agree but not much. Instead of clicking on an alternative, the user will click on the scale. If you desire, you can make it optional for the one who set up the survey if he prefers the most simple UI or the most efficient algorithm. It can be done graphical and very intuitively.

It requires some math to do it, but judged from your paper I am sure you can handle this.:-) Moreover I have found a guy doing a PhD in voting methods etc. who has coded the Cardinal Pairwise algorithm in MatLab, so you can create an interesting relationship here I think, anyway he will supply you the code I am sure. So it will be a minor effort. to implement it.

I also notice you concern for computation time. I personally would not be to oconcerned about this given the small datasets (you only add one/few observations at a time), but your experience might show this as a problem since you mention it. If computation time is "long" compared to the time it takes to present a pair of alternatives it can still be "short" compared to e.g. the time it takes to present alternatives and receive response.Generally you can have computation of a new dataset lagging behind the collection of new data and just lagging one response cycle will give you much computation time. If you allow simultaneous user inputs you anyway must distinguish between the dataset on which you base the selection of pairs from the one to which you add the new input.

I hope you are interested in developing you system in this direction, I definitely look forward to using it. Keep up the good work!

Regards

Jørn

[allourideas]

Hi Jørn,

Thanks for feedback.

We'll continue to work on the project, and when we have new papers, we will post a link to them from our blog: blog.allourideas.org.

Take care, Matt