Respect and Recognition with the Contributors Page in Daemo

[neilthemathguy]

What problem is being solved? If feasible, specific evidence or examples (e.g., screen shots, data) in support of the understanding of the problem.

Recognition is important and must be done in fair, respectful, and consistent manner whether it is in media, industry, academic communities, requesters/worker communities, or Daemo blog/forum posts. Several paid contributors and unpaid volunteer contributors have spent valuable time of their lives to advance this project. The proposal aims to recognize everyone’s efforts and avoid past unfair scenarios of recognition.

Proposal

As a community, we had took steps toward creating the Founding Members' primary contributors' page. We started working on this issue in November 2015.

Along with jsilver, angela @shirishgoyal @iceLearn rijhul tosh, many others have worked on designing and developing the page that will highlight every contributor from all of the submitted papers of Daemo. We spent huge amount of time and efforts on this. I’m not sure why the page wasn’t added to Daemo. Now it is the time to add this page on the platform

Primary Contributor Founding Member is anyone who has participated in Daemo and Crowd Research Collective as an author on submitted, accepted, rejected research paper. The development of Daemo, thus far, has happened through a series of papers and foundational activities such as user need-finding, brainstorming, data gathering, experiment design and execution, system design, development of the platform, cohort onboarding and community development, etc.

The page included the Founding Members Primary Contributors Section

• a photograph of a founding member primary contributor.
• names of contributors, sorted alphabetically by their first names.

Additions to the Founding Members Primary Contributors Section will include Along with the names of authors, the page should include:

• joining date of a contributor (MM/Year)
• slack usernames and affiliations of a contributor

Below the Founding Members Primary Contributors Section, all of the papers (submitted, accepted, rejected) should be listed in the chronological order. This section should contain:

• The title and theme of the paper, as listed below
• Authors should arranged by their author ranks in the paper
• The affiliation and page rank score (if available) for each author should be listed

Daemo project has evolved through series of research papers ---

1. UIST 2015

   • Daemo: a Self-Governed Crowdsourcing Marketplace
   • Accepted [Topic: Vision of the Platform and its Governance]
   • Author names, affiliations

2. CHI 2015

   • Daemo: A Crowdsourced Crowdsourcing Platform
   • Rejected [Topic: Vision of Daemo and its Pillars: Prototype Tasks and Boomerang]
   • Author names, affiliations, page rank score

3. UIST 2016

   • Boomerang: Rebounding the Consequences of Reputation Feedback on Crowdsourcing Platforms
   • Accepted [Topic: Boomerang Reputation System]
   • Author names, affiliations, page rank score

4. CSCW 2017

   • Crowd Guilds: Worker-led Reputation and Feedback on Crowdsourcing Platforms
   • Accepted [Topic: Worker Reputation and Organization Structure]
   • Author names, affiliations, page rank score

5. CSCW 2017

   • The Daemo Crowdsourcing Marketplace
   • Accepted [Topic: Daemo Reputation and Prototype Task Demo]
   • Author names, affiliations, page rank score

6. Collective Intelligence 2017

   • Designing A Constitution for a Self-Governing Crowdsourcing Marketplace
   • Accepted [Topic: Constitution]
   • Author names, affiliations, page rank score

7. HCOMP 2017

   • Prototype Tasks: Improving Crowdsourcing Results through Rapid, Iterative Task Design
   • Accepted [Topic: Prototype Tasks Authoring System]
   • Author names, affiliations, page rank

implications (short and long term) of it being executed. The Founding Members Primary Contributors page with above content fixes the recognition issue. It clearly highlights how project was evolved over the ~2.9 years. It gives volunteers ability to point to Daemo and mention their contributions. It makes rest of the world see how project was evolved at different stages. Note that the founding member or primary contributor status is not cohort dependent. Anyone who joins the platform, no matter what cohort or time, and makes significant contributions must be recognized as a founding member.

---

Use comments to share your response or use emoji 👍 to show your support. To officially join in, add yourself as an assignee to the proposal. To break consensus, comment using this template. To find out more about this process, read the how-to.

[shirishgoyal]

Just attaching the screenshot of previous work in this direction which was undertaken by the community.

[mbernst]

Can you please clarify who qualifies as a Primary Contributor? Sorry if I missed it, I didn't see that described in the proposal.

I don't love the term "Primary Contributor"...it suggests the existence of other adjectives. "Weak Contributor", etc. Can we change it? "Our Team" or something like that? Just "Contributors"?

I would also like to see in this proposal: is the only way for someone to be a Primary Contributor to coauthor a paper? Wouldn't that ignore people who are supporting the effort in other ways? What if we're not publishing papers at the moment? Then nobody can be added, which would be unfortunate.

[dmorina]

BREAKING CONSENSUS

There's no way to quantify contributions or if done as you suggest "must be done in fair, respectful, and consistent manner" then 90% of the folks in your lists would not be included. AND most of this stuff has nothing to do with Daemo so that's not where it belongs.

Suggestion

Proposal 1: Put it on crowdresearch.stanford.edu or wherever you want to keep a list who published (or attempted to publish) a paper. Majority of these papers if not all are not implemented or included in Daemo and these should be separate! Proposal 2: Don't do it at all

[neilthemathguy]

Thanks @mbernst, these are great questions.

@mbernst said

Can you please clarify who qualifies as a Primary Contributor? Sorry if I missed it, I didn't see that described in the proposal.

Primary Contributor is anyone who has participated in Daemo and Crowd Research Collective as an author on published OR rejected OR work in progress research papers. The development of Daemo, thus far, has happened through a series of papers and foundational activities such as user need-finding, brainstorming, data gathering, experiment design and execution, system design, engineering-development/deployment of the platform, cohort on-boarding and community development, etc.

See below for the modification of the term Primary.

@mbernst said

I don't love the term "Primary Contributor"...it suggests the existence of other adjectives. "Weak Contributor", etc. Can we change it? "Our Team" or something like that? Just "Contributors"?

I hear you. However, in this context, the opposite of a primary contributor is not a weak contributor. I've removed Primary if that is confusing. We should call them Founding Members who have laid down the foundation of Daemo and Crowd Research Collective. This solves the problems. For example, Alison Cossette (acossette) and the team proposed the idea of Open Governance and did the foundational work to bring it to fruition.

I’ve replaced the Primary Contributors to the Founding Members in my proposal.

I would also like to see in this proposal: is the only way for someone to be a Primary Contributor to coauthor a paper? Wouldn't that ignore people who are supporting the effort in other ways? What if we're not publishing papers at the moment? Then nobody can be added, which would be unfortunate

Yes, I've cleared this in the proposal. I think it is important to think through this in two steps.

1. As a first step, let’s recognize the accomplishments of contributors in the Crowd Research Collective and Daemo so far. In order to recognize these accomplishments, we need to start with research papers and platform development process --- the fundamental activities behind the existence and development of the Crowd Research Collective and Daemo, since its start in 2015, 2016, and till today in 2017. This process is inclusive and doesn’t ignore anyone, because everyone who is active right now is already part of the author’s list. People who didn’t make it to the author lists, but did contribute in whatever small ways or intermittently will be added in a alphabetical list below the papers section as per the last contributors’ page design (screenshot above).

2. As a second step, to recognize future contributors from new cohorts, we will collectively define the metric to incorporate other types of contributions, which will include the focus areas we decide to work on along with research papers.

Since it is a research project, I believe research papers will continue to remain an integral part of the whole project.

[mbernst]

Thanks @neilthemathguy!

OK, so weaving this together:

Primary Contributor is anyone who has participated in Daemo and Crowd Research Collective as an author on published OR rejected OR work in progress research papers.

and

we will collectively define the metric to incorporate other types of contributions, which will include the focus areas we decide to work on along with research papers.

So concretely this is proposing to include all coauthors on submissions to date as a starter group, and a commitment to develop a protocol for non-paper contributions moving forward? I'm good with that. Do you intend that the non-paper contribution proposal be included in this proposal in order to reach consensus, or developed later? If later, when?

We should call them Founding Members who have laid down the foundation of Daemo and Crowd Research Collective.

That sounds more accurate. The one challenge is that we'd need another proposal later to rename it as we add new people, since they wouldn't be founding...plus some of these people joined in later cohorts, so weren't really "Founding" if we're being careful. And I would dislike having a separate category of "Founding Contributors" vs. contributors who joined later --- that feels exclusionary, when we're trying to be inclusive. What about just "Daemo Contributors", or "Daemo Team"?

[neilthemathguy]

So concretely this is proposing to include all coauthors on submissions to date as a starter group, and a commitment to develop a protocol for non-paper contributions moving forward?

I would not call it just a starter group. A founding member is one who has made fundamental contributions to the platform. Without these contributions, Daemo would not have become reality. I'm afraid the way things have happened recently, in next few months, names and work of these people will be wiped out.

I'm good with that. Do you intend that the non-paper contribution proposal be included in this proposal in order to reach consensus, or developed later? If later, when?

I’ll add it in this proposal.

That sounds more accurate. The one challenge is that we'd need another proposal later to rename it as we add new people, since they wouldn't be founding...plus some of these people joined in later cohorts, so weren't really "Founding" if we're being careful. And I would dislike having a separate category of "Founding Contributors" vs. contributors who joined later --- that feels exclusionary, when we're trying to be inclusive. What about just "Daemo Contributors", or "Daemo Team"?

"[...]when we're trying to be inclusive[...]" I don't think so, are you saying that we are really trying to create inclusive environment? I'm sorry, many of the actions and proposals by some folks don't say so. Despite of many efforts on the contributors page, did we incorporate it to the platform? This has been going on since November 2015. I'm afraid, as it happened in the past, this issue will be dragged and ignored--- new batches will come and go, and nothing will happen.

[mbernst]

I would wish not to create a separation between people who joined in the first cohorts of Crowd Research / Daemo and people who join later, especially under the title "Founding", which has a status that others cannot attain. Otherwise, anyone who joins from here on out is inherently a second-class citizen who has a different title. If we wish to separate out cohorts, we could list out people by cohort rather than grouping some cohorts together?

are you saying that we are really trying to create inclusive environment?

This feels very charged to me, and I'm sorry for any role I've played in creating an environment you perceive as not inclusive. As you have mentioned in the past, I think much of the recent effort is dedicated toward formalizing and enacting what it means to have a more inclusive environment. I do not know why the team page fell off the radar, but we all collectively own up to it.

[neilthemathguy]

Thanks @mbernst.

I would wish not to create a separation between people who joined in the first cohorts of Crowd Research / Daemo and people who join later, especially under the title "Founding", which has a status that others cannot attain. Otherwise, anyone who joins from here on out is inherently a second-class citizen who has a different title. If we wish to separate out cohorts, we could list out people by cohort rather than grouping some cohorts together?

The proposal doesn’t say that any cohorts should be separated. The founding member or primary contributor status is not cohort dependent. If a person makes significant contributions, defined by the evaluation metric at particular time, she/he must be recognized with this status.

This feels very charged to me, and I'm sorry for any role I've played in creating an environment you perceive as not inclusive. As you have mentioned in the past, I think much of the recent effort is dedicated toward formalizing and enacting what it means to have a more inclusive environment. I do not know why the team page fell off the radar, but we all collectively own up to it.

This is not charged. It is frustration of a volunteer. I and many others feel disrespected in these conversations and many of the practices here.

[shirishgoyal]

I do not know why the team page fell off the radar, but we all collectively own up to it.

Seems like we all decided collectively to dump all these pages and just get done with it, our usual way of solving all the problems https://github.com/crowdresearch/daemo/commit/bc8da9a2a4f6c31ef655e9954a21bd0f33b2c1f5

[mbernst]

@neilthemathguy: As we discussed in the hangout after these last comments, giving credit is important to our community and to the goals of the effort, and I fully support this proposal.

How about something like "Creators" or "Core Team"? I prefer we not use the title "Founding Members" because "founding" usually refers to an initial group that started the project. This is not a blocking issue for me.

[neilthemathguy]

SUMMARY

We all agreed that this page existed, but removed, and it needs to be added back. For the next steps, I'll work out the design and metaphor for the page name.

[dmorina]

Status update from my side since I broke consensus: as it stands none of the comments have addressed the fundamental issues with this proposal as stated in the “breaking consensus” comment, which means that the status remains the same!

[dmorina]

p.s of course the PR stuff is relevant only if it ever passes the vote on whether this broken consensus is invalid or not, as it’s the next step for this proposal

[neilthemathguy]

@dmorina Said

[..] 1. it includes false information and 2. It’s not relevant to the codebase

Seeking opinions from other members of the Collective whether they think all of this is false information? @mbernst @iceLearn @shirishgoyal @markwhiting @qwertyone and others

[markwhiting]

I'm not sure what @dmorina meant by 'false information'. I assume he's implying that most methods of identifying contribution levels are flawed in some way or other. I think this is a valid point, and one that comes up every time these issues are discussed. But I also think we can design a simple rule that just lets us make it clear who will get rewarded for what. For example, we could say paper authorship or code authorship (having committed to a code repository) are activities that warrant recognition as core contributors. Another option would be to use a nomination process to put people at a more recognized level. Alternatively, we could not differentiate levels and instead just list anyone who has had a slack account as being part of the collective.

I am not sure what the community feels about this, and I think it might be interesting to try to find that out. The one thing I feel concretely is that we should have a simple definition that is clear and unequivocal in who is represented how.

I think the codebase comment is to say that this page should exist as part of crowdresearch.stanford.edu, which I think is a good suggestion, from an identity standpoint. Daemo is a platform created by the Stanford crowd research collective. This collective may create other things, and certainly can contribute research that is not always used on the Daemo platform, so it feels like putting this page at daemo.org is somewhat more limiting. I absolutely think it can be linked, but I think its reasonable to make its home affiliated with crowd research.

What do you think?

[dmorina]

@markwhiting correct, that is one of the points I was describing in the "Breaking Consensus" comment, and there's a second issue which makes this 'false information' which you have also mentioned in your comment and also I have described it above which is " This collective may create other things, and certainly can contribute research that is not always used on the Daemo platform" and most of the mentioned work did not end up in Daemo or not even relevant to Daemo which makes this proposal a false statement i.e saying that these are the folks who are behind Daemo or have built Daemo one way or another.

[markwhiting]

OK, thanks for clarifying.

[shirishgoyal]

Irrespective of what gets used in platform or not, people in the community are only working/researching to build platform. I really wonder if Daemo turns out without any of these research aspects, what novelty does it offer? What was the whole purpose of spending so much time on all these things? I cannot find where community decided to remove any of these research aspects either.

We individually can think of removing/disliking whatever we want but that doesn't mean the contribution wasn't made. We should remember that Daemo is not just a platform, it is more about community and bringing everyone on equitable ground. If some of us believe in below,

This collective may create other things, and certainly can contribute research that is not always used on the Daemo platform

then ideally we should mention all the contributors at all these places. We can start with platform, then wiki and then whatever is left.

it will have to go through the final quality check by the code reviewers and we have our own list of things we check when it comes to reviewing a PR, if that PR is based on the current proposal it will be rejected (and reverted if necessary) on the grounds that 1. it includes false information and 2. It’s not relevant to the codebase

Especially this part seems to violate the governance and operational group rights. Scope of code review only covers whether the code is working and works as intended rather than judging the content, which is supposed to happen in the proposal process. This is the prime reason why in-group voting for operational groups (mentioned in #15) can be detrimental for community building and consensus based decision making.

[dmorina]

I will provide a more detailed explanation which expands on the original breaking consensus comment before the voting begins

[neilthemathguy]

Voting

Respect and Recognition with the Contributors Page

As stated above recognition is important and must be done in fair, respectful, and consistent manner whether it is in media, industry, academic communities, requesters/worker communities, or Daemo blog/forum posts. Several paid contributors and unpaid volunteer contributors have spent valuable time of their lives to advance this project. The proposal aims to recognize everyone’s efforts and avoid misattribution.

The Founding Members/Primary Contributors page with above content fixes the recognition issue. It clearly highlights how project was evolved over the ~2.9 years. It gives volunteers ability to point to Daemo and mention their contributions. It makes rest of the world see how project was evolved at different stages. Note that the founding member or primary contributor status is not cohort dependent. Anyone who joins the platform, no matter what cohort or time, and makes significant contributions must be recognized as a founding member. These contributions must be reflected on every platform where Daemo and Crowd Research are mentioned.

As per the hangout (01/04/2008) and governance process #21 is already up for voting. http://youtu.be/ILc0EBkiKzg

[neilthemathguy]

@dmorina please add the paragraph for other option. We are already delayed by 2 days in the voting process.

[dmorina]

Voting

Please refer to https://github.com/crowdresearch/collective/blob/master/governance/process.md here's the main paragraph, and as I understand it, it means that we are voting on whether this broken consensus is valid or not.

If there is no consensus on a proposal, the proposal remains open and the deliberation process continues in the subsequent weeks. In the worst case, the discussion has had no activity between two weekly deadlines, and no consensus has been achieved. In such a "deadlock" situation, a majority voting process is pursued for resolving the decision by all the members of the Collective. Votes are taken for one week, from weekly deadline to weekly deadline. The decision is made by a majority vote: breaking consensus requires a strict majority (>50%). In other words, voting in favor of breaking consensus — typically, not making the change the proposal is suggesting — requires greater than half the votes. If breaking consensus acquires fewer votes than a strict majority, then the consensus passes: the proposal is approved. Voting records will be made public so that there is accountability.

Breaking consensus: a more detailed explanation

I totally agree with the issue posted here that "Recognition is important and must be done in fair, respectful, and consistent manner". However, I strongly disagree with the Proposal on how to solve this issue and here is why:

First, it is very debatable whether the authors of these papers should be given credit for Daemo or not, let's say that the answer to that is "Yes" for the moment and see if that list of authors is even correct or done in a fair way as the OP suggests that it should be. The process we have been following for those papers is PageRank (and that's the basis of almost this entire proposal by the OP), and while most of you might believe that it's just the members of CrowdResearch giving scores to one another and in the end the PageRank algorithm is ran to determine the final list that's not all there is to it. The initial data was collected from the crowd and then it was analyzed, and more often than not the folks who were analyzing these (crowd research admin group) did not like the results from those and have made adjustments until they agreed with the "modified results", here are some cases where that happened:

CASES (which can be supported by evidence on Slack upon request, at least I can provide this to the OP)

C1. CHI (September 2015), some page rank parameters had been changed because the initial results did not align with what the admin group thought was the correct ranking, and to "fix" that some threshold was introduced. As an example person X was listed as first author on the paper initally, and after the adjustments were made they ended up in position 999 [999 describes the anonymized position in that list]. These adjustments were done after the data was collected, without initially informing the voters, or letting them know afterwards, or even asking them to reallocate their points because this threshold meant that the script modified their scores by introducing a cap at 15% per peer.

C2. The cut-off threshold for who makes it to the final list is decided by a very small group of people and there's no real process behind it, it's just how these people "feel" like it should be. Having said that, UIST and CSCW had cases where the threshold was first decided and then moved because someone thought that person Y should be on the list, and that person was way below the initial threshold so by lowering it some other people who didn't really contribute to that paper were included too. In one of the cases the reasoning to include this person Y was the following "i think we should lower it to 0.005, to include Y, as it can be critical to growth of daemo - also to show Y as one of our official collaborators. ..."

C3. There were people with zero contributions on Slack or anywhere else who were included in the author lists due to how people voted, at least of what's publicly available.

C4. Certain people in the admin group had access to raw PageRank scores and were debating on whether they should ask person Z to change their scores because again they didn't seem right according to this group.

C5. Certain people from the same group were allowed to distribute the points after the deadline or in worst case last, and had access to the current scores of everyone else.

MY OPINION

O1. In the case presented in C2, the person who requested and changed the threshold based on his/her reasoning shown above they were clearly making judgements based on things that may happen in the future. I believe that the whole PageRank system was not appropriate for determining author rankings, in the end there were adjustments made based on the opinions on certain people who had pretty much the highest page rank scores (and were admins), being able to submit scores last while having access to current results and being a member with a high PageRank score meaning being the people with most influence in the network is a HUGE disadvantage for everyone else in the game.

To summarize, I believe that PageRank and the process described above is not the right one and it led to incorrect ranking of authors and wrong inclusion/exclusion of people. So, for this reason I don't think that using that as a reference to show who built Daemo is correct, fair and respectful especially to the people who actually contributed a lot.

Lastly, as stated in the original comment, there are papers and contributions which are irrelevant to Daemo.

Use comments to share your response or use emoji 👍 to vote for it and 👎 against it.

[iceLearn]

Just to document my opinion to the proposal, I am in the stand of recognition should be in towards stanford Crowd research collective and Daemo both places separately. Not only daemo.org.

When I signed up I did not have a strong opinion/desire on creating a fair marketplace on micro tasks etc, but I wanted to take part in crowdsourced research, collectively learn from diverse set of researchers and particularly I was interested in learning how elite institutions do research and learn from them. As a result of this process Daemo become a primary output which I learnt during the process and I contributed in the research. At the same time I contributed to the process of the crowd research on helping to onboard new members, had multiple hangouts to talk research practises but those were not part Daemo but the crowdresearch collective. Daemo is a product of Crowdresearch collective, therefore it should be recognize differently and crowdresearch collective should recognize differently in two places. ( Both SRC and Daemo can have their link of referencing each ) Practically, I would not get anything strong in return by saying I help to create Daemo, but more impact if I can point I was part of a worldwide crowdsource research collective and this is how I performed.

Collectively we should/can find a way to recognize contributions on Daemo and SRC. (IMO)

[neilthemathguy]

Dilrukshi (@iceLearn): as I understand your comment, you don't want to be listed on Daemo's webpage for the practical reasons you have mentioned above. Which is fine--- individuals can opt out and when the page will be created this will be taken into account 👍

[iceLearn]

@neilthemathguy it is great that you expressed your understanding of my comment.

you don't want to be listed on Daemo's webpage for the practical reasons you have mentioned above.

it is not what I meant.

What I expressed was that I believe we should have 2 places to recognize with different metrics.

I assigned my self to this proposal since I like the idea and support to it, it was something that I worked with others in the past.

Based on the last meeting, I believe few others also holds similar or closer opinions, that does not mean they do not want to be listed right?

Since I could not voice out in the meeting I made a comment here of my opinion.

I am little disturbed by your comment and feel I should not make any opinions as it is likely to be misinterpreted and judged, yet at the sametime it is because of you i was encouraged to make opinions since you always points the importance of it during meetings and ask each of us by names what is your opinion , what do you think about it ..etc which I really liked and learnt that best practise from you.

[neilthemathguy]

Thanks for the clarification. As I've said in past anyone can voice their opinions--- thats the reason I and others have spent huge amount of my time in creation of the governance process.

Based on your reply, I mentioned that if you or anyone who feels that their names should not appear anywhere, then you have that right. People can have different reasons of this. As I replied "individuals can opt out and when the page will be created this will be taken into account." Your comment, the way it is written, says that your work in more toward research not toward Daemo.

1. "[...] had multiple hangouts to talk research practises but those were not part Daemo but the crowdresearch collective [...]"
2. Practically, I would not get anything strong in return by saying I help to create Daemo, but more impact if I can point I was part of a worldwide crowdsource research collective and this is how I performed.

First of all this proposal is not about discussing the merits and demerits of any recognition criteria. The fundamental thing is there was a criterion, we all agreed to it, and we all followed it. This is not for a person to get individual returns, this is more about sticking to the right things that we all have agreed upon.

It is important to note that: as stated in the OP, in the hangouts, and in the recruitments of crowd, every effort of the Crowd Research Collective WAS and IS geared towards creation of Crowd Sourcing Platform; we have collectively called it Daemo.

I'm trying to understand the intent of this comment and its timing--- is it breaking consensus during the voting phase OR providing other perspective now that wasn't discussed above or before OR I'm missing something that needs to be added for voting?

I had asked opinions on this issue 6 days ago and only @markwhiting and @shirishgoyal replied

Seeking opinions from other members of the Collective whether they think all of this is false information? @mbernst @iceLearn @shirishgoyal @markwhiting @qwertyone and others

[neilthemathguy]

I would really like get things moving here. I've explained the voting process in details on Slack. People who didn't participate in the governance process or hangouts could go back and check the logistics details.

I would highly encourage folks who were part of the last hangout and Governance discussions (@mbernst, @markwhiting @shirishgoyal @iceLearn and others) to explain it to others who have questions. The voting on this is getting halted for no reasons.

[qwertyone]

I would like to add a helpful notion -- apply the concept of baseball grandfathering on this and move forward? For public records and disputes over contributions, let's borrow from baseball and recognize those who played the game as it was at the time. Those who were recognized keep the recognition and those who might deserve it are mentioned on the same level. If we have any disputes, place a steroids * as an acknowledgement of the controversy and a related note with those players and so on.

[markwhiting]

Sure.

I think they key points I would convey is that this vote helps us understand how the community feels about the explicit break in consensus from the standpoints that @dmorina and @neilthemathguy presented.

👍 are votes and they are conducted on two different posts made above. Other emoji are not counted toward votes. In other words, up vote on the post you prefer. Voting is completed at the upcoming meeting which is 9AM PDT on January 10, 2018. Here are the posts:

• Vote here to return to the original proposal and move forward with next steps of the proposal, which in this case might be something like a design using the current design language.
• Vote here to make the break in consensus become part of the proposal moving forward.

What exactly does this mean? Well, its not concrete (and I encourage @dmorina to provide some concrete interpretations), but presumably it would mean things like:

1. we agree on a new way to decide who contributed for the purposes of publicly communicating our community. e.g. don't just list authors, or list authors and be transparent about the implications of the mechanism used to decide on author rankings)
2. we host the relevant page in a place that is not on the Daemo platform but instead a page representing the crowd research collective.

So how should you vote?

My thought is that we should vote based on the explicit and immediate implications of one point of view or the other. I think we should assume that this kind of page will exist eventually independent of the outcome of the vote, but that some details about it could be different. Read the responses above and think about which set of details you find compelling.

What else can we do?

You can still have a conversation about this and talk about ideas or potential directions, however, please respect that it may be easier to get a good vote if we don't change the topic of the comments during the voting process.

[qwertyone]

I would seek to rephrase the issues on this page in the form of a new question in case the vote proposed by @markwhiting results in an impasse. As a practical alternative, what are the acceptable ways display recognition for the group? This thread has been focused on a content issue rather than a methodological one. Find the parts where there exists a most basic agreement of display is found and then the deficiencies can be addressed. It will be important as we start on-boarding for those seeking this information regarding merit can find it.

[shirishgoyal]

@qwertyone I think the conflict is whether this page should exist on Daemo platform and other associated platforms or not, and who all should be mentioned on that page.

• 1st option covers all platforms and all the contributors who have been participating in the community with some visible contribution (research papers, coding and all other domains).
• 2nd option only describes to put such page on crowd-research wiki with no information on who all need to be mentioned.

[dmorina]

@markwhiting I think the https://github.com/crowdresearch/collective/blob/master/governance/process.md#proposal-process is vague enough to allow different interpretations of how the vote should be carried out, so I don't think that the vote can move forward until a concrete process is in place and documented in the process.md file

And for the concrete interpretations of the outcome of the vote if breaking consensus passes please refer to the original "breaking consensus" comment Suggestion section

[neilthemathguy]

No need to halt the voting.

@markwhiting has listed above how the voting should be carried out. I have also explained the process as per the governance discussions and the document. I'm happy to explain it further and elaborate on the voting process. All the folks who have participated in the governance process and present during the hangout would be happy to explain the process and collectively decided interpretation.

[neilthemathguy]

@markwhiting what you wrote above--- isn't that your current interpretation as discussed and agreed during the governance process. @mbernst @shirishgoyal and others could you elaborate on this please?

Can you guys mention the process and your clear stance here?

I strongly feel now this just dragging and undermining the defined process and wasting the valuable time of volunteers. If that's the goal here, please let me know. We cannot just discuss things, agree on it, define the process and go back and forth for months on what we agreed on.

[rvaish]

In response to @dmorina's post:

Response jointly drafted by Rajan Vaish (@rvaish) and Michael Bernstein (@mbernst)

When the Crowd Research Initiative started (Originally launched as the UC Santa Cruz+Stanford: The Aspiring Researcher Challenge), the FAQ stated that the PI and organizers would determine the mechanism for the author order. This right was exercised by the PIs of the computer vision and data science projects, who made unilateral decisions on which participants would be authors without input from any crowd members. In the HCI project, due to a significantly larger number of active participants, we wanted to allow members to collectively recognize their peers. Therefore, to supplement our judgement, we introduced a distributed credit allocation system based on the PageRank algorithm.

As we tested out the algorithm for the first CHI submission, we were surprised to find a link ring in the votes (see paper, Figure 6). A link ring is when a small group of people give all of their credit to each other, a technique that tricks Google and other PageRank algorithms into amplifying the credit given to members of the ring. To explain: PageRank is essentially democratic in nature; by giving a recipient lots of credit, you are saying that you want the algorithm to weigh that recipient's opinion especially highly. What had happened was that many people across the community had given large amounts of credit to a highly visible and active member of the community, but she then routed it all into a link ring by giving 90% of her credit to one of her teammates. That destination teammate then routed her credit right back into the first person and her other teammates, amplifying the credit distortion. As a result, the proposed first author was someone who nobody outside of the link ring had voted highly. This practice, if followed widely, could delegitimize the whole reason for decentralized credit. Therefore, we addressed this issue by tweaking the algorithm to suppress link rings: setting a cap at 15% credit per peer (note the threshold = 0.15 here, code authored on Sept 12, 2015). This resulted in far more accurate results according to the PI. This cap has been maintained as a standard throughout. You can read more about it in our paper.

The PI then exercised their best judgment to set an appropriate score cutoff. Similar to traditional practices in research, PIs had the ultimate right in deciding who deserves to be an author and who doesn’t. Therefore, determining the PageRank threshold was the PIs' right to exercise. To ensure that oversights were not made, however, in certain circumstances and occasions, information was sought out about people on the cutoff border before making the final decision for where to place the cutoff. We also gave two chances for people to edit their scores: via an initial deadline and its publicly visible outcome, and then prior to the final deadline. We encouraged all members to edit their scores between these deadlines if they felt the original results did not represent their views. This was all done in good faith to credit the deserving.

Ultimately, PageRank in itself is decentralized and democratic: by giving credit, you are telling the system whose voices it should be listening to when calculating the result. Members who receive lots of credit have a larger influence on the final credit distribution. We observed rare instances of highly-ranked participants giving equal credit to every participant or giving credit to people who did not feel comfortable participating in public channels. We investigated and determined that they did not appear to be antisocial or strategic behaviors. So, we must consider those events in their full context and acknowledge that those participants 1) received credit, indicating that the collective trusted their opinions in this, and 2) felt that their behavior was the fairest way to give out credit at the time. The entire point of this algorithm is to trust the collective's opinion in this.

At no point was any person’s score manipulated or tampered with.

[mbernst]

Confirming that the post was jointly co-authored. 👍