Divide Credit for a Paper only among professors in the database

[vijay03]

First of all, I would like to say I love CSRankings, and I think it is the best system currently for rankings CS departments. This discussion is just an attempt at making a good thing better.

There was a (long!) discussion on Twitter: https://twitter.com/jeffbigham/status/929914049206325249 about how the way CSRankings currently shares credits among all authors is a disincentive against writing papers with many undergrad/grad authors.

This has been brought up in multiple places, and I partially agree with the critique. I don't completely buy Emery's take that it balances out because if you have multiple authors, you will have multiple papers and so it will even out. As Claire Le Goues @clegoue points out (https://twitter.com/clegoues/status/930270828322803712), people respond to micro-level disincentives in a strong way.

I think one straightforward way to fix this is to share credit only among professors in the database. That way, the number of students working on the project does not affect the ranking; I think this is the behavior we want.

The negatives of this approach that come to mind:

1. The rankings are not stable. If a student later becomes a professor, the rankings change.
2. It creates a disincentive against your students becoming professors, since it affects you negatively.
3. We would have to re-run the rankings every time a professor joined the database.

IMO, none of the negatives are really strong:

1. I don't really mind that the rankings are not stable, not sure if other people do.
2. I think the positives for a student becoming a prof significantly outweight the negatives of the professor's rankings dropping a bit (as mentioned by Sam Tobin-Hochstadt @samth here: https://twitter.com/samth/status/930275363468562433)
3. A student becoming a professor is a rare event, re-running the ranking in this case seems like a low-cost thing to do. We would only need to update the ranks of the people she has co-authored papers with (this should be a small set).

Overall, I think dividing points for a paper with students and grad students is not the right thing to do (they should be non-entities in this ranking eco-system), and it seems like an do-able fix.

Thoughts?

[emeryberger]

So everyone has context, here is the entry from the FAQ. Specifically neglected in the proposal above is the artificial incentive it would create to prefer industrial co-authors or authors not in the database for other reasons (along with other perverse incentives below):

---

A single faculty member gets 1/N credit for a paper, where N is the number of authors, regardless of their affiliation or status (faculty, student, or otherwise). The number never changes. A paper can count for at most 1.0, in the case that all authors are / end up becoming faculty in the database.

The key downside to counting papers without adjusting for authors is that it would make it trivial to inflate the effect of writing a single paper simply by adding authors. Splitting authorship credit means that authors are incentivized to appropriately treat authorship credits. Note that publication rates are normalized across areas.

---

How about adjusting the count only by faculty in the database (or some other means)?

Here are some of the numerous downsides of only including authors present in the database:

• Authorship counts would be difficult to calculate (manually).
• Authorship counts would be dynamic (that is, they would change over time).
• When an author dies and is no longer in the database, everyone else would have to have their credit increased (talk about perverse incentives).
• It would create an incentive for senior faculty to have their junior collaborators not get tenure (since they would then likely leave the database).
• It would favor collaboration with industry (not in the database) over collaboration with academics. Note that companies do not generally provide public access to their employee directories.
• It would create a disincentive for faculty to see their students get faculty appointments (since it would reduce credit).

[vijay03]

I definitely acknowledge that are trade-offs to sharing the point for a paper among professors. IMO, the pros outweigh the cons.

The conversation with Claire has inspired me to think about who this policy helps and hurts. Right now, it hurts the most powerless members of the ecosystem: the students. I acknowledge that whatever policy we come up will hurt someone, but I would much rather it be professors themselves or institutions rather than students.

It also looks like the current policy optimizes for rare cases such as students becoming professors or professors leaving academia. The common case is that professors are working with students, and my argument is that this common case is what should be optimized.

Let me handle the cons one by one:

Authorship counts would be difficult to calculate (manually).

Don't see why this is the case. The burden of adding themselves falls on the authors. We simply compute based on who is in the database.

Authorship counts would be dynamic (that is, they would change over time).

Why is this a problem? By avoiding this, who are we helping?

When an author dies and is no longer in the database, everyone else would have to have their credit increased (talk about perverse incentives).

Easily handled by this policy: deleting a prof from the database does not return their points to the common pool. Only a student becoming a prof changes the points awarded for a paper.

It would create an incentive for senior faculty to have their junior collaborators not get tenure (since they would then likely leave the database).

Other social incentives (like nobody applying to a dept that consistently denies its junior faculty tenure) trumps the fractions of a point lost due to this.

It would favor collaboration with industry (not in the database) over collaboration with academics. Note that companies do not generally provide public access to their employee directories.

Note that in today's system if you collaborate you lower your own share of the point -- so why would the newer policy matter? The current policy penalizes all collaboration. The proposed policy only penalized collaboration with other professors. Strictly a win I think.

It would create a disincentive for faculty to see their students get faculty appointments (since it would reduce credit).

Again, trumped by other social incentives.

Overall, I hope to convince you that while the new proposal has disadvantages, the benefit of being able to collaborate as widely as professors want with students significantly outweighs the disadvantages.

[clegoues]

Rather than adding my own issue (which I will if Emery prefers it), I'll add my own proposal: Rather than giving points to coauthors, simply grant institutions 1 point per paper with an author affiliated with that institution.

This rewards research done at an institution (which is what csrankings purports to rank), rather than by particular people, and saves Emery the trouble of having to keep track of who is where.

More importantly, it neither incentives nor disincentives collaboration, especially with students.

Emery's answer on Twitter suggests that (A) we must assume adversarial actors (I agree), (B) csrankings should be difficult to game (also agree) and (C) my proposal can be gamed easily by pulling in of people from other institutions to a paper.

My counterargument is that (1) the present system is easier to game than my proposed alternative, and (2) there are more additional counter-incentives to gaming the proposed alternative than there are the current system.

Consider the following (reasonable) hypothetical:

Assume an undergraduate working with a PI for two semesters (or a summer and a semester). A PI more or less controls what an undergraduate works on. The PI has choices:

(A) The PI could steer the student towards an empirical study, replication result, or simple additional analysis or evaluation that serves to motivate, complement, or extend a larger project conducted by a PhD student. The PI could encourage the PhD student to supervise the undergraduate, and the undergraduate to collaborate with the PhD student in the writing of a top-tier paper.

(B) The PI could steer the undergraduate towards a slightly different or possibly simpler question, maintaining a strict separation of concerns from the graduate student's project. They could write a shorter paper for a NIER track or similar, so the undergraduate student still receives the benefit of paper writing. The key point is, in this latter world, the PI keeps the work separate by design.

Both options are legitimate; the PI does not need to be dishonest or fraudulent to choose one or the other. In general, however, I would argue that (A) is usually globally preferable for the purposes of training both the PhD student and undergraduate (details elided).

The present CSRankings system explicitly encourages (B). A PI simply gets fewer "points" for including an undergraduate in a project. Moreover, as the PI, doing (B) is easy, and perhaps easier than (A) (no need to coordinate multiple students, nor teach a PhD student how to effectively supervise). It's also basically impossible to detect that someone doing (B) specifically to game CSrankings. And, this whole setup explicitly and primarily harms some of the most disempowered people in academia (the students).

So: the current system can be easily gamed in a way that doesn't even require PIs to be behave fraudulently, and the people it harms have virtually no recourse.

Given that both systems can be gamed, the question is thus which "gaming" is easier/harder, or better/worse.

The counterargument (from above) is that my proposed system can be gamed by pulling in of people from other institutions to a paper.

I'll start by being pedantic and noting that, in the proposed system, a PI doesn't directly benefit from fraudulently putting friends at other institutions on a paper. Instead, the proposed system would reward the construction of multi-institution fraud rings, where PI A (fraudulently) puts PI B (at another institution) onto a paper to, in exchange, be (fraudulently) added to one of their future papers.

That's much more circuitous an incentive than the incentive in the current system, where a PI can directly benefit themselves/their institution via a slight nudge of a student in a particular direction and a lack of a nudge to have students work together. As a gaming mechanism, it also requires explicit fraud and violation of established policies (institutional, ACM, etc) that exist to discourage that kind of behavior. The current system does not (because, again, there's nothing intrinsically dishonest about keeping two student projects separate).

(If there's no explicit fraud, then PI B belonged on PI A's paper because of a legitimate contribution, and all the proposed system has done is encourage multi-institutional collaborations. I have no problem with that.)

(Personal disclaimer, primarily for my students: I'm not going to behave one way or the other based on csrankings' approach!)

[fycus-tree]

562 and #564 are related. Namely, why not allow a dropdown to select Normalization: [None, 1 per institution, 1/# Authors, 1/# Indexed Authors]

[emeryberger]

I disagree with creating an artificial incentive to work with colleagues in different institutions vs. working with local colleagues.

The overall point is that there should be no easy way to "print money". That is what your proposal does: it takes one paper and inflates its value artificially. I am firmly opposed to any plan that does this.

As it stands, the only currency is papers in top conferences, which is as difficult to game as our peer review system. Adding people (from other institutions) as authors is free money. Listing friends as co-authors would only be the start.

[clegoues]

I think what's pushing me to continue this argument is the fact that the current system is 100% gameable. Keeping students from collaborating with one another is significantly easier than convincing my friends to engage in a cross-institutional fraud ring.

"Listing friends as co authors would only be the start."

...it would also be explicit fraud. Expressly disallowed by the ACM and most institutions. There are plenty of incentives to keep PIs from doing this. There are virtually none to keep PIs from keeping students from working together. Frankly, it's easier in a lot of ways. The current csrankings adds one more reason to avoid it.

[emeryberger]

Everyone should carefully consider what happened in the current US News rankings. They were played. For example, citations are counted, so people have formed citation cartels. They tell people to cite each other. They pay people to have bogus affiliations.

[emeryberger]

Fraud will happen, and it will not necessarily be detectable. It is better to have a system that is robust to attack than to try to cope with attacks later.

Right now, a paper is always worth one point at most. There is no means of inflating this currency. @clegoues's proposal is inflationary.

[clegoues]

I agree with you. Which is why I think it's likely that, if CSRankings does become adopted, people will be incentivized to change the way they advise students and manage multi-student projects. This, again, gets back to why I don't care much for metrics systems: they skew incentives. If we stipulate that they're going to exist, we should interrogate the incentives they introduce into the system. This system is relatively robust to attack, but it's not immune to it.

...Hence this argument, where we (well, you, I guess!) pick the preferred incentive.

[vijay03]

Can I just add that is easy to modify Claire's proposal so that it does not inflate?

We divide credit by N where N is the number of institutions on the paper. Each paper still adds upto 1. Profs are no longer penalized for collaborating with students.

[emeryberger]

What about when you have two professors from the same institution as co-authors?

[vijay03]

They share the credit that the institution gets. Institution get 1/X, each prof from that institution gets 1/XN if there are N profs.

[clegoues]

So, I think I genuinely like the modified proposal, but I'm still thinking about it.

To Emery's question: still only adds up to 1. I think this may require assigning points to institutions, not people. I'm basically OK with that, since the point is to rank departments, no? But, as mentioned, still working through the implications...

[emeryberger]

That's an improvement, but still doesn't address the issue of collaborations with industry, right?

[clegoues]

Can you just treat industry as academic units but just not present rankings for them?

[emeryberger]

(Also: CSrankings is faculty-centric: points only go to people; these are aggregated to form departments.)

[emeryberger]

Nope, that won't work. Absent from the database are both industry researchers and undergrads.

[clegoues]

Not sure I follow. If three institutions on a paper are UMass (Yuriy + student), CMU (Me + student), and Google, total points is 3. Yuriy gets 1/3, I get 1/3. Last 1/3 floats in the void.

(Overly specific example brought to you by the PLDI submission I'm working on with Yuriy as we speak....)

[emeryberger]

I have no way of knowing about Google affiliations.

[emeryberger]

I think it would be helpful for (one or both of) you take a look at the actual code (csrankings.ts) to see how calculations are performed; I think it would help guide your thought processes and understand the constraints it operates under.

[vijay03]

We can merge my proposal with Claire's:

UMass (Yuriy + student), CMU (Me + student), and Google

Total points: 1 UMass gets 0.5 point, CMU get 0.5 point Claire gets 0.5, Yuriy gets 0.5

[emeryberger]

So institutions don't get points except in aggregation. Let's focus on faculty.

[emeryberger]

I write a paper with Yuriy: I get 0.5, he gets 0.5. I write a paper with Yuriy and Claire: I get 0.25, he gets 0.25, she gets 0.5? Really?

[clegoues]

So, if it's Umass (Yuriy + student), CMU (Claire + Christian + student), and Google, Yuriy gets 0.5 point, and Christian and I each get 0.25.

...Does that work?

(I was just thinking that looking at the code might help, but I think I get your point well enough about people versus institutions....)

[emeryberger]

This is an obvious disincentive to collaboration across institutions, don't you think?

[emeryberger]

(Anyway, back later: speaking of papers, I need to work on mine...)

[fycus-tree]

I think it's important to think of what these metrics are measuring. The existing metric models papers as being written by some authors, assigns credit, and then looks at how much credit known authors have. This proposal suggests that papers can be modeled by looking at only "known" authors (unknown authors doesn't exist).

Two proposals 1) Average them! Given a paper with N authors and M <= N authors in the database, we've talked about 1/M, 1/N credit assignment, how about a mean of 1/M & 1/N? It'll mostly be dominated by 1/N (e.g. making Emery's problematic incentives weak), but would give back half the credit of unknown authors. 2) Give away "unknown" credit for free in some conditions. Compute the average fraction of credit assigned to unknown authors, call that T for T in [0,1], for each paper you have U [U=(N-M)/N] "unknown" credit, so instead of distributing 1-U credit to known authors (same as 1/N, current method), distribute 1-min(T,U) credit to known authors. This makes the first few authors "expensive", but additional authors past average are "free". I think this works well within the incentive system of normal academic research-- the system can only be gamed if you refuse to include the first few students, but they're the ones who did the work, wrote the paper, and are the hardest (socially, reputation-wise) to remove from papers. However, this still has the issues that Emery highlights (disincentive for unknowns to become known). If we flip the min to a max, we can solve that, but then rankings can be gamed by simply refusing to include additional students and that seems more perverse. Hence I think min is better, as unknowns becoming knowns is usually rare and considered good by most other metrics (academic lineage). 3) There's also a variant of this where we compute the average fraction of credit assigned to each author (A = avg of 1/N) and take 1-(N-M)*max(A,1/N) to assign to known authors. This normalizes by count of authors rather than fraction of authors. This may not generalize across different fields as well.

[emeryberger]

Taking a one-minute break from PLDI: Another key design guideline of CSrankings is that it be simple. Complex weighting schemes etc. are problematic, and I think raise more questions than they answer.

[vijay03]

Emery, did you have a chance to think about my proposal (just divide credit among know authors)? I think it fulfills all your criteria (simple, doesn't need a lot of changes to existing CSRankings), and I've explained here (https://github.com/emeryberger/CSrankings/issues/589#issuecomment-344375947) why I think the arguments against it don't hold up (IMO).

Good luck for PLDI! I'll wait for your response afterwards.

[emeryberger]

Hi @vijay03. I have already stated the biggest problems with your current proposal (to only count known authors). You assert it only creates a disincentive to work with other professors (which somehow is fine). I am not sure why creating a disincentive to work with other professors while incentivizing working with students is good – it would encourage faculty to avoid working with other faculty.

It also would create an artificial incentive to work with colleagues who are not in the database (e.g., in industry, or in non-CS departments, or in non-PhD granting institutions).

For example: say I write a paper with Dan Barowy, my PhD student who is now a faculty member at Williams College, or one with Charlie Curtsinger, my PhD student now at Grinnell, or with Ben Zorn, my long-time colleague at Microsoft Research (note: I have and will do this anyway). Boom, more credit for me than working with my colleagues David Jensen or Andrew McGregor (etc.) at UMass.

I note in passing that it would create an artificial incentive – one that would accrue primarily to the benefit of elite institutions with very strong undergrad programs, for example – to incorporate an army of undergrads who would be "free" for any paper.

It is exactly this kind of issue that makes tweaks to accounting fraught.

[vijay03]

Hi Emory,

As mentioned before, any scheme we devise will penalize someone. The current scheme penalizes all collaboration -- why is this okay? The "winner" in the existing scheme is the lone wolf writing papers.

I agree the new scheme incentivizes working with someone not in the database. However, I doubt if this will actually influence behavior in practice. As yourself note, there are other incentives to collaborate that trump the fraction of a point you lose by collaborating with someone. If people are collaborating under the points scheme of CSRanking today (which penalizes all collaboration regardless of whether they are in the database), I doubt the new proposal will stop people from collaborating.

The army of undergrads problem you bring up does not seem to occur in practice. I have never seen a paper in SOSP or OSDI with armies of undergrads powering them :)

Anyway, I will stop here. Perhaps Andrew Baumann is right, and I should stop spending energy on this.

[fycus-tree]

@vijay03 @emeryberger It seems there the simplest options are 1) Unknown authors are treated like everyone else (current model) 2) Unknown authors are ignored entirely (Vijay's proposal) 3) Unknown authors are treated as some fraction of a normal author [0.5 of a normal author if we average the scores of (1) and (2)] 4) Unknown authors could vary in value depending on the number of unknown authors on a paper (proposed above with a step function, where the first few unknown authors are treated as normal authors, and after a certain number, they become "free" and don't cost anything) (could also have one "free" unknown author per known author on a paper, etc.)

And then stuff gets really complicated. (1) seems fairly robust and hard to game, but seems to incorrectly model student contributions. (2) models student contributions better, but incentivizes collaboration with industry over collaboration with universities. [the 50 undergrads powering a paper seems, not only rare & unlikely but may actually be "correct"]. (3) splits the difference. (4) leaves space for a better student model but may suffer from the issues of (2).

[emeryberger]

Thanks for the summary, @fycus-tree.

Note that I don't agree that the current scheme penalizes collaboration. I believe the numbers basically make it a wash, and that in fact, scheme (2) absolutely would strongly discourage collaboration with other faculty members.

Anyone who has written papers knows it's easier to do more work when you have more people involved (up to a point!). "Many hands make light work. "

I just submitted four papers. I could not possibly have done all the work involved to produce those papers; in this time period, I think I could have produced one by myself (and not anywhere near as well!), at best two. In the current system, I get about 1.3 for these papers. This seems almost exactly right! It's very close to what I could have done by myself. This is no proof but it suggests that collaboration is mostly neutral (and probably helps).

I note that I would get a much larger boost under @vijay03's proposal — 2.83 — boosted because of the inclusion of people outside the DB in industry. It tilts the balance so much on some papers (for two of them, I would get 1 full point, as if I did everything — and I did not) that it actually makes me uneasy. Were this how it worked, I could see it being a very strong incentive to never collaborate with faculty and work exclusively with one's grad students and others not in the database. If you have a paper for which you are going to get 1 point, the marginal cost of adding a professor is huge.

TL;DR I remain unswayed that option (1) is broken, or more to the point, that we have found a better solution. I do thank everyone for this conversation; I remain open to being convinced by a better replacement.

[emeryberger]

I decided to get some data to support my claim that more authors means more papers (that is, that there is a linear relationship between the number of authors on publications and the number of publications).

For every author in CSrankings (~ 7000), I computed the number of pubs (in CSrankings venues) for the period 2007-2018, and the total number of authors across these papers.

The correlation is very strong (1 is perfect): r=0.964, with a p-value < 0.001. Fitting a least-squares linear regression yields the following linear relationship: number of pubs ~= 0.22 * number of authors + 0.74 (same p-value)