Missing context, uncritical view of the political economy of "Impact"

[ekansa]

Interesting document.

I think it gets many technical issues right. However, it's missing an important conversation about social and political factors in measuring research in this manner. Some concerns:

• "Impact" not always most important value: Setting up common standards for impact can mean local context gets lost. A main value in some data may center on the difficulty / impossibility of ever replacing that dataset because the data describes something historically unique (examples in some ecology, biodiversity, archaeology). These data should not be measured in the same way as a material science database.
• Impact is a somewhat arbitrary thing to measure: One could also try to measure social impacts such as education + mentorship involved in the creation of research data, equity and fairness in credit and attribution, or even well being of researchers!
• Measurement is political: Metrics and Altmetrics are a form of surveillance and reflect power hierarchies in the work-place. One imposes metrics on subordinates, and metrics have a disciplinary force. Metrics without other institutional reforms that empower the workforce being measured can seem paternalistic or just another form of dreary Taylorism.
• Arbitrary metrics can further erode creativity, academic freedom: Researchers need freedom to explore different kinds of data, ways of sharing data, and rationales for creating and sharing data. Pressure to optimize on a predefined set of metrics will inhibit such innovation + creativity. Metrics wouldn't matter as much if researchers had more freedom to choose institutions and career paths that used different kinds of metrics, but to impose uniform metrics over the whole research community will only reward conformity and homogeneity in research outcomes.

These are just a few of the issues I see in this general approach. If you are crafting a policy document, then it is important to recognize that their are labor, theoretical issues, and equity issues involved in metrics. I wrote more about these issues in my chapter here:

Kansa, Eric. 2014. “The Need to Humanize Open Science.” In Issues in Open Research Data, edited by Samuel A. Moore, 31-58. London: Ubiquity Press. http://dx.doi.org/10.5334/ban

and in this blog post: http://blogs.lse.ac.uk/impactofsocialsciences/2014/01/27/its-the-neoliberalism-stupid-kansa/

[alex-ball]

We will respond to your points in more detail soon, but for now we would like to make it clear that we are emphatically not crafting a policy document. The purpose of the How-to Guides is to give practical guidance on accomplishing particular research data management tasks. The tone is, 'If you want to do X, and you might for these reasons, here's how.' We are certainly not trying to argue, 'Academic performance should be judged like this.'

[MonicaDuke]

Thank you very much for engaging with our document and for taking the time to respond and share these more political perspectives on the topic of impact measurement, which are an important part of the conversation.

You are correct to point out that we have not dedicated much space to discussing them. Primarily, this is because this document (like others in the series) is intended more as a practical guide than an exhaustive treatment of all the issues, therefore the main aim (as Alex has explained) is to describe the 'How-To' in practical terms, whilst aiming to be neutral and not push a particular agenda.

However we have now addressed the importance of the discussions in response to your (and others' ) comments) by acknowledging the controversies and limitations more upfront, adding a paragraph in the introductory section. Whilst we want to acknowledge the wider context in which the developments on metrics are taking place, we feel we are right in putting the emphasis on why you may want to measure impact, rather than on why you may not want to.

Of course it is valid to point out the limitations - such as your first concern that there are values other than impact, and that impact can have many facets that could (with varying degrees of difficulty) be measured. We hope that within the limitations of the length of the document, its aim and timely delivery, we have already (briefly) acknowledged some of the wider issues, particularly in the references to the thoughts of Burrows and Colquhon that we included originally. Burrows in particular articulates his unease with the ‘measure everything’ culture and its effects on academia, whilst Colquhon strongly criticises (alt)metrics. (Unfortunately I checked the links for Burrows the other day and only the recording of his talk is available, the link to the slides is currently broken). We have also added a link to your blog post so that the interested reader will be able to follow on and explore the issues more deeply than we can achieve in our publication.

[ekansa]

Thanks! I really appreciate your collective efforts here. The additional paragraph noting institutional, labor, and political aspects of metrics is very welcome, since I think many people would not consider these issues.

Shall we close this issue?