search

sig-cm / JCDL-2020

A repository for planning and executing a workshop for SIG-CM at JCDL 2020
0 stars 1 forks source link

Review Paper # 1 #1

Closed nniiicc closed 4 years ago

nniiicc commented 4 years ago

@organisciak + @nniiicc review paper https://github.com/sig-cm/JCDL-2020/blob/master/JCDL_Where_the_rubber_meets_the_road_2020-6-28-FINAL.pdf

nniiicc commented 4 years ago

Actually - lets have Peter review this instead of Andrea (since we're asking her to also do the CFP outline)

organisciak commented 4 years ago

This paper studies scholarly research workflows, through the lens of compression, contextualizability, and composability. It seeks to address labor bottlenecks of semantic publishing by looking for places in existing practice that align with semantic publishing effort.

The paper is well-written and structured, building on interviews with PhD students and think-aloud observation of scholars' work. As you continue work on it, I recommend consideration in a few key areas.

First, the justification is sensible, but needs more background context. More detail should be provided on semantic publishing, and after presenting your personas, there's room to more thoroughly consider them in the context of semantic publishing. How might you tie the specific habits of the personas to meaningful semantic publishing work? Are there certain types of labor that are more transferable than others?

One of the most interesting takeaways is your observation that the common good goals of semantic publishing already align with middle-stage scholarly workflows. At the same time, I was glad to see some discussion about the locality and non-generalizability of personal workflows. Your personas show that - people tinker and try to develop personal approaches specifically because they are personal, making engineering a useful capture of that work difficult. This is where I'd suggest centering future discussion - around how to reconcile the promise inherent to that public/private alignment and the reality of trying to perform both tasks at once. In light of your findings, how might you design for scholarly workflows that are useful both locally and generally?

Finally, there's reference to future work that is currently being prepared. I suggest further describing that work, in terms of design and questions it will pursue.

Good work.

typo - "in this paper, investigate"

nniiicc commented 4 years ago

This paper addresses a timely topic and provides an initial analysis of the research question "Where (if at all) are there integrationpoints between semantic publishing and existing practices of reading and synthesizingthe scholarly literature?" The contribution of this work is a focus on domain experts that are seeking to use semantic technologies to synthesize empircal evidence. This focus is contrast to most studies of ontology development which focus on engineering of semantic web applications. The basis of this paper is observations from publicly available sources - such as video tutorials and informal web documents (e.g. blogs).

Overall - I found this paper highly relevant to conversations emerging about the labor of synthetic research reviews and scholarly publishing more generally.

Some places for extension or additions:

  1. There are a number of critiques of the semantic web - many see the entire field as having failed to create a reliable way to create semantic content. In future work it would be helpful to address these critiques directly - how and why is the semantic web worth studying if it has continued to be "5 years away" from implementation since 2000? (Note - I think this is a stawman argument, but its one that an uncharitable reader might pose in future reviews so a good counter example and motivation for your investigations would be helpful)

  2. Your personas are well described and offer generative examples - but these could be framed in the existing archetypes of the Semantic Web as a development community. I'm thinking here of the distinction between neat vs scruffy in engineering. Your personas oftens straddle these divides and it might be worth thinking about this dichotomy when describing end-users / domain experts vs engineers. Some helpful pointers to relevant literature:

  1. With regards to your 'explorer' and 'hacker' persona - I couldn't help but think of early work in annotation systems and digital reading more generally as being resonant with these descriptions. The HCI literature provides many examples of the ways in which early adopters hack scholarly systems to meet their own conceptual model of how these systems 'should' work - But the JCDL community also has a small but foundational literature here. The networked notebook idea was also very intruiging and could be a very interesting sub-study or full paper if you pursue that route. A few recommended citations below:
  1. The 'local semantic webs' argument at the end of page 8 needs a bit more fleshing out and alignment with the structuring of semantic web technologies. I'm thinking here of the idea of a wishbone model of ontology development where domain specific ontologies 'hang' off of more general higher level ontologies (see Guarino, N., & Welty, C. A. (2004). An overview of OntoClean. In Handbook on ontologies (pp. 151-171). Springer, Berlin, Heidelberg.) ... there is definitely something here but in future work that extends these observations I think it would be worthwhile to align your own recommendations with the affordances and practices of semantic web engineering

Small places of correction: