Discussion of badge draft

[rjleveque]

For discussion of the draft found on the wiki page [DRAFT] Open Science and Reproducible Badges.

Thanks @sr320 for getting this started.

Question: Do you see this as a badge for individual researchers or for a research group / lab? If the latter, it may be hard to figure out how to assign points without more guidance.

[sr320]

I was thinking at the individual level.

[benmarwick]

I agree that the individual level would be the simplest to start with.

[dacb]

Some of the items in cat. 1 appear to be specific to a publication. Is that the intention?

[benmarwick]

I've had a go at filling in some of the gaps following the discussion in the meeting today. Please have a look to see if I've forgotten or misremembered something.

Some design ideas...

There's a whole badge platform here, should we engage with it? Seems like a big committment https://wiki.mozilla.org/Badges

A very simple badge generator: shields.io

Or something along these lines: http://hexb.in/

The OSF badges: https://osf.io/tvyxz/wiki/1.%20View%20the%20Badges/

[arokem]

I like the Mozilla platform. It's obviously been thought through. For example, the fact that the badge links back to information about how the badge was issued and verifies it makes it much more meaningful than if people just testify about themselves that they are "reproducible". Even if they do their own self-assessment, it's still more meaningful than leaving it up to a vague interpretation.

Getting Mozilla's approval of our badging scheme is a badge of approval for that badging scheme. A meta-badge! :-)

[benmarwick]

@arokem are you interested to look further into the Mozilla platform and see what might be involved in setting up our own BadgeKit?

[dacb]

@arokem That would be fantastic!

[arokem]

I will look into it and post some impressions on this issue.

On Wed, Apr 8, 2015 at 4:59 PM, Ben Marwick notifications@github.com wrote:

@arokem https://github.com/arokem are you interested to look further into the Mozilla platform and see what might be involved in setting up our own BadgeKit?

— Reply to this email directly or view it on GitHub https://github.com/uwescience/reproducible/issues/3#issuecomment-91072617 .

[arokem]

Mozilla Open Badge Infrastructure (MOBI)

See here

Provides a conceptual and technical framework for granting individuals an online record of specific achievements, linking to the set of interactions that led to the issuing of the badge, and the work completed to get it.

=> MOBI badges are granted to individuals, not projects.

We could potentially grant a badge to an individual for having a project with a certain standard, and then 'stack' badges for having several projects related to a specific individual. Different badges can be granted for learning about something (e.g. going through an online training module), or for passing some (self?) audit.

=> MOBI supports verification of the badges with respect to an issuing organization: A centralized authority, or a community, which certifies the achievement.

One example from UC Davis can be found here

Specifically, MOBI provides a set of free software tools, and an open technical standard that can be used by a badge-granting authority to facilitate the administering and verification of badges. These tools facilitate the setting up and administering a web app for badging, and a data-base of badges granted. For details see this page and this page

Advantages: standard, thought-through way of doing this. Plugs into already established mechanisms for displaying and collecting badges (e.g. Mozilla's 'backpack').

Disadvantages: requires technical work to set up and administer - there might be easier ways to do this (?). Will require setting clear and uniform (self?) evaluation criteria, which may not be an achievable goal at this time.

On Wed, Apr 8, 2015 at 5:41 PM, Ariel Rokem arokem@gmail.com wrote:

I will look into it and post some impressions on this issue.

On Wed, Apr 8, 2015 at 4:59 PM, Ben Marwick notifications@github.com wrote:

@arokem https://github.com/arokem are you interested to look further into the Mozilla platform and see what might be involved in setting up our own BadgeKit?

— Reply to this email directly or view it on GitHub https://github.com/uwescience/reproducible/issues/3#issuecomment-91072617 .

[benmarwick]

@arokem would you like to try and set up a local instance of this infrastructure?

[rjleveque]

It would be great to try this out.

[arokem]

Not unless we first collectively decide that we're actually going to use it, and more importantly decide how we're going to use it. If you want to get a sense for what it would look and feel like, you can check out the mobi site.

On Thursday, April 9, 2015, Ben Marwick notifications@github.com wrote:

@arokem https://github.com/arokem would you like to try and set up a local instance of this infrastructure?

— Reply to this email directly or view it on GitHub https://github.com/uwescience/reproducible/issues/3#issuecomment-91314989 .

[rjleveque]

I think some of the things listed in Category 1 in the Draft should be moved to Category 2. As it stands, the Category 2 list only concerns data, not code. Things like "have a GitHub account", "Code has an open license", and "Code has persistent URL or DOI" seem to belong under "open" rather than "reproducible"?

Another question: it says the badge is good for one academic year. So in adding up points, e.g. 4 points for a publication with an associated repository or 1 point for each publication on the arXiv, does one only count publications within the last year?

[sr320]

@rjleveque I agree with you on the categories.

I think we can also come up with additional criteria.

Regarding the one academic year, personally I think this makes it too complicated. I think it should be you get a badge. Should be designed where there is room for growth (ie not many should be able to get Gold on the onset) so persons would re-assess to go up a level.

[benmarwick]

I've made some edits to incorporate those suggestions about classification of criteria and removing the one year limit.

I just came across this related project: http://www.artifact-eval.org/about.html, which gives out these badges:

[rjleveque]

Thanks @benmarwick! I mostly like the changes. Regarding the items in Category 2 such as "Code has a persistent URL" or "Data in UW ResearchWorks", is this meant to give 1 point for each such item, or 1 point total if you have at least one such item?

I didn't know about Publon. Should we also give a point for signing up for things like ORCID as a way to encourage it?

[sr320]

I was thinking 1 point per item. Yes I agree that giving points for services that we want to encourage or highlight like ORCID would be good.

[kallisons]

• [ ] points for entering data in the catalog
• [ ] catalog of badge items (that don't fit in the existing paper/dataset catalog)

[benmarwick]

This is a nice system for describing the openness of data that we might want to make mention of: 5 ★ Open Data

[sr320]

Some examples of simple badges

code

![starter](https://img.shields.io/badge/ROS%20Badge--grey.svg)

![silver](https://img.shields.io/badge/ROS%20Badge-36-lightgrey.svg)

![gold](https://img.shields.io/badge/ROS%20Badge-75-yellow.svg)

Would want them to go back to webpage (that eventually links out to list persons with badges & their details)

code

[![starter](https://img.shields.io/badge/ROS%20Badge--grey.svg)](https://github.com/uwescience/reproducible/wiki/%5BDRAFT%5D-Open-Science-and-Reproducible-Badges)

[![silver](https://img.shields.io/badge/ROS%20Badge-36-lightgrey.svg)](https://github.com/uwescience/reproducible/wiki/%5BDRAFT%5D-Open-Science-and-Reproducible-Badges)

[![gold](https://img.shields.io/badge/ROS%20Badge-75-yellow.svg)](https://github.com/uwescience/reproducible/wiki/%5BDRAFT%5D-Open-Science-and-Reproducible-Badges)

corresponding html that someone code embed

code

<a href="https://github.com/uwescience/reproducible/wiki/%5BDRAFT%5D-Open-Science-and-Reproducible-Badges"> <img src="https://img.shields.io/badge/ROS%20Badge-75-yellow.svg" alt="ROS Badge"></a>

[sr320]

Method for Calculating Scores from Google Sheets

Raw data looks something like

I created a second sheetand simply transferred header with =Sheet1!A1 in cell A1 etc along row 1.

For the first data cell Publications with Repo I counted the number of lines with (LEN(Sheet1!B2) - LEN(SUBSTITUTE(Sheet1!B2, CHAR(10), "")) + 1) and multiplied by 4 (points per publication) to get =(LEN(Sheet1!B2) - LEN(SUBSTITUTE(Sheet1!B2, CHAR(10), "")) + 1)*4

If it was a entry where a single point value was given for any entry I used =if(Sheet1!D2<>"",1,""). '1' would be changed based on point value and nothing would show up if nothing entered.

Finally I just summed up the rows!

Things that need attention is noting on form to leave blank if nothing to add (that is why my score is inflated).

---

The sheet I used to test this approach out is here

[benmarwick]

Some comments on entering data:

• Submit your code for review: Question not that clear,
• Have the substantive results of your publication reproduced by someone else using your code and data: Question not clear
• Code has one or more tests: not clear what this means
• One or more sets of raw data are published as soon as generated: tough question - maybe edit
• One or more sets of raw data are in open formats (eg. CSV, txt, etc.) : maybe redundant?
• What percentage of your publications from the last two years are Open Access?: not easy

[benmarwick]

Inspired by Steven's summaries, I made a simple shiny app so can see the scores and details of each respondent. The app pulls data from the google form (and scrapes the wiki page for the scoring), tallies the score and tells you what badge you earned. It also shows the score per criterion and the detailed responses. There's a drop-down menu that is populated by the names of the respondents, when a different name is chosen, the other tables auto-update.

The working live demo is here: https://benmarwick.shinyapps.io/ros_badge/ with clickable links out to the reseach products, and a little shields.io badge that updates with your score and colour (and links back to the badge info page). In case that's down (free tier is only 15h), I've made a repo from which you can run the app locally: https://github.com/benmarwick/ros_badges (all the code is here also)

Here's a screenshot:

There's a minor mismatch at the bottom of the bottom table.

[sr320]

An update on our progress ---- http://tinyurl.com/nvqza2g A single webpage that provides access to survey, results, and badges

[arokem]

Very cool!

In the meanwhile, we've also managed to set up an instance of the Mozilla Badgekit here:

http://45.55.180.20:3000

As anticipated above, it was a fun (and educational!) couple of days of hacking to get this going. With a bit more elbow-grease, this could possibly provide the backend to the system that Ben and Steven put together to link it up to the Open Badge system. My feeling at the moment is that this would be worthwhile if we anticipated creating other badges, and several different front-ends for different kinds of certification on top of this.

But give it a go around with this, and see what you think. Anyone with a @ uw.edu address should be able to log into our instance.

On Fri, Jun 5, 2015 at 12:05 PM, Steven Roberts notifications@github.com wrote:

An update on our progress ---- http://tinyurl.com/nvqza2g A single webpage that provides access to survey, results, and badges

— Reply to this email directly or view it on GitHub https://github.com/uwescience/reproducible/issues/3#issuecomment-109400242 .

[arokem]

Just to let you know that I will be turning of the service on that machine in a couple of days, so if you haven't had the chance to try it out, please do so before Monday evening.

[benmarwick]

I haven't been able to login at all, but have you spoke with Bill Howe about a potential future for badge kit in escience?

[arokem]

What happened when you tried logging in? Was it just an authentication issue, or was the badge-kit webpage not there at all?

I have not spoken to Bill about this yet.

On Sun, Jun 14, 2015 at 8:58 AM, Ben Marwick notifications@github.com wrote:

I haven't been able to login at all, but have you spoke with Bill Howe about a potential future for badge kit in escience?

— Reply to this email directly or view it on GitHub https://github.com/uwescience/reproducible/issues/3#issuecomment-111843010 .

[benmarwick]

I can see the page ok, I get a message about an incorrect password, I"m attempting to log in with my UWNetID credentials

[arokem]

I see - I think that you need to create a password for your uw email via Persona https://www.mozilla.org/en-US/persona/. I don't remember what exactly we did to get that going.

On Mon, Jun 15, 2015 at 8:45 AM, Ben Marwick notifications@github.com wrote:

I can see the page ok, I get a message about an incorrect password, I"m attempting to log in with my UWNetID credentials

— Reply to this email directly or view it on GitHub https://github.com/uwescience/reproducible/issues/3#issuecomment-112115071 .

[rjleveque]

Some comments:

( form refers to http://goo.gl/forms/jm08DOJ2EI, app refers to https://benmarwick.shinyapps.io/ros_badge/)

• I entered name and affiliation on two separate lines in the form and now there's an entry in the app for "Applied Mathematics, UW" with 0 points. We should just allow one input line for this.
• The form doesn't ask for an ORC ID, but I still got credit for it on the app.
• I agree with @benmarwick's comments from June 4 about some questions not being clear.
• For some categories (e.g. code review) it's not clear if work on software development, e.g. on https://github.com/clawpack in my case, should be counted or whether we only mean to count code related to specific publications.
• Should there be a maximum number of points for some items, e.g. some people might have 50 preprints on the arXiv but have done very little to make code available, should this be gold?

[rjleveque]

Ideas from some discussions the past few days:

• Anyone interested in working more on these badges is welcome to join the working group on Tuesday Sept 22 at 1:30pm in the Data Science Studio for discussion and modification of the form.
• It would be great if we could "finalize" this enough to first test with some others from eScience, and then test more broadly at the Moore/Sloan DSE summit October 5-7. (Sarah has reserved a breakout session time for this.)
• For this testing, it would be good to create another evaluation form where people could give feedback as they are going through the badges form. Questions to ask on this include:
  • better wording of questions?
  • are we asking about the right metrics?
  • usability of form?
  • other suggestions?

[sr320]

If it is helpful in terms of "finalizing" or getting everything ready to distribute- we also created a quick front-end with explanations, links etc during the sprint.

Link: http://htmlpreview.github.io/?https://github.com/sr320/tmp-badge/blob/master/rros-badge-web.html

---

In terms of another form, based on our effort just to get folks to fill out the form we made- it might be better to just have something very simple (page above just started issue). I think most of the improvement will come from just getting people to fill in info and see what pops up in back end. We caught a lot during sprint with the 2-3 persons that filled it out.

[arokem]

Now also available through the easy to remember: http://tinyurl.com/ros-badge

[dacb]

See the discussions from the DSE Summit trial breakout here: https://docs.google.com/document/d/1StKqThnvH0RAU-d8uoE353oWFT3JjfpW10Ola6BwWqo/edit