Development: Create guidance giving examples of specific metrics for the success of scientific software

[danielskatz]

Create a set of guidance giving examples of specific metrics for the success of scientific software in use, why they were chosen, what they are useful to measure, and any challenges/pitfalls; then publish this as a white paper

[npch]

Group discussion is here: https://docs.google.com/document/d/1lVLTpjUzrLBfV0LxCQg3G6ArVgJKlfG75gtWi6Qqms8/edit

[dansellars]

CANARIE Research Software registry polls APIs on registered software.

See https://science.canarie.ca/researchmiddleware/platforms/list/main.html

Program information: http://www.canarie.ca/software/

[npch]

Summary of work carried out on this topic at WSSSPE3.

1 What is a summary of the opportunity / challenge and key obstacles?

Metrics important for

• Tenure and promotion
• Scientific impact
• Discovery
• Reducing duplication
• Basis for potential industrial interest in adopting software
• Make case for funding
• No commonly-used standard for collecting or presenting metrics
• We don’t know if there is a common set of metrics
• We have to persuade projects that it is useful to collect metrics

2 Key next steps

• Skype phone call to coordinate in two or three weeks time
• Get started on IRB at Illinois (may need more thought into survey)
• Get started on white paper and associated survey

3 Are there folks who will be meeting to work toward those steps? Yes:

• Neil Chue Hong
• Emily Chen
• Ray Idaszak
• Iain Larmour
• Fraser Watson
• Gabrielle Allen gdallen@illinois.edu
• Dan Sellars
• Bernie Randles

Plus we will be contacting people who participated in a similar session at the 2015 NSF SI2 PI workshop, and this is open to others who may want to opt-in.

4 What's the plan for organizing yourselves going forward

• We have a white paper outline with sections assigned to the above individuals, plus see Section 2 response above for timeline.
• Organizing coordinating phone call

5 What else do you need

• IRB approval for surveys, collecting data
• Coordination with 2016 NSF SI2 PI workshop organizing committee to possibly piggyback on this event to offer survey to attendees in advance
• Coordination (mail communication, info page etc), via WSSSPE github or?

6 What are the key milestones and who is responsible

• Paper outline: https://docs.google.com/document/d/1MhbHPfk-o0_7ajNcmP1pq91Ojo642jA5nfLzwiXtKUA/edit (google doc link can be found on WSSSPE GitHub: https://github.com/danielskatz/WSSSPE/issues/48 )
• Roadmap for White Paper production
  • October: IRB paperwork as appropriate (Gabrielle, Emily for NSF)
  • October - December: Draft sections 1-3 (Paper outline has info of who does what)
  • October - December: Run surveys and collect information (Piggy back on NSF SI2 PIs meeting)
  • January - February 2016: Analyse results of data collection from projects
  • March - April 2016: Draft sections 4-7
  • May 2016: Draft section 8-9
  • May - June 2016: Get initial feedback from members of community and revise
  • (July 2016?) By time of next CFP for WSSSPE have complete draft of white paper
  • (Expectation) Responses to White Paper submitted to WSSSPE4

7 Any funding possibilities?

• Not discussed in our meeting, potentially down the road.

[npch]

If you are interested in contributing to the white paper, please leave a comment on this issue.

[danielskatz]

comment :)

[kyleniemeyer]

so commented!

[smarru]

a little comment :+1:

[iliant]

I'm interested, commented.

[GBruceB]

I am interested also

[knarrff]

ping

[JeffCarver]

I would like to contribute

[sctchoi]

I think I can help.

On Sep 29, 2015, at 15:52, JeffCarver notifications@github.com wrote:

I would like to contribute

— Reply to this email directly or view it on GitHub.