Thought leadership project to determine a way to make it easy for scientific communities to peer-review public-good software (or contributions to such software) #21
Potential Activity Scope
This thought leadership project aimed at determining a way to make it easy for scientific communities to peer-review public-good research software (or contributions to such software) seeks to bridge the gap between traditional scientific peer review and the evolving landscape of open-source software contributions.
Core aspects of this activity could include:
Understanding the Landscape: The project begins with a comprehensive understanding of the current landscape of public-good research software and the existing mechanisms for peer review. This involves analyzing the types of software, the communities involved, the nature of contributions, and the standards and practices for quality assessment.
Identifying Challenges and Opportunities: The next step is to identify the specific challenges and opportunities related to peer-reviewing public-good research software. This may include issues such as lack of standardized review criteria, difficulty in assessing code quality, challenges in recognizing and crediting contributions, and opportunities for collaboration and community engagement.
Engaging with Stakeholders: A critical aspect of the project is engaging with various stakeholders, including researchers, developers, reviewers, publishers, funders, and policymakers. Their insights, experiences, and perspectives are essential in shaping a solution that is practical, relevant, and widely accepted.
Developing a Sustainable Framework or Platform: Based on the insights gathered, the project aims to develop a framework or platform that facilitates the peer review of public-good research software. This may involve defining clear review criteria, creating guidelines and tools for reviewers, establishing mechanisms for recognizing and crediting contributions, and integrating with existing publishing and funding processes.
For a project addressing this activity with more time/resources, the following elements could also be useful:
Piloting and Refining: Before wide-scale implementation, a promising solution could be piloted with selected scientific communities. This allows for testing, refinement, and adaptation based on real-world experiences and feedback.
Dissemination and Adoption: Once refined, the solution should be disseminated widely within the scientific community. This involves promoting awareness, providing training and support, and collaborating with publishers, funders, and institutions to encourage adoption.
Monitoring and Continuous Improvement: The project should include ongoing monitoring and continuous improvement to ensure that the solution remains effective, relevant, and responsive to the changing needs and dynamics of scientific communities and public-good research software.
Fostering a Culture of Open Review: Beyond the technical solution, the project also seeks to foster a culture of open review and collaboration within scientific communities. This involves promoting values, norms, and practices that encourage transparency, inclusivity, and excellence in the review of public-good research software.
A key assumption underpinning this activity is that peer-review is considered a mark of quality.
Examples of possible ideas to explore include establishing a new journal or working with existing journals
or conferences, or via existing community organizations or software-focused organizations to advance this agenda.
Potential Objectives
To conduct a comprehensive analysis of the current landscape of public-good research software and existing mechanisms for peer review, understanding the types, communities, contributions, and standards involved.
To identify specific challenges and opportunities related to peer-reviewing public-good research software, including issues such as standardization, code quality assessment, recognition, and collaboration.
To engage with various stakeholders, including researchers, developers, reviewers, publishers, funders, and policymakers, gathering insights, experiences, and perspectives to shape the solution.
To develop a practical and widely accepted framework or platform that facilitates the peer review of public-good research software, defining clear criteria, guidelines, tools, recognition mechanisms, and integration with existing processes.
To pilot the proposed solution with selected scientific communities, testing, refining, and adapting it based on real-world experiences and feedback.
To disseminate the refined solution widely within the scientific community, promoting awareness, providing training and support, and encouraging adoption.
To monitor and continuously improve the solution, ensuring it remains effective, relevant, and responsive to the changing needs and dynamics of scientific communities and public-good research software.
Targeted Impacts
To improve the way public-good research software is reviewed, creating a robust and accessible system that ensures quality, credibility, and impact.
To empower scientific communities to actively engage in expert review of public-good research software, fostering collaboration, innovation, and excellence.
To influence policies and practices within research institutions, publishers, and funders in support of public-good research software.
To contribute to the broader movement towards open science, collaboration, and transparency within the scientific community.
Ultimately, the impact of this activity can be gauged using the following types of criteria:
Adoption Rates: The number of scientific communities, publishers, and institutions that adopt the developed framework or platform for peer review. A higher adoption rate would indicate broader acceptance and success.
Impact on Public-Good Research Software: The impact of the peer-review process on the quality, credibility, and utilization of public-good research software. This could include metrics such as citation rates, collaboration rates, or improvements in software quality.
Stakeholder Engagement:The level of engagement and collaboration with various stakeholders, including researchers, developers, reviewers, publishers, and policymakers. Success in this area would reflect a more inclusive and responsive approach.
Cultural Shift: Changes in attitudes, norms, and practices within scientific communities related to the review of public-good research software. Surveys, interviews, and case studies could reveal shifts towards more openness, collaboration, and excellence.
Continuous Improvement: Ongoing adaptation and improvement of the framework or platform, reflecting responsiveness to feedback, changes in the field, and emerging needs and opportunities.
Dissemination and Awareness: The reach and impact of dissemination efforts, such as workshops, publications, social media engagement, etc. Success in this area would indicate effective communication and awareness-raising.
Alignment with Open Science Principles: How well the initiative aligns with and contributes to broader open science principles and goals. This could involve assessing transparency, accessibility, inclusivity, and other key values.
Economic and Resource Efficiency: The cost-effectiveness and resource efficiency of the initiative, ensuring that it provides value and sustainability in its approach.
This potential activity was curated as part of "Charting the Course: Policy and Planning for Sustainable Research Software," a Sloan Foundation-funded project within URSSI dedicated to supporting the future of research software through evidence-informed policy work (Project contacts are: @danielskatz and @dr-eric-jensen). If you are interested in working on this, please add a comment.
Potential Activity Scope This thought leadership project aimed at determining a way to make it easy for scientific communities to peer-review public-good research software (or contributions to such software) seeks to bridge the gap between traditional scientific peer review and the evolving landscape of open-source software contributions.
Core aspects of this activity could include:
Understanding the Landscape: The project begins with a comprehensive understanding of the current landscape of public-good research software and the existing mechanisms for peer review. This involves analyzing the types of software, the communities involved, the nature of contributions, and the standards and practices for quality assessment.
Identifying Challenges and Opportunities: The next step is to identify the specific challenges and opportunities related to peer-reviewing public-good research software. This may include issues such as lack of standardized review criteria, difficulty in assessing code quality, challenges in recognizing and crediting contributions, and opportunities for collaboration and community engagement.
Engaging with Stakeholders: A critical aspect of the project is engaging with various stakeholders, including researchers, developers, reviewers, publishers, funders, and policymakers. Their insights, experiences, and perspectives are essential in shaping a solution that is practical, relevant, and widely accepted.
Developing a Sustainable Framework or Platform: Based on the insights gathered, the project aims to develop a framework or platform that facilitates the peer review of public-good research software. This may involve defining clear review criteria, creating guidelines and tools for reviewers, establishing mechanisms for recognizing and crediting contributions, and integrating with existing publishing and funding processes.
For a project addressing this activity with more time/resources, the following elements could also be useful: Piloting and Refining: Before wide-scale implementation, a promising solution could be piloted with selected scientific communities. This allows for testing, refinement, and adaptation based on real-world experiences and feedback.
Dissemination and Adoption: Once refined, the solution should be disseminated widely within the scientific community. This involves promoting awareness, providing training and support, and collaborating with publishers, funders, and institutions to encourage adoption.
Monitoring and Continuous Improvement: The project should include ongoing monitoring and continuous improvement to ensure that the solution remains effective, relevant, and responsive to the changing needs and dynamics of scientific communities and public-good research software.
Fostering a Culture of Open Review: Beyond the technical solution, the project also seeks to foster a culture of open review and collaboration within scientific communities. This involves promoting values, norms, and practices that encourage transparency, inclusivity, and excellence in the review of public-good research software.
A key assumption underpinning this activity is that peer-review is considered a mark of quality. Examples of possible ideas to explore include establishing a new journal or working with existing journals or conferences, or via existing community organizations or software-focused organizations to advance this agenda.
Potential Objectives
Targeted Impacts
Ultimately, the impact of this activity can be gauged using the following types of criteria:
This potential activity was curated as part of "Charting the Course: Policy and Planning for Sustainable Research Software," a Sloan Foundation-funded project within URSSI dedicated to supporting the future of research software through evidence-informed policy work (Project contacts are: @danielskatz and @dr-eric-jensen). If you are interested in working on this, please add a comment.