Research study to reveal key factors in the academic impact of research software

dr-eric-jensen commented 12 months ago

Potential Activity Scope Identify factors (manual, such as evaluations, and automated, such as crawling repos) that are part of impact and publicize them. Some known factors to investigate include quantifying code contributions, code review, mentoring. Determine good practices (for formatting or housing the information on these factors) so those factors are discoverable and/or queryable.

Potential Objectives

To identify both manual and automated factors that are part of impact in the realm of software development, such as evaluations, crawling repositories, quantifying code contributions, code review, and mentoring.
To investigate known factors and potentially uncover new ones that contribute to the understanding of impact within the software development community.
To determine good practices for formatting or housing the information on these factors, ensuring that they are easily discoverable and/or queryable.
To create guidelines or tools that facilitate the implementation of these good practices, supporting organizations and individuals in making use of the identified factors.
To publicize the identified factors and good practices, engaging with the software development community and other relevant stakeholders to promote awareness and adoption.
To evaluate the effectiveness of the identified factors and good practices, gathering feedback and making continuous improvements to ensure they remain relevant and useful.

Targeted Impacts If successful, this effort could help deliver the following impact objectives:

To enhance the understanding of impact within the software development community, providing a more nuanced and comprehensive view of what contributes to success and value in this field.
To foster a culture of transparency and accountability in software development, by promoting the use of clear and accessible metrics for evaluating impact.
To support the professional development of software developers, by providing clear guidelines and tools for tracking and demonstrating their contributions and achievements.
To facilitate collaboration and knowledge sharing within the software development community, by making information on impact factors easily discoverable and queryable.
To contribute to the quality and effectiveness of research software development by promoting good practices that encourage thoughtful evaluation, continuous improvement, and a focus on meaningful impact.

This potential activity was curated as part of "Charting the Course: Policy and Planning for Sustainable Research Software," a Sloan Foundation-funded project within URSSI dedicated to supporting the future of research software through evidence-informed policy work (Project contacts are: @danielskatz and @dr-eric-jensen). If you are interested in working on this, please add a comment.

mdbarker commented 9 months ago

See Research Software Visibility Infrastructure Priorities Report by Karthik Ram and James Howison, DOI 10.5281/zenodo.8404846

Particularly recommendations 3 and 7, both in terms of their explanation of the problem, and recommendations:

Create a software use infrascope as an observatory based on software mentions in publications to highlight particular areas of strength or opportunity in Australia. Software lacks visibility in research, making it challenging to identify crucial software for specific research areas. We recommend that the ARDC play a role in developing a “software use infrascope” based on recognizing software in the full text of publications to systematically understand the software used by Australian researchers, categorized by research topics. This tool would enable stakeholders to identify software trends, support training programs, and assess the impact of software packages, enhancing research collaboration and funding opportunities.
Provide guidance for implementing web analytics to understand usage. Software projects require reliable data to showcase their impact and understand their user base. While traditional metrics like download counts have credibility issues, focus group participants emphasized the potential of website analytics to gauge impact. We recommend that the ARDC promote the use of web analytics on the documentation and websites of research software projects, offering guidance, training, and resources to maximize their benefits.

danielskatz commented 9 months ago

the clickable link to the report: https://doi.org/10.5281/zenodo.8404846

si2-urssi / policy

Research study to reveal key factors in the academic impact of research software #18