Evaluating criteria of ”best paper” awards

[mlagisz]

Project Lead: @mlagisz

Mentor: @OLSJUJU

Welcome to OLS-6! This issue will be used to track your project and progress during the program. Please use this checklist over the next few weeks as you start Open Life Science program :tada:.

---

Week 1: Meet your mentor!

• [x] Meet mentor for 30 minutes
• [x] Create an account on GitHub
• [x] Check if you have access to the HackMD notes set up for your meetings with your mentor
• [x] Prepare to meet your mentor(s) by completing a short homework provided in your shared notes
• [x] Complete your own copy of the open leadership self-assessment and share it to your mentor If you're a group, each teammate should complete this assessment individually. This is here to help you set your own personal goals during the program. No need to share your results, but be ready to share your thoughts with your mentor.
• [x] Make sure you know when and how you'll be meeting with your mentor.

Week 2 - Cohort Call (Welcome to Open Life Science!)

• [x] Attend call or catch up via YouTube

• [x] Create an issue on the OLS-6 GitHub repository for your OLS work and share the link to your mentor.

• [x] Draft a brief vision statement using your goals

  This lesson from the Open Leadership Training Series (OLTS) might be helpful

• [x] Leave a comment on this issue with your draft vision statement & be ready to share this on the call

• [x] Check the Syllabus for notes and connection info for all the cohort calls.

Week 3: Meet your mentor!

• [x] Meet mentor
• [x] Look up two other projects and comment on their issues with feedback on their vision statement
• [x] Complete this compare and contrast assignment about current and desired community interactions and value exchanges
• [x] Complete your Open Canvas (instructions, canvas)
• [x] Share a link to your Open Canvas in your GitHub issue
• [x] Start your Roadmap
• [x] Comment on your issue with your draft Roadmap
• [x] Suggest a cohort name at the bottom of the shared notes and vote on your favorite with a +1

Week 4: Cohort Call (Tooling and roadmapping for Open projects)

• [x] Attend call or catch up via YouTube
• [x] Look up two other projects and comment on their issues with feedback on their open canvas.

Week 5 and later

• [x] Meet mentor
• [x] Create a GitHub repository for your project
• [x] Add the link to your repository in your issue
• [x] Use your canvas to start writing a README.md file, or landing page, for your project
• [x] Link to your README in a comment on this issue
• [x] Add an open license to your repository as a file called LICENSE.md
• [x] Add a Code of Conduct to your repository as a file called CODE_OF_CONDUCT.md
• [x] Invite new contributors to into your work!

This issue is here to help you keep track of work as you start Open Life Science program. Please refer to the OLS-6 Syllabus for more detailed weekly notes and assignments past week 4.

Week 6

• [x] Attend call or catch up via YouTube

Week 7

• [x] Meet mentor

Week 8

• [ ] Attend call or catch up via YouTube

Week 9

• [ ] Meet mentor

Week 10

• [ ] Attend call or catch up via YouTube

Week 11

• [ ] Meet mentor

Week 12

• [ ] Attend call or catch up via YouTube

Week 13

• [ ] Meet mentor

Week 14

• [ ] Attend call or catch up via YouTube

Week 15

• [ ] Meet mentor

[mlagisz]

DRAFT VISION STATEMENT: This project aims to conduct a survey of current practices in awarding “best papers” awards by research journals, usually given to early career researchers. We will evaluate assessment criteria and historical biases in gender composition of the names of past awardees. Research awards can propagate existing biases in academia in terms of rewarding novel results rather than robust and transparent research. They can also contribute to the “Matthew Effect” where already privileged groups become rewarded. As such, revealing which awards do/do not provide equitable access and evaluation can lead to systemic change in how publications, especially these by early career researchers, are assessed and recognised. We will invite early- and mid-career researchers across disciplines to contribute to the survey. To do so we will organise a hackathon during an upcoming AIMOS2022 conference. We will write a manuscript for an interdisciplinary journal and disseminate findings at international conferences. We expect that our findings will contribute to culture change fostering more equitable and open science.

[mlagisz]

Open Canvas: https://docs.google.com/presentation/d/1E_e_4sDBhyjUsTmf62sK6MSSf7BMy1v04cJwQ0_2fSY/edit?usp=sharing

[amangoel185]

@mlagisz, I really like the aim of the project and I believe it is extremely essential to evaluate such biases and metrics to advance open science in a more diverse and equitable way!

[mlagisz]

Roadmap Draft.

This project aims to assess transparency and equity of the "best paper" awards for individual research publications across disciplines. Such awards are usually given to early-career researchers and can be stepping stones to building careers in research.

To collect data, we will conduct a survey of the publishing landscape. The project will be conducted collaboratively as an open project. We expect participants to be mostly early- and mid-career researchers from across disciplines and countries.

Initial data collection will be done as a hackathon event, with data expanded, checked and cleaned later on in an asynchronous mode. The results of the survey will be disseminated as a preprint, journal publication, blog posts and presentations, hopefully resulting in more transparency and equity in "best paper" research awards.

What do we need to do?

• This comment is a draft roadmap to be refined in the coming weeks.
• Please feel free to comment on it and suggest where it could be improved.
• The milestones are the major stages of the project and the tasks are the potential places where you can contribute to.
• Contribution Guidelines and Code of Conduct will be developed and added to the final Roadmap in a dedicated GitHub repository.

Milestone 1 / Stage 1: Planning.

Timeline: early October - mid November.

During Stage 1, we will prepare all documents and other resources for the project.

• [x] Mission Statement.
• [x] Open Canvas.
• [x] Milestones.
• [x] GitHub repo.
• [x] Hackathon proposal submission.
• [x] Draft protocol.
• [x] Protocol piloting.
• [x] Protocol registration.
• [x] Advertise the hackathon.

Milestone 2 / Stage 2: AIMOS Hackathon.

Timeline: end of November - early December.

During Stage 2, we will organise a hackathon. It will be held during/after AMOS2022 hybrid conference (28-30 November 2022). the hackathon will be an opportunity to introduce people to the project and perform searches for eligible "best paper" awards.

• [x] Contribution guidelines.
• [x] Code of Conduct.
• [x] Lists of journals to be screened.
• [x] Collaboration zone - Google Docs and Zoom.
• [ ] Run the hackathon.
• [ ] Participant contributions.
• [ ] Post-hackathon update.
• [ ] Post-hackathon feedback.

Milestone 3 / Stage 3: Data processing.

Timeline: early September - mid January.

During Stage 3, we collect additional data if needed (finding more awards, data coding), perform quality checks and initial analyses.

• [ ] Additional data.
• [ ] Data extractions.
• [ ] Data quality.
• [ ] Data analyses.
• [ ] Post-analyses update.

Milestone 4 / Stage 4: Reporting.

Timeline: early February - mid April.

During Stage 4, we will conduct final analyses and write a report in a format of a preprint and a journal article. Results will be also disseminated as presentations and blog posts.

• [ ] Final analyses.
• [ ] Draft report.
• [ ] Contributors feedback.
• [ ] Final report.
• [ ] Share data and code in GitHub repository.
• [ ] Preprint in MetaRxiv.
• [ ] Manuscript.
• [ ] Dissemination.

Completed Milestones.

A placeholder for the final version of the Roadmap to be hosted in a GitHub repository.

[Susana465]

wow amazing work! Honestly your issue has inspired me to get my stuff together and crack on with the suggested assignments. Really interesting and important work - I love it!

[r-huo]

Super cool! What an interesting topic to pick up and very meaningful for early stage researchers.

[Romullol]

I really liked the aim of this project! This is a very important subject to be discussed. I liked the organization of your goals as well. Congrats for the great work! =)

[mlagisz]

https://github.com/mlagisz/survey_best_paper_awards

[mlagisz]

https://github.com/mlagisz/survey_best_paper_awards/edit/main/README.md

[macziatko]

This is such a great idea! If we want to see a change we need to support our claims with data. A true researcher spirit for making the future better. I applaud.