[Story] Review the organization contribution types

Sage-Bionetworks / sage-monorepo

Where OpenChallenges, Schematic, and other Sage open source apps are built

https://sage-bionetworks.github.io/sage-monorepo/

Apache License 2.0

23 stars 12 forks source link

[Story] Review the organization contribution types #2345

Open tschaffter opened 10 months ago

tschaffter commented 10 months ago

What product(s) is this story for?

OpenChallenges

As a user, I want

No response

Description

[!IMPORTANT]
This initial goal of this ticket was to clarify one type of organization contribution (sponsor) to address a feedback received from DREAM. The goal of this ticket is now extend to the review of all the org contribution types.

We received this feedback from DREAM:

Also, the use of the word 'sponsor' on the new site is confusing. In the grant world the 'sponsor' is the organization paying for the project. We don't pay for any challenges. We receive funding from sponsors for the challenges. I am just not clear how 'sponsor' is being used in the context of that site although maybe it means platform being used/organization hosting?

What about using "benchmarking community" for organizations like DREAM, Kaggle, MICCAI, etc.?

Acceptance criteria

A clear definition is defined for each contribution role
A contribution role is defined for DREAM and similar organization
The OC Data have been updated
If needed, the backend and frontend have been updated

Tasks

No response

Anything else?

No response

Have you linked this story to a GitHub Project?

[X] I have linked this story to a GitHub Project and set its metadata.

gaiaandreoletti commented 10 months ago

I agree using "benchmarking community" instead of sponsors or possible even "crowdsourcing community"

gaiaandreoletti commented 10 months ago

added it ina separate tab and ready for discussion

tschaffter commented 10 months ago

@gaiaandreoletti Can you please add an update to this ticket? We usually do it like this:

## Update yyyy-mm-dd

Description of where this ticket stands, whether there are blockers, expected completion date.

gaiaandreoletti commented 10 months ago

Updated 2023-11-20

added it in a separate tab and ready for discussion with the main group. I would consider the ticket closed.

tschaffter commented 10 months ago

@gaiaandreoletti See the section "Acceptance criteria" of this ticket that lists the requirements that must be met before we can close this ticket.

A clear definition is defined for each contribution role => Are these contribution roles defined somewhere?
A contribution role is defined for DREAM and similar organizations => What are the organizations that have been assigned the new contribution role (should depend on the approved definition of Benchmarking community)?
If needed, the backend and frontend have been updated => Neither Rong nor I have done this. The above steps must be completed first before Rong or I can start this step.

Cc @vpchung

gaiaandreoletti commented 10 months ago

@tschaffter I was hoping we would have discussed this during the OC meeting. A tab now called "Benchmarking Community" in the OpenChallenges Data lists all Benchmarking communities. These include:

DREAM
BioCreative
CAFA
CAGI
CAMDA
CAMEO
CAMI
CAPRI
CASP
GAW
iDash
MICCAI
precisionFDA
sbv IMPROVER
Kaggle
CACHE
OpenEBench
CodaLab
Codabench
OpenML
PapersWithCode
Grand Challenge
Health Data Hub
PhysioNet
HeroX
BioHackathons

Definition: Benchmarking Community: Benchmarking communities in biological data involves evaluating and comparing their performance, characteristics, and attributes to establish standards and identify patterns. This practice provides a framework for understanding their structure, composition, and dynamics. Communities engaged in benchmarking often seek funding from sponsors to support their activities. The financial support obtained from sponsors plays a crucial role in the success and sustainability of benchmarking efforts.

tschaffter commented 10 months ago

Looking good in general.

Benchmarking communities in biological data involves evaluating and comparing their performance [...]

What do you mean by "the performance of a benchmarking community"? Maybe mention tools and (AI/ML) models whose performance is evaluated.

Another elements that could be added to the description is that a benchmarking community may focus on very specific tasks (e.g. CAGI, CAMI) or benchmark a broad range of biomedical tools (e.g. DREAM that benchmarks various tasks like gene network reconstruction, breast cancer detection from imaging data, etc.).

Please add the updated description to a Google doc so that we can find it easily when new contributions are added to the DB as well as for enable Jake and other to contribute to the description until it's approved.

tschaffter commented 10 months ago

Added to Sprint 23.12

tschaffter commented 8 months ago

Added to Sprint 24.1

tschaffter commented 8 months ago

@gaiaandreoletti See "Important" note added to the first comment of this ticket.

tschaffter commented 7 months ago

@gaiaandreoletti Can you please share an update for this task?

gaiaandreoletti commented 7 months ago

2024-02-09 Glossary updated and ready for review by the team

vpchung commented 7 months ago

@gaiaandreoletti the doc looks good to me!! Should we expose this on the OC soon?

gaiaandreoletti commented 7 months ago

Thanks @vpchung, if @tschaffter and @rrchai @maria agrees I would say yes!

tschaffter commented 7 months ago

@gaiaandreoletti What do you think about surfacing this call for comments in the Slack challenges channel to invite feedback from outside of the OC core team?

gaiaandreoletti commented 7 months ago

2024-02-19 Glossary updated after review from the team and this doc as well (https://docs.google.com/document/d/1lRvBLGYhTOBKhOw_SzWBxcfogzsIvTwUIkUmfe78yvI/edit?pli=1)

tschaffter commented 7 months ago

@gaiaandreoletti Can you share an update about this task?

gaiaandreoletti commented 7 months ago

Will share the new revised doc on the slack for final approval by the group