FeTS Challenge Task 2 Workflow

[aristizabal95]

Overview

• [x] Evaluating hospitals (Data Owners) create MedPerf accounts.
• [x] DKFZ has an account and is a benchmark owner and model owner
• [x] MLCommons are admins (Alejandro, Johnu, Micah)
• [x] FeTS challenge has it's own dedicated MedPerf server (www.fets.medperf.org or similar)
• [x] DKFZ registers the reference benchmark pipeline and test dataset
• [ ] Evaluating hospitals register their datasets
• [ ] Evaluating hospitals run the test pipeline against their data to test compatibility/compute resources
• [ ] Competitors implement MLCubes according to template provided
• [ ] Competitors submit their MLCubes to DKFZ via Synapse
• [ ] DKFZ evaluates submissions for compliance, hosts and registers them
  • [ ] Executes script that runs the compatibility test with the provided model
  • [ ] Stores a list of cubes that succeeded, emails back competitors with failed submissions
• [x] MedPerf team provides script to evaluating hospitals to download and verify submissions
• [ ] MedPerf team provides script to evaluating hospitals to run each model in the benchmark (results are not uploaded at this point)
• [ ] Evaluating sites (optionally) review and upload all results
• [ ] DKFZ can view the results in MedPerf for scoring and analysis

DKFZ

• [x] Work with MLCube and MedPerf teams to create competitor instructions and materials
  • [x] Documentation for challenge competitors
  • [x] Documentation for Data participants
• [x] Create DKFZ medperf account
• [x] Collect MedPerf account names from evaluating hospitals and pass list to MedPerf admins
  • [x] Medperf admins create accounts for all evaluating hospitals
  • [x] Send credentials to hospitals
• [x] Create FeTS Challenge Benchmark (expand this)
  • [x] Implements data preparation mlcube
  • [x] Implements reference model mlcube
  • [x] Implements metrics mlcube
  • [x] Host benchmark mlcubes
  • [x] Host test dataset
  • [x] Submits data preparation mlcube
  • [x] Submits reference model mlcube
  • [x] Submits metrics cube
• [x] Collect submissions
• [x] Vet submissions
  • [x] For non-compliant submissions, contact competitors as able
• [x] Host submissions
• [x] Register submissions in MedPerf
• [x] Associate submissions to the benchmark
• [x] Send list of submission URLs and hashes to MedPerf team
• [x] Wait for results
• [ ] Analyze results in MedPerf to determine challenge outcome

Evaluating Hospitals

• [ ] Gather their QC'd/labeled data onto one system, in file structure defined by DKFZ
• [ ] Install medperf
• [ ] Download FeTS Challenge package (includes MedPerf, extra scripts, scripts can download benchmark cubes)
• [ ] Provide DKFZ desired MedPerf user account name
• [ ] Receive account name and password from MedPerf admins (via DKFZ??)
• [ ] Test MedPerf login
• [ ] Change user password (password change API required)
• [ ] Use script to download and verify reference mlcubes
• [ ] Test Benchmark reference pipeline
  • [ ] check for correct use of GPU
• [ ] Run data preparation step and register with medperf
• [ ] Request benchmark association (this exercises full pipeline on their data)
• [ ] Use script to download and verify all competitor MLCubes
• [ ] Use script to evaluate all competitor MLCubes. Results are not uploaded in this step.
• [ ] Sanity check results to ensure no leaked information
• [ ] Use script to upload results to MedPerf

Competitors

• [x] Download MLCube template and instructions
• [x] Create MLCube for their model
• [x] Optionally test with MedPerf offline mode
• [x] Upload MLCube to synapse
• [x] Respond to any issues found by DKFZ in vetting stage.

MedPerf Team

• [x] Create FeTS Challenge MedPerf server instance by July 1st @ fets.medperf.org (or similar)
• [x] Create test MedPerf server instance by July 1st @ jaham.medperf.org
• [x] Create additional admin accounts
• [x] Create DKFZ account (or is this a shared FeTS organizer account?)
• [x] Receive account names for Evaluating Hospitals
• [x] Create Evaluating Hospital accounts
  • [x] Alpha group by July 3rd, pending getting required info
• [x] Send temporary passwords to Evaluating Hospitals (via DKFZ??)
  • [x] Alpha group by July 3rd, pending getting required info
• [x] Review Evaluating Hospital instructions for alpha testers (by July 3rd)
• [x] Micah, Alejandro and Hasan to run alpha evaluator instructions by July 3rd
  • [x] Identify test data
• [x] Host FeTS Benchmark Singularity Cubes for alpha evaluators
  • [x] Download from Synapse
  • [x] Convert to singularity and tar
  • [x] Upload singularity cubes to GCP
• [x] Host test dataset for alpha evaluators (get from Synapse)

[mzenk]

Thanks for starting this :) Just two minor remarks/corrections:

• Correct link to the current model cube template
• The submission data prep/metrics cubes (also singularity) have to be downloaded as well (apart from submissions) in a script before MedPerf starts.

[hasan7n]

oudated