openproblems-bio / task_label_projection

Automated cell type annotation from rich, labeled reference data
https://openproblems.bio/results/label_projection/
MIT License
0 stars 0 forks source link

Label projection != label transfer #1

Open rcannood opened 2 months ago

rcannood commented 2 months ago

This benchmark does not benchmark a method's performance on labelling a totally new dataset, given an existing reference dataset. Instead, it takes a large dataset and then uses one batch as test holdout.

mxposed commented 2 months ago

Hi Robrecht! Your comment brings up two questions:

  1. This task was set up like this, because the correct labels on the test part were easily available. How would we get ground truth (or just correct labels) for label_transfer task between two datasets?
  2. When transferring labels from one dataset to another, there can be a situation where some (or all) query cell types are not represented in the reference. I was thinking of emulating this by withholding some cell types in label_projection task. How do you think this should be handled in the label_transfer?