search

theislab / scib-reproducibility

Additional code and analysis from the single-cell integration benchmarking project
https://theislab.github.io/scib-reproducibility/
MIT License
52 stars 14 forks source link

Need help calculating task overall metric scores #26

Closed mvfki closed 1 month ago

mvfki commented 1 month ago

Hi scib team,

I'm recently working on new integration methods and have been using scib to evaluate the outcomes. Thanks for the well packaged tool!

One problem I just encountered is that I couldn't reproduce the overall scores from the published scores of the tasks in the scib paper. I tried it with the sheet for human immune task from supplementary file 1. So if I take the five batch removal scores from the second row (the first task), I couldn't reproduce the same number for the overall batch correction score, with the method mentioned in the Method section of the paper (average of min-max scaled scores). Can you please clarify the calculation of this part so that I can have overall scores calculated for my integration methods and know how they ranks with existing methods you've evaluated?

Thanks! Yichen

mvfki commented 1 month ago

Oh I got it now after one night sleep. For one task (e.g. human immune), you put all scores of all integration in a table one run per row one metric per column. Then by min-max scaling you actually did it column-wise with NAs ignored. And finally the meann was taken for each integration run each category (i.e. bio/batch).