broadinstitute / grit-benchmark

Benchmarking a metric used to evaluate a perturbation strength
BSD 3-Clause "New" or "Revised" License
5 stars 5 forks source link

Figure 1 discussions #23

Open gwaybio opened 3 years ago

gwaybio commented 3 years ago

In #18 I added the first draft of figure 1, lets use this issue to discuss.

figure1

AnneCarpenter commented 3 years ago

For part a, are replicates replicates or are they different guides? Or it depends on whether you're calculating the Grit of a guide vs the Grit of a gene?

Is it intentional that the lines connect the target to each control (vs connecting each replicate to each control)?

I agree with the concern mentioned today about using median vs mean to insulate a bit against cases where a single guide is 'bad' aka ineffective. OTOH, one would want a case of 3 great guides to yield a higher score than 2 great and one terrible guide, so 🤷

gwaybio commented 3 years ago

For part a, are replicates replicates or are they different guides? Or it depends on whether you're calculating the Grit of a guide vs the Grit of a gene?

Yes, good point. I use "replicate" here for convenience in understanding. It really is "group". "replicate" is a kind of "group". So, you can calculate grit w.r.t. replicates (aka same guide different well) or w.r.t. to the gene group (aka all wells of different guides targeting the same gene).

The fact that the software handles this seamlessly is another benefit :)

Is it intentional that the lines connect the target to each control (vs connecting each replicate to each control)?

Yep! Every target gets a grit score. In calculating the target's grit score, we never perform calculations directly between the replicate profiles and the controls. It truly is the target's reproducibility within the group with respect to controls.

I agree with the concern mentioned today about using median vs mean to insulate a bit against cases where a single guide is 'bad' aka ineffective. OTOH, one would want a case of 3 great guides to yield a higher score than 2 great and one terrible guide, so 🤷

Yeah... exactly. Also, at least in the Cell Health data, and empirically, it makes very little difference. Spearman correlations ~.98 (cell 7)