Evaluation metric

[LorenzoMerotto]

As mentioned during the last meeting, I post here to discuss about the fairness of the correlation coefficient as the evaluation metric. In this case I selected the 'uniform' scenario, without mRNA bias added (since I didn't had TPM values yet).

What we can see is that for the CD4/CD8 T cells the correlation coefficients are not so good. Since we have a few cell types the simulated fractins for each one would be low. With the correlation coefficients we get an insight on how the relative differences across samples are maintained in the estimated cell proportions, but I wonder if this is a right evaluation metric in this scenario.

• Scatterplot with the individual cell types

• Scatterplot with all cell types together

• Simulated and reconstructed fractions

Simulated	Reconstructed

@FFinotello proposed to consider the sum of squared errors, which I think would be good.

Alternatives?

• Simulate less samples
• Compare the results intra-sample: however I fear that we would have the same problem when using the uniform scenario

[FFinotello]

Hi @LorenzoMerotto thanks for sharing this data. I understand your point, but I would like to have a deeper look also at the other results.

Meanwhile, here are some unstructured thoughts:

• Scaden results looks pretty good although the "even" scenario is very simple;
• There is a systematic bias in the "all cell types" plot; I would test it in a more realistic scenario where you simulate mRNA bias with a data-driven approach --@alex-d13 can give you some directions on the best strategy available in the latest version of the simulator;
• Adding RMSE values is definitely informative but you also "count" the effect of having systematically over/under-estimated values (i.e values over/under the diagonal);
• Having per-sample plots (with dots colored by cell type) is also informative and should show you if the method performs particularly well or bad in some samples, and you should see fewer outliers. It's an orthogonal piece of information. And definitely, a point where you could discuss with @constantin-zackl: using deconvExplorer, you could generate interactively different "views" of the same output data.
• I still see the two outlier points in the CD4 plot as... outliers. The "even" scenario is very simple because it spans a big range of values, whereas in a real scenario some rare cells (e.g. DCs) are confined in a much narrower range. Even in this case, it is important that estimates align along the bisector because this allows you to say (correctly) that we have more DCs in sample A than in sample B.

[mlist]

It's a really important point raised here by @LorenzoMerotto and I agree that correlation coefficients do not always best reflect the outcome. As @FFinotello points out neither does the sum of squares since it will not account for global shifts in the same way as the correlation does. So indeed both measures are complementary. I'd propose to also fit a simple linear model as we considered for evaluating Simbu before. In a linear model, one can look at the intercept as a global shift and the slope as a global bias. Maybe intercept, slope and sum of squares together will give a good impression of the results?