07. Model for Stain5

[EchteRobert]

In this issue I will post all the results on Stain5 with various models and with various evaluation metrics. I first trained a model on 5 plates from Stain2, Stain3, and Stain4, for a total of 15 training plates. I then use this model to run inference directly on Stain5. After that I fine-tune the model (transfer learning) using 3 plates from Stain5 and 10, 20, 40, and 80% of the available compounds for training/fine-tuning. I also tested this with 1 plate and the same fractions, but multiple plates are required for generalizing to the feature patterns of Stain5. I fine-tune the model by training for 100 epochs and then taking the best validation mAP model. There are perhaps better ways to do this, but this is a first proof-of-concept experiment.

Main takeaways

• As expected, the model does not directly translate to Stain5. This is not due plate/batch effects as those are much smaller. Instead, it is due to different experimental conditions which cause a shift in the features distributions, so that the model is no longer able to correctly aggregated the single cell data. See https://github.com/broadinstitute/FeatureAggregation_single_cell/issues/7#issuecomment-1148750020 for the hierarchical cluster map showing that Stain2, Stain3, and Stain4 are in a different cluster than Stain5 altogether.
• Secondly, fine-tuning the model does increase the performance on plates that are similar to the training data (i.e. using confocal plates will increase performance on confocal plates and using widefield will increase performance on widefield, not both at the same time). However, in order to generalize to unseen compounds I need to use more than 3 plates. This is the same issue I was having before with training the models: using 1 training plate does not generalize to unseen plates and using 3 training plates does not generalize to unseen compounds. So I probably need to use 5 or more plates to generalize to this type of data.

Results

Out of distribution model

| plate | Training mAP model | Training mAP BM | Validation mAP model | Validation mAP BM | PR model | PR BM | Batch | |:-------------------|---------------------:|------------------:|-----------------------:|--------------------:|-----------:|--------:|:-------------| | BR00120530 | 0.26 | **0.28** | 0.23 | **0.39** | 58.9 | 58.9 | CondA PE | | BR00120530confocal | 0.03 | **0.29** | 0.03 | **0.4** | 5.6 | 56.7 | CondA PE | | BR00120526confocal | 0.03 | **0.29** | 0.02 | **0.36** | 3.3 | 58.9 | CondA Thermo | | BR00120526 | **0.35** | 0.28 | **0.38** | 0.37 | 72.2 | 56.7 | CondA Thermo | | BR00120536confocal | 0.06 | **0.25** | 0.05 | **0.37** | 17.8 | 55.6 | CondB PE | | BR00120536 | 0.02 | **0.25** | 0.03 | **0.35** | 4.4 | 54.4 | CondB PE | | BR00120532confocal | 0.05 | **0.24** | 0.06 | **0.35** | 8.9 | 50 | CondB Thermo | | BR00120532 | 0.15 | **0.23** | 0.18 | **0.38** | 36.7 | 50 | CondB Thermo | | BR00120274 | 0.17 | **0.23** | 0.21 | **0.34** | 31.1 | 54.4 | CondC PE | | BR00120274confocal | 0.03 | **0.21** | 0.03 | **0.36** | 4.4 | 52.2 | CondC PE | | BR00120270 | 0.25 | **0.26** | 0.29 | **0.38** | 55.6 | 48.9 | CondC Thermo | | BR00120270confocal | 0.02 | **0.26** | 0.03 | **0.39** | 2.2 | 52.2 | CondC Thermo |

Fine-tuned model 10%

| plate | Training mAP model | Training mAP BM | Validation mAP model | Validation mAP BM | PR model | PR BM | Batch | |:-------------------|---------------------:|------------------:|-----------------------:|--------------------:|-----------:|--------:|:-------------| | BR00120530 | 0.22 | 0.28 | 0.2 | 0.39 | 43.3 | 58.9 | CondA PE | | BR00120530confocal | 0.03 | 0.29 | 0.03 | 0.4 | 5.6 | 56.7 | CondA PE | | BR00120526confocal | 0.03 | 0.29 | 0.02 | 0.36 | 3.3 | 58.9 | CondA Thermo | | BR00120526 | 0.3 | 0.28 | 0.36 | 0.37 | 53.3 | 56.7 | CondA Thermo | | BR00120536confocal | 0.06 | 0.25 | 0.05 | 0.37 | 10 | 55.6 | CondB PE | | BR00120536 | 0.03 | 0.25 | 0.04 | 0.35 | 5.6 | 54.4 | CondB PE | | BR00120532confocal | 0.05 | 0.24 | 0.07 | 0.35 | 11.1 | 50 | CondB Thermo | | BR00120532 | 0.13 | 0.23 | 0.18 | 0.38 | 27.8 | 50 | CondB Thermo | | BR00120274 | 0.16 | 0.23 | 0.2 | 0.34 | 27.8 | 54.4 | CondC PE | | BR00120274confocal | 0.03 | 0.21 | 0.04 | 0.36 | 5.6 | 52.2 | CondC PE | | BR00120270 | 0.23 | 0.26 | 0.28 | 0.38 | 42.2 | 48.9 | CondC Thermo | | BR00120270confocal | 0.03 | 0.26 | 0.03 | 0.39 | 4.4 | 52.2 | CondC Thermo |

Fine-tuned model 20%

| plate | Training mAP model | Training mAP BM | Validation mAP model | Validation mAP BM | PR model | PR BM | Batch | |:-------------------|---------------------:|------------------:|-----------------------:|--------------------:|-----------:|--------:|:-------------| | BR00120530 | 0.22 | 0.28 | 0.23 | 0.39 | 57.8 | 58.9 | CondA PE | | BR00120530confocal | 0.03 | 0.29 | 0.02 | 0.4 | 5.6 | 56.7 | CondA PE | | BR00120526confocal | 0.02 | 0.29 | 0.02 | 0.36 | 2.2 | 58.9 | CondA Thermo | | BR00120526 | 0.32 | 0.28 | 0.31 | 0.37 | 75.6 | 56.7 | CondA Thermo | | BR00120536confocal | 0.05 | 0.25 | 0.05 | 0.37 | 17.8 | 55.6 | CondB PE | | BR00120536 | 0.09 | 0.25 | 0.03 | 0.35 | 23.3 | 54.4 | CondB PE | | BR00120532confocal | 0.05 | 0.24 | 0.06 | 0.35 | 10 | 50 | CondB Thermo | | BR00120532 | 0.27 | 0.23 | 0.21 | 0.38 | 61.1 | 50 | CondB Thermo | | BR00120274 | 0.22 | 0.23 | 0.21 | 0.34 | 53.3 | 54.4 | CondC PE | | BR00120274confocal | 0.03 | 0.21 | 0.04 | 0.36 | 8.9 | 52.2 | CondC PE | | BR00120270 | 0.33 | 0.26 | 0.34 | 0.38 | 75.6 | 48.9 | CondC Thermo | | BR00120270confocal | 0.03 | 0.26 | 0.04 | 0.39 | 3.3 | 52.2 | CondC Thermo |

Fine-tuned model 40%

| plate | Training mAP model | Training mAP BM | Validation mAP model | Validation mAP BM | PR model | PR BM | Batch | |:-------------------|---------------------:|------------------:|-----------------------:|--------------------:|-----------:|--------:|:-------------| | _Fine-tune plates_ | | | | | | | | | BR00120526 | **0.35** | 0.28 | 0.34 | **0.37** | 78.9 | 56.7 | CondA Thermo | | BR00120532 | **0.27** | 0.23 | 0.21 | **0.38** | 57.8 | 50 | CondB Thermo | | BR00120270 | **0.36** | 0.26 | 0.33 | **0.38** | 78.9 | 48.9 | CondC Thermo | | _Hold-out plates_ | | | | | | | | | BR00120530 | 0.26 | **0.28** | 0.22 | **0.39** | 65.6 | 58.9 | CondA PE | | BR00120530confocal | 0.03 | **0.29** | 0.02 | **0.4** | 3.3 | 56.7 | CondA PE | | BR00120526confocal | 0.02 | **0.29** | 0.03 | **0.36** | 3.3 | 58.9 | CondA Thermo | | BR00120536confocal | 0.05 | **0.25** | 0.04 | **0.37** | 17.8 | 55.6 | CondB PE | | BR00120536 | 0.06 | **0.25** | 0.04 | **0.35** | 14.4 | 54.4 | CondB PE | | BR00120532confocal | 0.05 | **0.24** | 0.06 | **0.35** | 15.6 | 50 | CondB Thermo | | BR00120274 | **0.25** | 0.23 | 0.22 | **0.34** | 54.4 | 54.4 | CondC PE | | BR00120274confocal | 0.03 | **0.21** | 0.03 | **0.36** | 4.4 | 52.2 | CondC PE | | BR00120270confocal | 0.03 | **0.26** | 0.03 | **0.39** | 6.7 | 52.2 | CondC Thermo |

Fine-tuned model 80%

| plate | Training mAP model | Training mAP BM | Validation mAP model | Validation mAP BM | PR model | PR BM | Batch | |:-------------------|---------------------:|------------------:|-----------------------:|--------------------:|-----------:|--------:|:-------------| | _Fine-tune plates_ | | | | | | | | | BR00120526 | **0.37** | 0.28 | **0.39** | 0.37 | 85.6 | 56.7 | CondA Thermo | | BR00120532 | **0.3** | 0.23 | 0.26 | **0.38** | 78.9 | 50 | CondB Thermo | | BR00120270 | **0.39** | 0.26 | 0.36 | **0.38** | 87.8 | 48.9 | CondC Thermo | | _Hold-out plates_ | | | | | | | | | BR00120530 | 0.25 | **0.28** | 0.23 | **0.39** | 63.3 | 58.9 | CondA PE | | BR00120530confocal | 0.03 | **0.29** | 0.03 | 0.4 | 8.9 | 56.7 | CondA PE | | BR00120526confocal | 0.03 | **0.29** | 0.03 | **0.36** | 2.2 | 58.9 | CondA Thermo | | BR00120536confocal | 0.06 | **0.25** | 0.06 | **0.37** | 23.3 | 55.6 | CondB PE | | BR00120536 | 0.05 | **0.25** | 0.03 | **0.35** | 26.7 | 54.4 | CondB PE | | BR00120532confocal | 0.05 | **0.24** | 0.08 | **0.35** | 17.8 | 50 | CondB Thermo | | BR00120274 | **0.26** | 0.23 | 0.26 | **0.34** | 60 | 54.4 | CondC PE | | BR00120274confocal | 0.03 | **0.21** | 0.04 | **0.36** | 3.3 | 52.2 | CondC PE | | BR00120270confocal | 0.03 | **0.26** | 0.03 | **0.39** | 8.9 | 52.2 | CondC Thermo |

[EchteRobert]

Number of cells per well per plate histogram

[EchteRobert]

Model trained on Stain5

I trained a model using 5 plates from Stain5, leaving only one plate out (the rest of the plates are confocal). The goal is to see if the same training approach as for Stain234 can be used to generalize to Stain5 plates.

Main takeaways

• The model is not generalizing to Stain5 as easily as it was to the other Stain experiments. Previously models trained on 3 training plates were already able to generalize to most validation compounds. Here, this is not the case even with 5 training plates. There are two main differences which may be making it harder for the model to identify the profiles:
  • The compound concentration is nearly halved (3um --> 1.875um)
  • The cell seeding is halved (2/2.5k cells/well --> 1k cells/well)
• There is something weird going on with plate BR00120536, as it is a training plate, but the model is not able to learn it's representations. I will not investigate this issue further as it currently not a priority.

Stain5 results

| plate | Training mAP model | Training mAP BM | Validation mAP model | Validation mAP BM | PR model | PR BM | Batch | |:-------------------|---------------------:|------------------:|-----------------------:|--------------------:|-----------:|--------:|:-------------| | _Training plates_ | | | | | | | | | BR00120526 | **0.58** | 0.28 | **0.39** | 0.37 | 100 | 56.7 | CondA Thermo | | BR00120536 | 0.17 | **0.25** | 0.03 | **0.35** | 56.7 | 54.4 | CondB PE | | BR00120532 | **0.45** | 0.23 | 0.25 | **0.38** | 90 | 50 | CondB Thermo | | BR00120274 | **0.49** | 0.23 | 0.24 | **0.34** | 87.8 | 54.4 | CondC PE | | BR00120270 | **0.53** | 0.26 | 0.35 | **0.38** | 97.8 | 48.9 | CondC Thermo | | _Validation plates_ | | | | | | | | | BR00120530 | **0.32** | 0.28 | 0.22 | **0.39** | 78.9 | 58.9 | CondA PE | | BR00120530confocal | 0.03 | 0.29 | 0.02 | 0.4 | 3.3 | 56.7 | CondA PE | | BR00120526confocal | 0.03 | 0.29 | 0.02 | 0.36 | 3.3 | 58.9 | CondA Thermo | | BR00120536confocal | 0.05 | 0.25 | 0.06 | 0.37 | 17.8 | 55.6 | CondB PE | | BR00120532confocal | 0.05 | 0.24 | 0.08 | 0.35 | 14.4 | 50 | CondB Thermo | | BR00120274confocal | 0.03 | 0.21 | 0.04 | 0.36 | 11.1 | 52.2 | CondC PE | | BR00120270confocal | 0.03 | 0.26 | 0.03 | 0.39 | 10 | 52.2 | CondC Thermo |