Exploring preprocessing pipelines in Scenario 1

[johnarevalo]

We report the performance of multiple preprocessing steps:

• mad: Median absolute deviation normalization
• clip: clip outlier values to 500.
• drop: drop any column with an oulier value.
• imputemedian: impute outliers with median value.
• imputeknn: impute outiler values with knn.
• sphering: whitening transformation estimated with negative controls and applied over the whole dataset.
• featselect: Feature selection process using "variance_threshold", "correlation_threshold", operations from pycytominer.
• int: rank-based Inverse normal transformation.

The best performing pipeline is mad_drop_int_featselect and mad_int_featselect

Performance comparison

 

---

We also tried sphering as an additional step to help aligning values. We searched for the best regularization parameter for the top-2 pipelines described above.

Sphering does not increase the performance metrics

Sphering vs other pipelines (including some batch correction methods)

  

Sphering exploration


[johnarevalo]

@shntnu Based on the results above, we will try both mad_drop_int_featselect and mad_int_featselect for other scenarios and will report the best. We are not going to include sphering.

[shntnu]

Thank you for documenting all this. Is it reasonable to say that sphering may have been improving results in the past because it was compensating for some of the issues with the data that are now being fixed by these other new preprocessing steps? Of course, it's hard to test that specifically, but that's the only hypothesis I could come up with.

Also, can you link to the sphering code you used?

[johnarevalo]

Is it reasonable to say that sphering may have been improving results in the past because it was compensating for some of the issues with the data that are now being fixed by these other new preprocessing steps?

Yes, I also think that's a reasonable explanation.

I'm using a copy from your pycytominer PR.

https://github.com/carpenter-singh-lab/2023_Arevalo_BatchCorrection/blob/aac9bfd2854984cf0309fcb6786abb0358ec0e7f/zca.py#L16

[shntnu]

Yes, I also think that's a reasonable explanation.

I'm using a copy from your pycytominer PR.

Sounds good

For our notes, we decided it's better to use a copy because we were worried about package conflicts if we updated pycytominer. We could have used a separate conda env for each rule, but this (copying) is simpler (for now :D)