Frozen data version 1

[gwaybio]

I update pycytominer and add associated fixes as described in #62

TODO

• [x] Check example batch 2 data
• [x] Run full pipeline for both batches, add profiles
• [x] Rerun cytominer vs. pycytominer comparison notebook
• [x] Rerun consensus signature notebook - note that I will no longer need to recode dose
• [x] Rerun spherize notebook

In the next PR, I will migrate from git lfs to dvc

[gwaybio]

@shntnu - this is good to go. Sorry for the HUGE amounts of files (most are just profiles).

Please pay extra attention to any updated documentation. Any code changes will require a complete rerun (which I'll only do if absolutely necessary). If necessary, I can address #65 simultaneously.

The next step will be to update to dvc!

[shntnu]

Please pay extra attention to any updated documentation.

I focused on only .py and .md

I didn't see any documentation changes other than the README.md

Did I miss any documentation?

Any code changes will require a complete rerun (which I'll only do if absolutely necessary). If necessary, I can address #65 simultaneously.

No need to do #65 except perhaps this thing you suggested:

I can add a prominent note to make sure these are dropped in all downstream analyses in a README in #63

[gwaybio]

Did I miss any documentation?

Nope, I think you got it all. Thank you!

I can make all of these changes, and we should be good to merge soon

[gwaybio]

Alright @shntnu - this is ready for your eyes again. Here is what changed:

• Adding note to how we decided to remove features 8a794f9630c4f18110d119bbc8cc2bd70cc3db8b
• Rearranging documentation and minor fixes 6adb1c3b52ff485b74dacf9b7247989333610509
• Adding the new blocklist features 510fe92c0c0d1ca2646cc955133ab4fd574349f6
• Using the new blocklist feature file in feature selection 2201875748de880a3b987946786fd7b92719d456
• Adding the updated consensus files 49ec1c774a427059c3ee264aefa2b80477c3c93c
• Fixing sphering (sphering based on time as well and dropping new blocklist features) 7c96161c81d223612dca9c02c0bc5eb3be38a26c
• Adding spherized profiles 3a47efd44ca8324162469d218a74870083dbeea0
• Adding a UMAP visualization of spherized profiles acb3e89600438a1560e8bc6967d257037e90e0c3

[gwaybio]

Do you want to update this as well? https://figshare.com/articles/dataset/Blacklist_Features_-_Cell_Profiler/10255811

I don't think so... Although i do think that we want to update this figshare document to include other version-specific CellProfiler blocklists. At the very least, much more thought needs to go into updating it (much more thought for me at least!). As a separate but related note: I really want to do a deep dive into CellProfiler features... i think its the first step to understanding generic morphology features, which we'll want to annotate with more interpretable biology. It'll also help us with interpreting DeepProfiler features in the future.

[shntnu]

As a separate but related note: I really want to do a deep dive into CellProfiler features... i think its the first step to understanding generic morphology features, which we'll want to annotate with more interpretable biology. It'll also help us with interpreting DeepProfiler features in the future.

There are two resources that I can think of that will be relevant for this effort

1. A well-documented readme https://github.com/carpenterlab/2016_bray_natprot/wiki/What-do-Cell-Painting-features-mean%3F
2. An incomplete notebook https://github.com/cytomining/cytominergallery/blob/105be87878d13024ef283e284cf085351b498883/notebooks/empty_readouts.Rmd (knit here https://rpubs.com/shantanu/cp_feature_stats)