search

broadinstitute / lincs-cell-painting

Processed Cell Painting Data for the LINCS Drug Repurposing Project
BSD 3-Clause "New" or "Revised" License
25 stars 13 forks source link

Adding spherized profiles #60

Closed gwaybio closed 3 years ago

gwaybio commented 3 years ago

I spherize all plates of batch 1 data using all DMSO profiles as a reference. I apply feature selection to the full dataframe of concatenated level 4a data, and output the spherized data.

I also needed to change the name of a script from profile.py to profile_cells.py. This solves the issues I described in https://github.com/broadinstitute/lincs-cell-painting/issues/59#issuecomment-794050963

Merge steps

gwaybio commented 3 years ago

Note, I didn't perform any sanity checks in this data

gwaybio commented 3 years ago

I added batch two spherized profiles after merging #60

gwaybio commented 3 years ago

In https://github.com/broadinstitute/lincs-cell-painting/pull/48#issuecomment-795380885, @shntnu noted:

cytomining/pycytominer#128. If the default value of epsilion=1e-6 is fine, then we needn't fix that issue right now. How would we know whether it is fine or not? I suppose we can just do it very crudely and empirically for now: do the results improve similar to what we've seen in past analysis by Ted et al.?

If they do, then epsilion=1e-6 is fine and there's nothing to be done here. If they don't improve then we need to think harder about the plan Update: I just noticed #60 so you're all set to figure out whether there's anything to be done here.

I agree that an empirical test that reproduces the improvement Ted saw in regards to non-spherized vs. spherized data would make us all set. However, I don't think we do it in this pull request, and not even in this repo. Instead, @adeboyeML can take these profiles and run them through the pipeline he created in https://github.com/broadinstitute/lincs-profiling-comparison.

So, I propose the following:

  1. I sanity check that all batch 2 plates were processed (prob should have done in #58 ...)
  2. I make some minor modifications to this PR
  3. @shntnu reviews and we merge this PR once he approves
  4. @adeboyeML runs these data through his replicate reproducibility assessment pipeline
  5. If we see similar improvement as Ted, then we're good. If not, we need to adjust epsilon to what Mohammad used (this value is around somewhere...)

edit, i'll add the PR-specific steps to the beginning of this PR

gwaybio commented 3 years ago

@shntnu - this is now ready for your eyes, when you get a chance

gwaybio commented 3 years ago

Did to mean to propose the PR merge after this step, not before?

we merge first, then Adeniyi checks using data from the merge

shntnu commented 3 years ago

we merge first, then Adeniyi checks using data from the merge

Got it. My concern was bloating the repo in case you need to reprocess. But I trust your judgment in figuring of what order works best.

Excited to have this in the repo!!

PS – if you do need to end up replacing, I'd recommend actually deleting the files as I did here https://github.com/jump-cellpainting/pilot-cpjump1-data/pull/9#issuecomment-802756355

gwaybio commented 3 years ago

PS – if you do need to end up replacing, I'd recommend actually deleting the files as I did here jump-cellpainting/pilot-cpjump1-data#9 (comment)

Awesome, this is good to keep in mind.

We might at some point also consider moving from gitLFS to dvc. It was super easy to get setup, and plays very nicely with AWS. I did this in the grit-benchmark repo ( in broadinstitute/grit-benchmark#28)

In the most recent commit, I added a bunch of comments to two different README files. We might want to edit them before first official release, but we can open a new, documentation-focused PR then. I am going to merge!

shntnu commented 3 years ago

The notebook says

Here, we load in all normalized profiles (level 4a) data across all plates and apply a spherize transform using the DMSO profiles as the background distribution.

but it should say

Here, we load in all normalized profiles (level 4a) data across all plates, apply the standard set of feature selection operations, and then apply a spherize transform using the DMSO profiles as the background distribution.

It's not worth updating anything; I'm just adding a note here for ourselves.

gwaybio commented 3 years ago

good catch. I added #77 so we can make sure to improve (I agree it is not urgent, but someone new could start there (good practice for editing a file using github 😄 ))