Create profiling recipe for DeepProfiler output

[shntnu]

@Arkkienkeli has produced Cell Painting CNN v1 embeddings for all of JUMP. Currently these are at the single-cell level.

We now need a workflow that does everything downstream.

We will use this issue to discuss and plan and likely move this to a new repo.

@jccaicedo and @shntnu will create a Snakemake workflow similar to https://github.com/broadinstitute/jump-profiling-recipe, but will start at the single cell level (Level 2) not at the replicate level (Level 3)

Source:https://github.com/cytomining/pycytominer/blob/main/media/pipeline.png

Here are the steps based on this notebook recommended by @Arkkienkeli

• Site-level median
• Well-level mean
• Experiment-level Sphering
• Treatment-level mean
• No feature selection, no outlier removal, and no per-feature normalization (i.e. no standardization, only Sphering)

We will need input from

• @Zitong-Chen-16 for Snakemake
• @ashah03 for figuring out how to profitably use SkyPilot if we can and maybe figuring out how to get write access to the staging bucket

Also from

• @niranjchandrasekaran for general pointers about JUMP
• @ErinWeisbart for general pointers about workflow, because this workflow will eventually end up in Cimini lab's hands

---

Location of data

• we will need to rename the folder Cell_Patining_CNN_conv6a_b3667957 to fix the typo, when we sync it over to s3://cellpainting-gallery
• @Arkkienkeli seems to have used parquets instead of npz, and this works well!

---

Notes

• Slack discussions are here https://broadinstitute.slack.com/archives/C06CZUGR1C5/p1709341592744799
• The artefacts should be structured like described here https://broadinstitute.github.io/cellpainting-gallery/data_structure.html#workspace-dl-folder-structure. We can propose changes to the structure if it doesn't make sense.
• Mike Ando is planning to produce well level profiles for EffNet embeddings created using a Terra workflow https://github.com/jump-cellpainting/datasets-private/issues/50#issuecomment-1982217038. However, if our workflow is general enough, we can use our workflow to process it instead (and we will keep him posted in which case)
• Whatever we produce through this effort will likely become a de facto standard – or at least a baseline – in the field :) So while we don't want to overthink this, we should certainly justify each step to some reasonable extent.

[jccaicedo]

@viditagr and I have started with level-2 to level-3 aggregation and want to implement the output in the correct format. @shntnu can you point us to how the aggregated files are supposed to be structured, named and organized in the directories?

[jccaicedo]

What we are doing is to create a parallel directory called profiles and follow the specified directory structure with batches and plates. Inside we save a single file with the well-level aggregated data in parquet format.

[shntnu]

can you point us to how the aggregated files are supposed to be structured, named and organized in the directories?

I missed replying to this

Please see below

https://broadinstitute.github.io/cellpainting-gallery/data_structure.html#workspace-dl-folder-structure

[shntnu]

Update from Caicedo lab, which I am posting here for @jessica-ewald's visibility:

We are making progress with the aggregation, but we found a few corrupted parquet files. Nikita is helping fix this, and we will also take the opportunity to fix the typo in the Cell Painting directory names. After this is complete, the aggregated data will be available together with the fixed data.