Open shntnu opened 5 months ago
@viditagr and I have started with level-2 to level-3 aggregation and want to implement the output in the correct format. @shntnu can you point us to how the aggregated files are supposed to be structured, named and organized in the directories?
What we are doing is to create a parallel directory called profiles
and follow the specified directory structure with batches and plates. Inside we save a single file with the well-level aggregated data in parquet format.
can you point us to how the aggregated files are supposed to be structured, named and organized in the directories?
I missed replying to this
Please see below
Update from Caicedo lab, which I am posting here for @jessica-ewald's visibility:
We are making progress with the aggregation, but we found a few corrupted parquet files. Nikita is helping fix this, and we will also take the opportunity to fix the typo in the Cell Painting directory names. After this is complete, the aggregated data will be available together with the fixed data.
@Arkkienkeli has produced Cell Painting CNN v1 embeddings for all of JUMP. Currently these are at the single-cell level.
We now need a workflow that does everything downstream.
We will use this issue to discuss and plan and likely move this to a new repo.
@jccaicedo and @shntnu will create a Snakemake workflow similar to https://github.com/broadinstitute/jump-profiling-recipe, but will start at the single cell level (Level 2) not at the replicate level (Level 3)
Here are the steps based on this notebook recommended by @Arkkienkeli
We will need input from
Also from
Location of data
Cell_Patining_CNN_conv6a_b3667957
to fix the typo, when we sync it over to s3://cellpainting-galleryNotes