Embeddings directory structure

[shntnu]

Related: https://github.com/jump-cellpainting/aws/issues/85#issuecomment-1245809221

[ErinWeisbart]

@Arkkienkeli A couple questions that I'm happy integrate into the documentation before merging:

• For someone with only basic knowledge of deep learning, can you describe in 1 sentence each what is found in the collated, consensus, and embeddings folders?
• For efficientnet_v2_imagenet1k_s_feature_vector_2_ec756ff which part of the string is an identifier for the deep learning network and which part the hash for the model?

For @shntnu

• Must the various .parquet files be .parquet or do we allow other file types? .csv.gz? .sqlite?

[Arkkienkeli]

@ErinWeisbart collated - contains csv or parquet files with well-level profiles for all plates. consensus - contains csv or parquet files with treatment-level profiles. embeddings - contains npz or parquet files with single-cell features (extracted from single full image).

In this example, efficientnet_v2_imagenet1k_s_feature_vector_2_ec756ff -> efficientnet - name of the network (EfficientNet), imagenet1k - the dataset that was used for training, ec756ff - hash. We also agreed with @shntnu that it is possible to use other identifiers for model, for example zenodo doi (see folder structure for DeepProfiler).

[ErinWeisbart]

Thanks so much Nikita. I've moved things around a little bit and added some additional information to the documentation based on your help. @Arkkienkeli and @shntnu would you take a look to make sure that what I've added is correct? Then I'm happy to have this merged.

[Arkkienkeli]

@ErinWeisbart Looks good to me! Thank you.

[shntnu]

@ErinWeisbart LGTM!

Thanks for the edits :)

[shntnu]

I've moved things around a little bit and added some additional information to the documentation based on your help. @Arkkienkeli and @shntnu would you take a look to make sure that what I've added is correct? Then I'm happy to have this merged.

@ErinWeisbart @Arkkienkeli many thanks! Merging now