Image-based Profiling Template

This repository was derived from a template repository located at https://github.com/cytomining/profiling-template. The purpose of the repository is to weld together a versioned data processing pipeline with versioned processed output data for a single image-based profiling experiment.

(Derived from this template)

DO THE FOLLOWING AFTER GENERATING A NEW REPO:

• Change the title of the README to Image-based Profiling for [ProjectName](project-url).
• Delete the Setup section
• Keep the notes section, and edit as necessary.
• Keep the metadata section if you are adopting it, and edit as necessary.

Setup

To correctly initialize the repository, we need to perform several manual steps.

Step 0: Create a new repository using this repository as a template

By spinning up a new repo using this repo as a template, you will retain all code, configuration files, computational environments, and directory structure that a standard image-based profiling workflow expects and produces.

Step 1: Fork the profiling-recipe repo

We first want to fork the official profiling recipe located at https://github.com/cytomining/profiling-recipe.

• Result: The fork creates a copy of a recipe repository.
• Goals: 1) Remove the connection to official recipe updates to avoid unintended weld versioning reversal; 2) Enable independent updates to fork code that does not impact official recipe.
• Execution: See forking instructions.

Step 2: Create a submodule inside this repository of the forked recipe

Next, we will create a submodule in this repo.

• Result: Adding a submodule initiates the weld.
• Goals: 1) Link the processing code (recipe) with the data (current repo); 2) Require a manual step to update the recipe to enable asynchronous development.
• Execution: See below

# In your terminal, clone the repository you just created (THIS REPO)
USER="INSERT-USERNAME-HERE"
REPO="INSERT-NAME-HERE"
git clone git@github.com:$USER/$REPO.git

# Navigate to this directory
cd $REPO

# Add the recipe submodule
git submodule add https://github.com/$USER/profiling-recipe.git profiling-recipe

Refer to "Adding a submodule" for more details.

Step 3: Commit the submodule

Lastly, we will commit the submodule to github.

• Result: Committing this change finalizes the weld.
• Goals: 1) Track the submodule (recipe) version with the current repository.
• Execution: See below

# Add, commit, and push the submodule contents
git add profiling-recipe
git add .gitmodules
git commit -m 'finalizing the recipe weld'
git push

Step 4: Fill out experiment information

This step ensures that relevant experimental information is captured. We recommend all users, both internal and external, to diligently track project metadata. For those without a system, a template is provided below.

Metadata template

Fill out the whole template on the first batch; for subsequent batches only add what is different.

Notes

To download the data in this repo, first, clone this repo

git clone git@github.com:<org>/<repo>.git

then, download the profiles

cd <repo>
dvc pull

AWS configuration

The DVC cache is typically stored in an AWS S3 bucket, so you will need run aws configure before running dvc pull.

If the DVC location is not publicly accessible, you will need AWS credentials to access it.

If the DVC location is not publicly accessible, to access the files stored via DVC, you will need to created a IAM user with the AmazonS3ReadOnlyAccess policy attached:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:Get*",
                "s3:List*",
                "s3-object-lambda:Get*",
                "s3-object-lambda:List*"
            ],
            "Resource": "*"
        }
    ]
}