Create Binarization and Cell Creation Module

[pickles-bread-and-butter]

Overview:

There needs to be a module that takes a Suite2p output stat.npy file and binarizes the masks and writes them into a .json file in ROI format. There already exist methods for doing the binarization and loading from Suite2p file in Ophys ETL Pipelines.

Tasks:

• [ ] Create new method for transforming masks to LIMS format
• [ ] Create new schema'ed entry point that (1) loads from stat.npy (exists) (2) binarizes (exists) (3) writes to LIMS-formatted json file (see above task). Entry point uses validated argschema inputs and output schemas.

Validation Criteria:

• [ ] ROI file format as defined in design doc method is fully tested.
• [ ] A module exists with a CLI that can run a stat.npy file through and produce binarized ROIs in a ready-for-LIMS output json.
• [ ] ingestion of output into dev LIMS cell_rois table is demonstrated.
• [ ] The module is fully-tested, including its entry point. Entry point tests are unit tests with mocked functionality and not integrations tests.

[pickles-bread-and-butter]

SIngularity command for binarizing and creating ROIs should be something very similar to Singurality run suite2p_ophys_etl.simg python -m convert_rois --json input.json This can be run in conjunction with the suite2p wrapper by running Singularity run suite2p_ophys_etl.simg python -m suite2p_wrapper --json input.json && Singurality run suite2p_ophys_etl.simg python -m convert_rois --json input.json This command should realistically produce all rois from suite2p and then binarize and convert the format.

[wbwakeman]

Thanks for providing the command. I guess this helps me understand why I was asking about it. Right now Nick is working on the strategy, which includes the formatting for how LIMS will write the command that gets run. It essentially defines the command as: string saved in the LIMS database some arguments saved in lims --input_json /path/input.json --output_json /path/output.json

So, this leads to two questions:

1. Can the two module commands be combined into one?
2. Can the module take in the path to the output json file so that it explicitly knows where to write the file

It also gets me wondering whether any 'bind' arg is needed so that the singularity container can read/write from the Isilon file system. e.g. --bind /allen/programs/braintv/production/:/allen/programs/braintv/production

[pickles-bread-and-butter]

To answer your questions 1: Yes they can be run as one command by specifying double & characters between the two commands 2: Yes that is currently how the module operates, one field in the input.json will need to be called output_json and this is the file path that we want to save the generated rois to.

The bind variables I'm not 100% sure about as I don't know too much about file permissions on the Isilon, @njmei would you konw if we need to specify anything to have read/write permissions?

[wbwakeman]

Right. I guess what I was asking was whether they could be combined into a single command that matches the design pattern I specified in that comment, essentially: command --input_json /path/input.json --output_json /path/output/json

Also, I see now that we wrote up the design document as having the output path in the input file. I'm not extremely opposed to this, but it would be nice to match the LIMS design pattern that is commonly used for most recent modules.

[njmei]

@wbwakeman @isaak-willett I realized this as well, we should be making use of Argschema's --output_json arg instead of specifying the output json within the input json data structure.

These are the default fields that Argschema provides regardless of additional Schema you add:

  --input_json INPUT_JSON
                        file path of input json file
  --output_json OUTPUT_JSON
                        file path to output json file
  --log_level LOG_LEVEL
                        set the logging level of the module (default=ERROR)

[pickles-bread-and-butter]

Using --output_json will work with the module as designed. @wbwakeman we can absolutely have them as one command but that will require either a custom entry point in the singularity image to run both modules or another python file on top of suite2p_wrapper and create_rois to tie them together. Neither option is too terribly difficult, I would personally prefer having another python file if we're going to try and have one command for both pieces in the workflow step.

[wbwakeman]

Great. Sorry I missed that output_json subtlety while we were working on the design doc. I will update it now.

I'll leave it up to the team on the best way to combine those two commands. It would be great if you can post the single example command when you have it. Sorry if this is pain at the last minute; it will make the LIMS integration a lot more straightforward.