Bundle structure for SmartSeq2

[kbergin]

The current mechanism for SS2 bundles is that each cell is in it's own bundle. This keeps the data organized in the smallest possible increments and allows the user the most flexibility. It produces thousands of bundles for each SS2 experiment, which then creates thousands of workflows and thousands of analysis results.

There has been discussion on which way is the best way to store the data for users and for the DCP scaling capacity. Other suggestions include meta-bundle structures that allow analyses to be performed in groups of cells but keep the smallest unit of storage possible.

RFC: https://docs.google.com/document/d/1ur1aqA-UnJOVklS9eL0wzP-j2Rmd1oC0LnV2JQJUrhU/edit

AC:

• [ ] Ingestion of bundle structure for SS2 is updated
• [ ] Pipeline can run on arbitrary cells per bundle size
• [ ] Testing in integration is updated
• [ ] Analysis outputs are formatted in a similar way to 10x data and are consistent with Matrix service's needs

[kbergin]

@hannes-ucsc Kevin said you were thinking about this and asked me to tag you!

[theathorn]

@mweiden Scaling SLOs may require this.

[hannes-ucsc]

Has this RFC been accepted?