[Regulatory Effect Prediction][Metric] Subsampling Stability

[katosh]

What is the metric? We subsample expression or atac signals before imputation and compare multiple results to assess stability.

How should it be implemented? Parameters quantile and quantile_cells=1 to dictate which quantile of read should be sampled and how many cells should be affected.

Which task(s) could it be used for? Imputation and potentially any other method that can be benchmarked for stability with respect to the count matrix.

Edit: The project card is: https://github.com/orgs/singlecellopenproblems/projects/3#card-57959969

[LuckyMD]

Would this be a new task? Or is it related to a particular method output that is already produced in a task?

[katosh]

We decided to do metrics first and use dummy tasks to test them. I am just starting to get to know the framework. Initially, I assumed the metric would get the input data and apply a task to it. But looking at the api it seems the metric only receives the result of a specific task. Is that right?

[katosh]

I the latter case (the metric only receives one result) I would suggest to implement a meta-method that subsamples and applies the same method multiple times before passing on the combined result. I think for this purpose I can store the results in e.g. ad.layers or ad.uns.

[LuckyMD]

Is that right?

Yes. A metric is only meaningful within the confines of a task in open problems. The particular metric you're suggesting here actually looks like a combination of a data loader (subsampled data), and a metric (imputation stability). This would probably only be meaningful for a task that alters the expression (or other omic) data directly, but not something like label projection. So maybe this could be suggested for tasks such as denoising, batch integration, or multimodal integration.

[katosh]

I am happy to move the task to the appropriate topic. It came up as a metric for imputation (3. in https://docs.google.com/document/d/1mbbfAAQJBKNrmskwxy2p3TN6zJ-hVA_euWqKnk0LwEU/edit#heading=h.5uzp1ncjrw70).

[katosh]

To be more specific, I imagine this to be a bootstrap. We would do n different subsamples and apply the same method on the individual subsamples. Then we calculate the variance of the result to determine the stability of the applied method. I believe this can be a good consistency check for many methods but I aim to implement it specifically for imputation.

[LuckyMD]

Okay, this makes a lot more sense now. The multiple subsampling aspect is actually not that straightforward to implement in open problems. At the moment we require a strict separation of dataset, method, and metric. So if you have an imputation method, then you would need 1 version of the data, each method to work on this data input, and then all metrics to evaluate each method's output on this version of the data. Hence metrics will not know about multiple subsamplings.

The way that @emdann and @dburkhardt are currently getting around this in the differential abundance task is to set the task API such that each data loader must have multiple versions the loaded data, the method must then pull these apart and work on each version, and concatenate the output again, and a metric evaluates this concatenated output. That constitutes a fully new task definition.

Ideally we can find a way to make this type of evaluation more easy in the future. Maybe as a wrapper around a task rather than a metric within the task itself.

[katosh]

Could I make the metric specific for a type of task that returns multiple results? Then I could leave the subsampling to a specialized meta task that applies a given method on the subsamples and bundles the results.

[katosh]

I am also in the Regulatory effect prediction voice chat channel if you like to talk about it.

[emdann]

Hi @katosh. To provide multiple subsamples to one method you could implement the subsampling algorithm with the data loader and store list of indices of each sample in adata.uns, together with a seed/ID. Then the method can do the filtering of cells to use internally.

[katosh]

Thanks @emdann for the input. I talked with @LuckyMD and some interesting ideas came up how bootstrapping may be integrated into the open problems backend in the future. For now, we will focus on single dataset evaluations of any imputation methods. A metric will compare the imputed results with some ground truth. This will be part of another task.

@LuckyMD You mentioned the denoising task. However, it seems this task focuses on the RNA expression. My goal is to use multisomal data to do imputation for the RNA expression and the ATAC signal with focus on ATAC. Do you this can be realized in the denoising task or do we need a specific ATAC denoising?

[LuckyMD]

@LuckyMD You mentioned the denoising task. However, it seems this task focuses on the RNA expression. My goal is to use multisomal data to do imputation for the RNA expression and the ATAC signal with focus on ATAC. Do you this can be realized in the denoising task or do we need a specific ATAC denoising?

I would probably make this a separate task... maybe as a subtask of regulatory effect prediction in the end then.

[katosh]

Is there something special for the implementation of a subtask or can I just follow the tutorial on implementing a new task?

[LuckyMD]

new task is fine... if you like, you could create an extra wrapper folder with a preceding _ to signify a parent task name... but that can always be done later.

[katosh]

Alright, I am already on it :)

[katosh]

The PR is here: https://github.com/singlecellopenproblems/SingleCellOpenProblems/pull/296