Open pablo-gar opened 9 months ago
Is there a reason that this isn't just kept as internal validation so that users can rely on is_primary_data
curation?
There are cells marked as is_primary_data = False
that are not true numerical replicates of the counter-part primary cells. Modelers have expressed interest in including those for modeling, but currently there is no way to identify them easily.
Can you clarify why modelers want to identify numerical replicates? Is it so they can avoid/filter them? Or something else?
@pablo-gar - is there a 1/2 pager or related census issue that motivated your prototype?
After a call we decided on investigating more about the value of this field for external users to understand better how to best support the users.
I recommend to the following tasks that I can own:
Timeline:
DEFERRED DUE TO INTERNAL ORG CHANGES
1. Interviews 3-5 users (modelers).
2. Characterize the level of granularity.
I've been looking into this a bit, and wanted to add a couple points here:
I think we should have a more nuanced and detailed discussion of what exactly the is_primary_data
field means and why you may want to use it in the docs.
Some examples of when you may want to use this:
is_primary_data==False
I haven't been able to run the finddups.ipynb
notebook on recent tiledbsoma releases, so can't see the current status. However, I think there are multiple reasons to believe that it may have false positives
data
and indptr
, while it should be hashing the data
and indices
. This means that the hash isn't actually taking into account which genes are expressed, just the level of expression for a set of unknown genes and number of genes expressed.That said, I do think this is probably useful for identifying samples shared between datasets, just might be wrong on individual cells.
I think I would like to know which observations are actually the same cell, but I don't know that I think we can do that with high confidence at the moment. In large part this is due to the curation process.
Motivation
Census users with modeling needs have expressed interest to identify true numerical duplicates for cells, as compared to
is_primary_data
which is curation-based and in some cases a cell that is not primary data is also not a true numerical duplicate.In addition, this would help us identify errors of
is_primary_data
annotations where two or more duplicate cells have been annotated as `is_primary_data=True.Proposed metadata variable
A variable in
obs
that creates an ID based in the numerical vector of counts per each cell. We have built a prototype in Census as to how implement this -- it boils down to hashing the numerical vectors for each cell.See the prototype here. https://github.com/chanzuckerberg/cellxgene-census/blob/main/tools/cell_dup_check/finddups.ipynb