Fix cell neighbors matrix/clustering dropping for cells that don't have any neighbors

[camisowers]

Please refer to our FAQ and look at our known issues before opening a bug report.

Describe the bug Currently, the kmeans neighborhood analysis notebook runs with a neighborhood matrix as the input. This is a matrix with each cell as a row, and the columns are all the cell phenotypes. However, for cells that have no neighbors within the specified radius (i.e. a row of zeros in the matrix), these rows are dropped from the data. Subsequently, such cells are excluded from the kmeans clustering and are missing from the new cell table saved at the end of the notebook.

Expected behavior This data is excluded from clustering to avoid an additional cluster containing cells with no close neighbors. There are a few options:

does not change clustering

• We can continue to drop these cells from the neighbors matrix, but merge the post clustering cell table with the old cell table and assign NaN values in the cluster column for any missing cells

potentially changes clustering

• Issue a warning with what percentage of cells would be potentially dropped, and suggest an increase in the distlim radius
• Currently, cells do not count themselves as a neighbor. We could set self_neighbor=True so that even cells in the middle of nowhere would have at least one neighbor and thus be clustered

[ngreenwald]

Can you take a look at the TONIC dataset and see what fraction of cells this applies to?

[camisowers]

Seems to automatically drop only 1-2% of cells. We will issue a warning for any more than 5%, suggest an increase in the pixel radius, and ensure that cells without a cluster assignment are still retained in the cell table with NaN value.