Cannot set custom colors for categorical variables

[LucaCalderoni]

Hi! I am testing napari-spatialdata on a public Nanostring Cosmx dataset (https://kero.hgc.jp/Breast_Cancer_Spatial.html), and I would like to set a custom color for each label of a specific categorical variable, i.e., the cell types obtained with an external tool. I tried to manually insert a dictionary in the uns slot of the anndata object used as 'table' in the spatialdata object, in the format {:, :, ..... }. However, after I load the spatialdata object on napari and I select the desired annotation in the 'observations' panel on the right, an error message pops up, and the annotation is not visualized at all. Here is the full error:

######################################################################################################

---------------------------------------------------------------------------

AssertionError                            Traceback (most recent call last)

File ~/anaconda3/envs/SpatialData_prova/lib/python3.12/site-packages/napari_spatialdata/_widgets.py:65, in ListWidget.__init__.<locals>.<lambda>(item=<PyQt5.QtWidgets.QListWidgetItem object>)

     62 self._unique = unique

     63 self._viewer = viewer

---> 65 self.itemDoubleClicked.connect(lambda item: self._onAction((item.text(),)))

        self = <napari_spatialdata._widgets.AListWidget object at 0x7f989cf30680>

        item = <PyQt5.QtWidgets.QListWidgetItem object at 0x7f98dff132f0>

     66 self.enterPressed.connect(self._onAction)

     67 self.indexChanged.connect(self._onAction)

File ~/anaconda3/envs/SpatialData_prova/lib/python3.12/site-packages/napari_spatialdata/_widgets.py:142, in AListWidget._onAction(self=<napari_spatialdata._widgets.AListWidget object>, items=('InSituType_Simple',))

    139 features["index"] = index

    140 self.model.layer.features = features

--> 142 properties = self._get_points_properties(vec, key=item, layer=self.model.layer)

        item = 'InSituType_Simple'

        vec = cell_ID

1                  BC cells

2                  BC cells

3         Mix BC cells TAMs

4         Mix BC cells TAMs

5                  BC cells

               ...         

1850              Blood ECs

1851             Mast cells

1852    Myoepithelial cells

1853      Mix BC cells TAMs

1854                   CAFs

Name: InSituType_Simple, Length: 1832, dtype: category

Categories (12, object): ['BC cells', 'Blood ECs', 'CAFs', 'DCs', ..., 'NK cells', 'Plasma cells',

                          'T cells', 'TAMs']

        self = <napari_spatialdata._widgets.AListWidget object at 0x7f989cf30680>

    143 self.model.color_by = "" if self.model.system_name is None else item

    144 if isinstance(self.model.layer, (Points, Shapes)):

File ~/anaconda3/envs/SpatialData_prova/lib/python3.12/functools.py:946, in singledispatchmethod.__get__.<locals>._method(*args=(cell_ID

1                  BC cells

2           ...ls',

                          'T cells', 'TAMs'],), **kwargs={'key': 'InSituType_Simple', 'layer': <Labels layer '11_labels'>})

    944 def _method(*args, **kwargs):

    945     method = self.dispatcher.dispatch(args[0].__class__)

--> 946     return method.__get__(obj, cls)(*args, **kwargs)

        method = <function AListWidget._ at 0x7f98df9387c0>

        obj = <napari_spatialdata._widgets.AListWidget object at 0x7f989cf30680>

        cls = <class 'napari_spatialdata._widgets.AListWidget'>

        args = (cell_ID

1                  BC cells

2                  BC cells

3         Mix BC cells TAMs

4         Mix BC cells TAMs

5                  BC cells

               ...         

1850              Blood ECs

1851             Mast cells

1852    Myoepithelial cells

1853      Mix BC cells TAMs

1854                   CAFs

Name: InSituType_Simple, Length: 1832, dtype: category

Categories (12, object): ['BC cells', 'Blood ECs', 'CAFs', 'DCs', ..., 'NK cells', 'Plasma cells',

                          'T cells', 'TAMs'],)

        kwargs = {'key': 'InSituType_Simple', 'layer': <Labels layer '11_labels' at 0x7f989d46d520>}

File ~/anaconda3/envs/SpatialData_prova/lib/python3.12/site-packages/napari_spatialdata/_widgets.py:248, in AListWidget._(self=<napari_spatialdata._widgets.AListWidget object>, vec=cell_ID

1                  BC cells

2           ...ls',

                          'T cells', 'TAMs'], **kwargs={'key': 'InSituType_Simple'})

    245 else:

    246     merge_df = pd.merge(element_indices, vec, left_on="element_indices", right_index=True, how="left")

--> 248 merge_df["color"] = merge_df[[vec.name](http://vec.name/)].map(color_dict)

        merge_df =       element_indices    InSituType_Simple

0                   1             BC cells

1                   2             BC cells

2                   3    Mix BC cells TAMs

3                   4    Mix BC cells TAMs

4                   5             BC cells

...               ...                  ...

1849             1850            Blood ECs

1850             1851           Mast cells

1851             1852  Myoepithelial cells

1852             1853    Mix BC cells TAMs

1853             1854                 CAFs

[1854 rows x 2 columns]

        color_dict = array([{'BC cells': '#ff4500', 'CAFs': '#f7e9c7', 'Mix BC cells TAMs': '#800080', 'Myoepithelial cells': '#daa520', 'Blood ECs': '#efd594', 'Mural cells': '#a77e18', 'T cells': '#001b00', 'Plasma cells': '#ff728a', 'TAMs': '#0000ff', 'DCs': '#000037', 'NK cells': '#003400', 'Mast cells': '#add8e6'}],

      dtype=object)

        vec = cell_ID

1                  BC cells

2                  BC cells

3         Mix BC cells TAMs

4         Mix BC cells TAMs

5                  BC cells

               ...         

1850              Blood ECs

1851             Mast cells

1852    Myoepithelial cells

1853      Mix BC cells TAMs

1854                   CAFs

Name: InSituType_Simple, Length: 1832, dtype: category

Categories (12, object): ['BC cells', 'Blood ECs', 'CAFs', 'DCs', ..., 'NK cells', 'Plasma cells',

                          'T cells', 'TAMs']

    249 if layer is not None and isinstance(layer, Labels):

    250     index_color_mapping = dict(zip(merge_df["element_indices"], merge_df["color"]))

File ~/anaconda3/envs/SpatialData_prova/lib/python3.12/site-packages/pandas/core/series.py:4700, in Series.map(self=0                  BC cells

1                  B...ls',

                          'T cells', 'TAMs'], arg=array([{'BC cells': '#ff4500', 'CAFs': '#f7e9c7'...', 'Mast cells': '#add8e6'}],

      dtype=object), na_action=None)

   4620 def map(

   4621     self,

   4622     arg: Callable | Mapping | Series,

   4623     na_action: Literal["ignore"] | None = None,

   4624 ) -> Series:

   4625     """

   4626     Map values of Series according to an input mapping or function.

   4627 

   (...)

   4698     dtype: object

   4699     """

-> 4700     new_values = self._map_values(arg, na_action=na_action)

        arg = array([{'BC cells': '#ff4500', 'CAFs': '#f7e9c7', 'Mix BC cells TAMs': '#800080', 'Myoepithelial cells': '#daa520', 'Blood ECs': '#efd594', 'Mural cells': '#a77e18', 'T cells': '#001b00', 'Plasma cells': '#ff728a', 'TAMs': '#0000ff', 'DCs': '#000037', 'NK cells': '#003400', 'Mast cells': '#add8e6'}],

      dtype=object)

        self = 0                  BC cells

1                  BC cells

2         Mix BC cells TAMs

3         Mix BC cells TAMs

4                  BC cells

               ...         

1849              Blood ECs

1850             Mast cells

1851    Myoepithelial cells

1852      Mix BC cells TAMs

1853                   CAFs

Name: InSituType_Simple, Length: 1854, dtype: category

Categories (12, object): ['BC cells', 'Blood ECs', 'CAFs', 'DCs', ..., 'NK cells', 'Plasma cells',

                          'T cells', 'TAMs']

        na_action = None

   4701     return self._constructor(new_values, index=self.index, copy=False).__finalize__(

   4702         self, method="map"

   4703     )

File ~/anaconda3/envs/SpatialData_prova/lib/python3.12/site-packages/pandas/core/base.py:919, in IndexOpsMixin._map_values(self=0                  BC cells

1                  B...ls',

                          'T cells', 'TAMs'], mapper=array([{'BC cells': '#ff4500', 'CAFs': '#f7e9c7'...', 'Mast cells': '#add8e6'}],

      dtype=object), na_action=None, convert=True)

    916 arr = self._values

    918 if isinstance(arr, ExtensionArray):

--> 919     return arr.map(mapper, na_action=na_action)

        arr = ['BC cells', 'BC cells', 'Mix BC cells TAMs', 'Mix BC cells TAMs', 'BC cells', ..., 'Blood ECs', 'Mast cells', 'Myoepithelial cells', 'Mix BC cells TAMs', 'CAFs']

Length: 1854

Categories (12, object): ['BC cells', 'Blood ECs', 'CAFs', 'DCs', ..., 'NK cells', 'Plasma cells',

                          'T cells', 'TAMs']

        mapper = array([{'BC cells': '#ff4500', 'CAFs': '#f7e9c7', 'Mix BC cells TAMs': '#800080', 'Myoepithelial cells': '#daa520', 'Blood ECs': '#efd594', 'Mural cells': '#a77e18', 'T cells': '#001b00', 'Plasma cells': '#ff728a', 'TAMs': '#0000ff', 'DCs': '#000037', 'NK cells': '#003400', 'Mast cells': '#add8e6'}],

      dtype=object)

        na_action = None

    921 return algorithms.map_array(arr, mapper, na_action=na_action, convert=convert)

File ~/anaconda3/envs/SpatialData_prova/lib/python3.12/site-packages/pandas/core/arrays/categorical.py:1555, in Categorical.map(self=['BC cells', 'BC cells', 'Mix BC cells TAMs', 'M...ls',

                          'T cells', 'TAMs'], mapper=array([{'BC cells': '#ff4500', 'CAFs': '#f7e9c7'...', 'Mast cells': '#add8e6'}],

      dtype=object), na_action=None)

   1545     warnings.warn(

   1546         "The default value of 'ignore' for the `na_action` parameter in "

   1547         "pandas.Categorical.map is deprecated and will be "

   (...)

   1551         stacklevel=find_stack_level(),

   1552     )

   1553     na_action = "ignore"

-> 1555 assert callable(mapper) or is_dict_like(mapper)

        mapper = array([{'BC cells': '#ff4500', 'CAFs': '#f7e9c7', 'Mix BC cells TAMs': '#800080', 'Myoepithelial cells': '#daa520', 'Blood ECs': '#efd594', 'Mural cells': '#a77e18', 'T cells': '#001b00', 'Plasma cells': '#ff728a', 'TAMs': '#0000ff', 'DCs': '#000037', 'NK cells': '#003400', 'Mast cells': '#add8e6'}],

      dtype=object)

   1557 new_categories = self.categories.map(mapper)

   1559 has_nans = np.any(self._codes == -1)

AssertionError:

######################################################################################################

I have no clue about this, especially considering that a student of mine previously succeeded in setting custom colors with an older version of napari-spatialdata. Currently I am using the 0.5.4.dev2+gf84b79b version of napari-spatialdata on napari 0.5.4 in python 3.12.3 on an ubuntu machine. I am still a beginner with python, can anyone give me any advice on how to solve this? Thank you in advance!

[LucaMarconato]

Hi, can you please try this code here: https://github.com/scverse/napari-spatialdata/issues/242#issuecomment-2317186193?

[LucaMarconato]

There was a typo in the code I linked, please try this one on the latest napari-spatialdata main:

from spatialdata.datasets import blobs_annotating_element
from napari_spatialdata import Interactive
import spatialdata as sd

sdata = blobs_annotating_element("blobs_labels")

##
labels = sdata["blobs_labels"]

print(sd.get_element_instances(labels))
obs = sdata["table"].obs
obs["strings"] = ["A", "A", "B", "B", "C"]

##
sdata["table"].uns["strings_colors"] = {
    "A": "#FF5733",  # red
    "B": "#3498DB",  # blue
    "C": "#2ECC71",  # green
}

Interactive(sdata)

It should give you this:

If you can't reproduce the bug with the code above please modify the code so that I can reproduce your bug with a small script. Thanks a lot!

[LucaCalderoni]

Hi! I lauched the code above, but it gave me different colors: I did not incur in the bug though. I launched the code above on my windows laptop after I downloaded the latest main branch, so as soon as I get back to my ubuntu machine in my workplace I'll try to do the same and see if the bug occurs again. On the ubuntu machine I downloaded the development branch 0.5.4.dev2+gf84b79b because I needed a specific fix, but now I see that it has been merged into the main branch so I should have no problems.

[LucaMarconato]

Thank you for sending the details. It's a bit strange because yesterday another user tried the code and it seems that things worked: https://scverse.zulipchat.com/#narrow/stream/315824-spatial/topic/napari-spatialdata/near/476109235. I will look into this; meanwhile please send any other details that could help reproduce the bug.

[LucaCalderoni]

It seems these bugs somewhat depend on the operating system. For example, I just tried on my windows laptop to visualize some custom annotations that were not included in the metadata csv file. I manually inserted them in the adata inside the spatialdata object as additional entries of the obs slot. This operation worked perfectly on linux, but on my windows it does not work, for some reason all cells are uniformly colored with a dull gray color, regardless of the annotation I select on the right. Even the annotations taken from the metadata csv are not colored. The same happened some time ago to my student on his windows machine, that's why I suspect these bugs might be different for each OS.

[Czaki]

Please check the napari versions. We have introduced bugfixes in rendering colormaps. So using the old napari version may impact different visible colors

[LucaCalderoni]

I confirm I am using the latest napari (0.5.4) on both windows and ubuntu.

[LucaCalderoni]

Hi! I have another update: I just downloaded the latest napari-spatialdata main branch on the linux machine in my workplace, and then I tried again to set customized colors through a dictionary in the uns slot of the anndata object. This time no error messages popped up, however napari ignored my instructions and colored the annotation with its default palette. Furthermore, I cannot visualize anymore the annotation label of each cell by hovering on it with the mouse cursor.

[josenimo]

Hey @LucaCalderoni, @LucaMarconato

I just wanted to add weight to this. a legend of what colors mean in Napari interactive is really critical for smooth quality control. Hover information in a little corner in my opinion is not enough.

I am not sure of the capabilities of napari in this sense, but from my naive point of view a little legend matching color to categorical in a small pop up or box would be good enough, setting our own { categoricals : color } dictionaries to set colors would be even better.