ValueError: Cannot setitem on a Categorical with a new category, set the categories first

[pgibb96]

@schlegelp I wanted to move our conversation to GitHub in order to make reading the traceback a bit easier. Currently running fafbseg 1.1.0 and python-catmaid 2.0.2.

(cloudvolume) petergibb@macbook-pro ~ % ipython            
Python 3.8.2 (default, May  6 2020, 02:49:43) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.17.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import fafbseg
WARNING: Could not load OpenGL library.

In [2]: import pymaid

In [3]: manual = pymaid.CatmaidInstance('https://neuropil.janelia.org/tracing/fafb/v14',
   ...:                                 api_token='[token]',
   ...:                                 http_user='[user]',
   ...:                                 http_password='[password]',
   ...:                                 caching=False,
   ...:                                 max_threads=20)
INFO  : Global CATMAID instance set. Caching is OFF. (pymaid)

In [4]: x = pymaid.get_neuron(12526613, remote_instance=manual)

In [5]: resp = fafbseg.move.merge_into_catmaid(x, target_instance=manual, tag='wilson_lab', min_node_overlap=4, min_overlap_size=1, merge_limit=.75, min_upload_size=3, min_upload_nodes=1)
A2_like_IH_4: 7 overlapping fragments found                                                                                                                                                                                       
Large (>10 nodes) overlapping fragments:                                                                                                                                                                                          
               name        id  n_nodes  n_connectors  overlap_score
 A2_Presynaptic_146  16989666    47118           154              6
 A2_Presynaptic_151  12526614    20162           454           4534
          A2_like_4  12526387    12131           626           4905
  neuron 5693177 JW  12526606     5716           237             12
 A2_Presynaptic_153  12526609     5702            20              6
       A2_like_IH_4  12526613     5175            17           5066
 neuron 11538884 ET  11538883     1708             0            964

            Please check these large fragments for overlap and deselect
            neurons that you DO NOT want to have merged by clicking on
            their names in the legend.
            Hit ENTER when you are ready to proceed or CTRL-C to cancel.

        Please check the fragments that potentially overlap with the input neuron (white).
        Deselect those that should NOT be merged using the arrows keys.
        Hit ENTER when you are ready to proceed or CTRL-C to abort

[?]                name        id  n_nodes  n_connectors  sampler_count  overlap_score: 
   ◯  A2_Presynaptic_146  16989666  47118  154  0     6
   ◯  A2_Presynaptic_151  12526614  20162  454  0  4534
 ❯ ◉           A2_like_4  12526387  12131  626  0  4905
   ◯   neuron 5693177 JW  12526606   5716  237  0    12
   ◯  A2_Presynaptic_153  12526609   5702   20  0     6
   ◯        A2_like_IH_4  12526613   5175   17  0  5066
   ◯  neuron 11538884 ET  11538883   1708    0  0   964

        Above fragments and your input neuron will be merged into a single neuron.
        All annotations will be preserved but only the neuron used as merge target
        will keep its name and skeleton ID.
        Please select the neuron you would like to use as merge target!
              name        id  n_nodes  n_connectors  sampler_count  overlap_score
[?] Choose merge target:  A2_like_4  12526387  12131  626  0  4905
 ❯  A2_like_4  12526387  12131  626  0  4905

Processing neuron "A2_like_IH_4" (12526613) [0/1]
Generating union of all fragments... Done.
Extracting new nodes to upload... ---------------------------------------------------------------------------
ValueError                                
Traceback (most recent call last)
<ipython-input-5-5debd1bf34b3> in <module>
----> 1 resp = fafbseg.move.merge_into_catmaid(x, target_instance=manual, tag='wilson_lab', min_node_overlap=4, min_overlap_size=1, merge_limit=.75, min_upload_size=3, min_upload_nodes=1)

~/opt/anaconda3/envs/cloudvolume/lib/python3.8/site-packages/fafbseg/utils.py in wrapper(*args, **kwargs)
     35         try:
     36             # Execute function
---> 37             res = function(*args, **kwargs)
     38         except BaseException:
     39             raise

~/opt/anaconda3/envs/cloudvolume/lib/python3.8/site-packages/fafbseg/move/merge.py in merge_into_catmaid(x, target_instance, tag, min_node_overlap, min_overlap_size, merge_limit, min_upload_size, min_upload_nodes, update_radii, import_tags, label_joins, sid_from_nodes, mesh)
    316 
    317         # And then break into continuous fragments for upload
--> 318         frags = navis.break_fragments(only_new)
    319         print('Done.', flush=True)
    320 

~/opt/anaconda3/envs/cloudvolume/lib/python3.8/site-packages/navis/morpho/manipulation.py in break_fragments(x)
   1497         comp = sorted(comp, key=len, reverse=True)
   1498 
-> 1499         return core.NeuronList([graph.subset_neuron(x,
   1500                                                     list(ss),
   1501                                                     inplace=False) for ss in comp])

~/opt/anaconda3/envs/cloudvolume/lib/python3.8/site-packages/navis/morpho/manipulation.py in <listcomp>(.0)
   1497         comp = sorted(comp, key=len, reverse=True)
   1498 
-> 1499         return core.NeuronList([graph.subset_neuron(x,
   1500                                                     list(ss),
   1501                                                     inplace=False) for ss in comp])

~/opt/anaconda3/envs/cloudvolume/lib/python3.8/site-packages/navis/utils/misc.py in wrapper(*args, **kwargs)
     57         try:
     58             # Execute function
---> 59             res = function(*args, **kwargs)
     60         except BaseException:
     61             raise

~/opt/anaconda3/envs/cloudvolume/lib/python3.8/site-packages/navis/graph/graph_utils.py in subset_neuron(x, subset, inplace, keep_disc_cn, prevent_fragments)
   1590         # We have to run this in a separate function so that the lock is applied
   1591         # to the copy
-> 1592         subset_neuron(x,
   1593                       subset=subset,
   1594                       inplace=True,

~/opt/anaconda3/envs/cloudvolume/lib/python3.8/site-packages/navis/utils/misc.py in wrapper(*args, **kwargs)
     57         try:
     58             # Execute function
---> 59             res = function(*args, **kwargs)
     60         except BaseException:
     61             raise

~/opt/anaconda3/envs/cloudvolume/lib/python3.8/site-packages/navis/graph/graph_utils.py in subset_neuron(x, subset, inplace, keep_disc_cn, prevent_fragments)
   1629     # We won't produce new slabs but roots and leaves might change
   1630     x.nodes.loc[x.nodes.parent_id < 0, 'type'] = 'root'
-> 1631     x.nodes.loc[~x.nodes.node_id.isin(x.nodes.parent_id.values), 'type'] = 'end'
   1632 
   1633     # Filter connectors

~/opt/anaconda3/envs/cloudvolume/lib/python3.8/site-packages/pandas/core/indexing.py in __setitem__(self, key, value)
    668 
    669         iloc = self if self.name == "iloc" else self.obj.iloc
--> 670         iloc._setitem_with_indexer(indexer, value)
    671 
    672     def _validate_key(self, key, axis: int):

~/opt/anaconda3/envs/cloudvolume/lib/python3.8/site-packages/pandas/core/indexing.py in _setitem_with_indexer(self, indexer, value)
   1763                 # scalar value
   1764                 for loc in ilocs:
-> 1765                     isetter(loc, value)
   1766 
   1767         else:

~/opt/anaconda3/envs/cloudvolume/lib/python3.8/site-packages/pandas/core/indexing.py in isetter(loc, v)
   1689                     # set the item, possibly having a dtype change
   1690                     ser = ser.copy()
-> 1691                     ser._mgr = ser._mgr.setitem(indexer=pi, value=v)
   1692                     ser._maybe_update_cacher(clear=True)
   1693 

~/opt/anaconda3/envs/cloudvolume/lib/python3.8/site-packages/pandas/core/internals/managers.py in setitem(self, indexer, value)
    532 
    533     def setitem(self, indexer, value) -> "BlockManager":
--> 534         return self.apply("setitem", indexer=indexer, value=value)
    535 
    536     def putmask(

~/opt/anaconda3/envs/cloudvolume/lib/python3.8/site-packages/pandas/core/internals/managers.py in apply(self, f, align_keys, **kwargs)
    404                 applied = b.apply(f, **kwargs)
    405             else:
--> 406                 applied = getattr(b, f)(**kwargs)
    407             result_blocks = _extend_blocks(applied, result_blocks)
    408 

~/opt/anaconda3/envs/cloudvolume/lib/python3.8/site-packages/pandas/core/internals/blocks.py in setitem(self, indexer, value)
   1680 
   1681         check_setitem_lengths(indexer, value, self.values)
-> 1682         self.values[indexer] = value
   1683         return self
   1684 

~/opt/anaconda3/envs/cloudvolume/lib/python3.8/site-packages/pandas/core/arrays/categorical.py in __setitem__(self, key, value)
   2008         # something to np.nan
   2009         if len(to_add) and not isna(to_add).all():
-> 2010             raise ValueError(
   2011                 "Cannot setitem on a Categorical with a new "
   2012                 "category, set the categories first"

ValueError: Cannot setitem on a Categorical with a new category, set the categories first

In [6]: 

[schlegelp]

It's still not quite clear to me why this neuron in particular caused this error but I tracked it down and seem to have fixed it (it's in navis, not pymaid). For now, you will need to update navis from Github as I haven't pushed the fix to PyPI yet:

pip3 install git+git://github.com/schlegelp/navis@master

I'm still suspicious as to how this situation occurred in the first place - let me know if you run into more errors with this neuron and I will need to dig a bit deeper.

[pgibb96]

Thank you! I will let you know if I encounter any other issues.

[pgibb96]

I no longer have the category issue, but now I have the same issue my lab mate has (TypeError: Object of type int32 is not JSON serializable).

Generating union of all fragments... Done.
Extracting new nodes to upload... Done.
---------------------------------------------------------------------------                            
TypeError                                 Traceback (most recent call last)
<ipython-input-5-1d10d9a890ae> in <module>
----> 1 resp = fafbseg.move.merge_into_catmaid(x, target_instance=manual, tag="wilson_lab", min_node_overlap=4, min_overlap_size=1, merge_limit=.75, min_upload_size=3, min_upload_nodes=1)

~/opt/anaconda3/envs/cloudvolume/lib/python3.8/site-packages/fafbseg/utils.py in wrapper(*args, **kwargs)
     35         try:
     36             # Execute function
---> 37             res = function(*args, **kwargs)
     38         except BaseException:
     39             raise

~/opt/anaconda3/envs/cloudvolume/lib/python3.8/site-packages/fafbseg/move/merge.py in merge_into_catmaid(x, target_instance, tag, min_node_overlap, min_overlap_size, merge_limit, min_upload_size, min_upload_nodes, update_radii, import_tags, label_joins, sid_from_nodes, mesh)
    446         if update_radii and 'radius' in n.nodes.columns and np.all(n.nodes.radius):
    447             print('Updating radii of existing nodes... ', end='', flush=True)
--> 448             resp = update_node_radii(source=n, target=ol,
    449                                      remote_instance=target_instance,
    450                                      limit=merge_limit,

~/opt/anaconda3/envs/cloudvolume/lib/python3.8/site-packages/fafbseg/move/merge.py in update_node_radii(source, target, remote_instance, limit, skip_existing)
    529     new_radii = source.nodes.iloc[nn_ix[nn_dist <= limit]].radius.values
    530 
--> 531     return pymaid.update_radii(dict(zip(tn_ids, new_radii)),
    532                                remote_instance=remote_instance)

~/opt/anaconda3/envs/cloudvolume/lib/python3.8/site-packages/pymaid/cache.py in wrapper(*args, **kwargs)
    175         rm.caching = False
    176         # Execute function
--> 177         res = function(*args, **kwargs)
    178         # Set caching to old value
    179         rm.caching = old_value

~/opt/anaconda3/envs/cloudvolume/lib/python3.8/site-packages/pymaid/upload.py in update_radii(radii, chunk_size, remote_instance)
   1533         # We have to explicitly convert the state in a json string because passing
   1534         # it to requests as "post" will fuck this up otherwise
-> 1535         update_post['state'] = json.dumps(update_post['state'])
   1536 
   1537         this_resp = remote_instance.fetch(update_radii_url, post=update_post,

~/opt/anaconda3/envs/cloudvolume/lib/python3.8/json/__init__.py in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys, **kw)
    229         cls is None and indent is None and separators is None and
    230         default is None and not sort_keys and not kw):
--> 231         return _default_encoder.encode(obj)
    232     if cls is None:
    233         cls = JSONEncoder

~/opt/anaconda3/envs/cloudvolume/lib/python3.8/json/encoder.py in encode(self, o)
    197         # exceptions aren't as detailed.  The list call should be roughly
    198         # equivalent to the PySequence_Fast that ''.join() would do.
--> 199         chunks = self.iterencode(o, _one_shot=True)
    200         if not isinstance(chunks, (list, tuple)):
    201             chunks = list(chunks)

~/opt/anaconda3/envs/cloudvolume/lib/python3.8/json/encoder.py in iterencode(self, o, _one_shot)
    255                 self.key_separator, self.item_separator, self.sort_keys,
    256                 self.skipkeys, _one_shot)
--> 257         return _iterencode(o, 0)
    258 
    259 def _make_iterencode(markers, _default, _encoder, _indent, _floatstr,

~/opt/anaconda3/envs/cloudvolume/lib/python3.8/json/encoder.py in default(self, o)
    177 
    178         """
--> 179         raise TypeError(f'Object of type {o.__class__.__name__} '
    180                         f'is not JSON serializable')
    181 

TypeError: Object of type int32 is not JSON serializable

[schlegelp]

The last error should be fixed with f41d1a379a971bab62be971a317b374d099235ed. Try updating to pymaid 2.0.3 and see if the error is gone:

pip3 install python-catmaid -U

[schlegelp]

Any luck on this @pgibb96 ?

[pgibb96]

Oh yes, sorry! That fixed the error.