KeyError: 13 when using 'get_connectors_between'

[cytheria43]

Hi, KeyError xxx occurred when I tried to fetch all connectors between 2 sets of skeleton ids,including all pre- and post- directions.

The running code: conn = pymaid.get_connectors_between([203840],[16],directional=False)

The KeyError:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
~/anaconda3/envs/neuroglancer_ver220/lib/python3.6/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   2897             try:
-> 2898                 return self._engine.get_loc(casted_key)
   2899             except KeyError as err:

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

KeyError: 13

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
<ipython-input-12-3cb4f9d6407c> in <module>
----> 1 conn = pymaid.get_connectors_between([203840],[16],directional=False)
      2 conn

~/anaconda3/envs/neuroglancer_ver220/lib/python3.6/site-packages/pymaid/cache.py in wrapper(*args, **kwargs)
    314         try:
    315             # Execute function
--> 316             res = function(*args, **kwargs)
    317         except BaseException:
    318             # If error was raised, remove new entries from cache

~/anaconda3/envs/neuroglancer_ver220/lib/python3.6/site-packages/pymaid/fetch.py in get_connectors_between(a, b, directional, remote_instance)
   1584     # Get user list and replace IDs with logins
   1585     user_list = get_user_list(remote_instance=remote_instance).set_index('id')
-> 1586     df['creator1'] = [user_list.loc[u, 'login'] for u in df.creator1.values]
   1587     df['creator2'] = [user_list.loc[u, 'login'] for u in df.creator2.values]
   1588 

~/anaconda3/envs/neuroglancer_ver220/lib/python3.6/site-packages/pymaid/fetch.py in <listcomp>(.0)
   1584     # Get user list and replace IDs with logins
   1585     user_list = get_user_list(remote_instance=remote_instance).set_index('id')
-> 1586     df['creator1'] = [user_list.loc[u, 'login'] for u in df.creator1.values]
   1587     df['creator2'] = [user_list.loc[u, 'login'] for u in df.creator2.values]
   1588 

~/anaconda3/envs/neuroglancer_ver220/lib/python3.6/site-packages/pandas/core/indexing.py in __getitem__(self, key)
    871                     # AttributeError for IntervalTree get_value
    872                     pass
--> 873             return self._getitem_tuple(key)
    874         else:
    875             # we by definition only have the 0th axis

~/anaconda3/envs/neuroglancer_ver220/lib/python3.6/site-packages/pandas/core/indexing.py in _getitem_tuple(self, tup)
   1042     def _getitem_tuple(self, tup: Tuple):
   1043         try:
-> 1044             return self._getitem_lowerdim(tup)
   1045         except IndexingError:
   1046             pass

~/anaconda3/envs/neuroglancer_ver220/lib/python3.6/site-packages/pandas/core/indexing.py in _getitem_lowerdim(self, tup)
    784                 # We don't need to check for tuples here because those are
    785                 #  caught by the _is_nested_tuple_indexer check above.
--> 786                 section = self._getitem_axis(key, axis=i)
    787 
    788                 # We should never have a scalar section here, because

~/anaconda3/envs/neuroglancer_ver220/lib/python3.6/site-packages/pandas/core/indexing.py in _getitem_axis(self, key, axis)
   1108         # fall thru to straight lookup
   1109         self._validate_key(key, axis)
-> 1110         return self._get_label(key, axis=axis)
   1111 
   1112     def _get_slice_axis(self, slice_obj: slice, axis: int):

~/anaconda3/envs/neuroglancer_ver220/lib/python3.6/site-packages/pandas/core/indexing.py in _get_label(self, label, axis)
   1057     def _get_label(self, label, axis: int):
   1058         # GH#5667 this will fail if the label is not present in the axis.
-> 1059         return self.obj.xs(label, axis=axis)
   1060 
   1061     def _handle_lowerdim_multi_index_axis0(self, tup: Tuple):

~/anaconda3/envs/neuroglancer_ver220/lib/python3.6/site-packages/pandas/core/generic.py in xs(self, key, axis, level, drop_level)
   3491             loc, new_index = self.index.get_loc_level(key, drop_level=drop_level)
   3492         else:
-> 3493             loc = self.index.get_loc(key)
   3494 
   3495             if isinstance(loc, np.ndarray):

~/anaconda3/envs/neuroglancer_ver220/lib/python3.6/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   2898                 return self._engine.get_loc(casted_key)
   2899             except KeyError as err:
-> 2900                 raise KeyError(key) from err
   2901 
   2902         if tolerance is not None:

KeyError: 13

Used the latest verison of pymaid.

[schlegelp]

Strangely the same code works for me.

The problem appears to be a user missing from your user list. The ID 13 suggests that it's not a user that got added just now and your local list did not reflect that yet.

Could you please see if pymaid.get_user_list() returns something sensible? In my case, I get a list of 352 users and id 13 maps to somebody with first name Addy.

(please don't post any details from the user list in your response)

[schlegelp]

Oh wait. Are you pulling from the public CATMAID instance?

[schlegelp]

I just checked the same query against https://fafb.catmaid.virtualflybrain.org/ but that too works for me.

[cytheria43]

@schlegelp yes! I'm pulling from the public CATMAID service https://fafb.catmaid.virtualflybrain.org/.

And I found it would produce different results when change the input neuron ids.

Get the topological links from CATMAID Graph widget as reference,

when there's no link between neuron 46105 and neuron 16 ,no error happens even though a null dataframe outputs.

But when there actually exists some links between neuron 46105 and 57503 , it would always appear the same error.

[schlegelp]

Again: both of these work for me as expected. Moreover, when I pull from https://fafb.catmaid.virtualflybrain.org/ all users have been anonymised and there is only a single user called "AnonymousUser" with id 1.

Could you please share a full minimal working example. For example, this is what I'm doing:

>>> import pymaid 
>>> rm = pymaid.CatmaidInstance(server='https://fafb.catmaid.virtualflybrain.org/', api_token=None)
>>> conn = pymaid.get_connectors_between([203840],[16],directional=False)  
>>> conn.head()
   connector_id                     connector_loc  node1_id  ...  confidence2  creator2                         node2_loc
0        474311  [421266.62, 128160.97, 194720.0]    474312  ...            5       NaN  [421099.47, 128249.95, 194960.0]
1        573905   [434605.5, 164921.61, 216160.0]    573904  ...            5       NaN   [435825.1, 164631.62, 216120.0]
2       1482374   [434664.8, 165543.78, 215800.0]   1277055  ...            5       NaN  [434900.44, 165491.66, 215880.0]
3       1666500  [436649.84, 145266.72, 210040.0]   1666499  ...            5       NaN  [436473.38, 145003.86, 210040.0]
4       2873212    [437215.0, 146896.0, 221360.0]   1275576  ...            5       NaN  [438710.78, 147283.06, 221440.0]

Could you please run the above code in a fresh Python session?

Also please check the output of pymaid.get_user_list():

>>> pymaid.get_user_list()
   id          login full_name first_name last_name            color
0   1  AnonymousUser                                 [0.0, 1.0, 0.0]

[schlegelp]

Actually, I realised that there was a fix for this kind of situation sitting on the Github repo but hadn't made it to PyPI yet. I've now pushed a new version 2.0.4 to PyPI. Could you update and try again please?

$ pip3 install python-catmaid -U

[schlegelp]

Quick ping whether that issue still exists?