graceful failures in l2_skeleton

[jefferis]

Via R but I guess it will be the same in python. 720575940452114071 is a very small body (2 supervoxels). This is annoying when you are skeletonising 200 neurons in a single call and one fails ...

> fp$flywire$l2_skeleton("720575940452114071")
Error in py_call_impl(callable, dots$args, dots$keywords) : 
  AxisError: axis 1 is out of bounds for array of dimension 1

Detailed traceback:
  File "/Users/jefferis/Library/r-miniconda/envs/r-reticulate/lib/python3.8/site-packages/fafbseg/flywire/l2.py", line 237, in l2_skeleton
    l2_eg = np.unique(np.sort(l2_eg, axis=1), axis=0)
  File "<__array_function__ internals>", line 180, in sort
  File "/Users/jefferis/Library/r-miniconda/envs/r-reticulate/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 1004, in sort
    a.sort(axis=axis, kind=kind, order=order)

[schlegelp]

Oh right - I hadn't considered root IDs that have only a single L2 chunk (and hence no L2 graph). I will make other L2 functions (flywire.l2_info() and flywire.l2_dotprops) deal with this but for the skeletons this is a bit more tricky:

Two potential solutions that come to mind

1. Return a single-node skeleton or
2. Add a omit_failures parameter to l2_skeleton() which would then simply skip failed attempts (including invalid IDs, etc).

I suspect 1 would potentially cause trouble further down the line. How do you feel about 2?

[jefferis]

Thanks! On the nat side, nlapply has an OmitFailures argument that is basically there whenever looping over neurons, skeletons etc. It has three modes (T,F,NA). T omits with a warning; F includes a null object with a warning. NA (the default) errors out. So that's what I'm used to ...

btw this is another slightly different failure:

> fp$flywire$l2_skeleton('720575940622943240')
Error in py_call_impl(callable, dots$args, dots$keywords) : 
  TypeError: loop of ufunc does not support argument 0 of type float which has no callable sqrt method

Detailed traceback:
  File "/Users/jefferis/Library/r-miniconda/envs/r-reticulate/lib/python3.8/site-packages/fafbseg/flywire/l2.py", line 284, in l2_skeleton
    tn = navis.TreeNeuron(swc, id=root_id, units='1 nm', **kwargs)
  File "/Users/jefferis/Library/r-miniconda/envs/r-reticulate/lib/python3.8/site-packages/navis/core/skeleton.py", line 154, in __init__
    self.nodes = x
  File "/Users/jefferis/Library/r-miniconda/envs/r-reticulate/lib/python3.8/site-packages/navis/core/skeleton.py", line 369, in nodes
    self._set_nodes(v)
  File "/Users/jefferis/Library/r-miniconda/envs/r-reticulate/lib/python3.8/site-packages/navis/core/skeleton.py", line 382, in _set_nodes
    graph.classify_nodes(self)
  File "/Users/jefferis/Library/r-miniconda/envs/r-reticulate/lib/python3.8/site-packages/navis/utils/misc.py", line 251, in wrapper
    return function(*args, **kwargs)
  File "/U

[schlegelp]

b1405c11c127855735f5f2343c1262dd87d9ef84 adds a omit_failures parameter with that behaviour to l2_skeleton and l2_dotprops:

• omit_failures=None (default) will raise an exception
• omit_failures=True will log a warning and skip the offending neuron (might result in an empty NeuronList)
• omit_failures=False will log a warning and produce empty TreeNeuron/Dotprops

Important to note that I did not use a blanket try/except block to implement this - so it will only catch "expected" errors such as single-L2-chunk neurons.

This commit also fixes the second error you reported which turned out to be some strange interactions that lead to changes in the data type of x/y/z coordinates.

These commits currently only live on Github but I can push a new version to PyPI if that's relevant for you?

[jefferis]

Oh cool! Thanks.

[schlegelp]

Version 1.6.0 which includes the above changes is now on PyPI

[jefferis]

Thanks a lot. Testing. I now get a new error on neurons that previously passed through quite happily. Perhaps most neurons have some points that can't be refined? In that case the error message should perhaps be modified along the lines of

Unable to refine all points: missing (some) L2 info for root ID 720575940625800018.

Also shouldn't drop_missing=True just drop the points and keep going?

  ValueError: Unable to refine: no L2 info for root ID 720575940625800018 available. Set `drop_missing=False` to use unrefined positions.

Detailed traceback:
  File "/Users/jefferis/Library/r-miniconda/envs/r-reticulate/lib/python3.8/site-packages/fafbseg/flywire/l2.py", line 240, in l2_skeleton
    n = l2_skeleton(id, refine=refine, drop_missing=drop_missing,
  File "/Users/jefferis/Library/r-miniconda/envs/r-reticulate/lib/python3.8/site-packages/fafbseg/flywire/l2.py", line 335, in l2_skeleton
    raise ValueError(msg)

[schlegelp]

🤦 It should only complain when all points would be dropped but I made a logic error. It's fixed now on Github and I pushed a new version (1.6.1) to PyPI.

[schlegelp]

Actually version 1.6.1 made it better but still not correct: I had to push another fix which is now in 1.6.2 on both Github and PyPI.

[jefferis]

Do you need to pass on omit_failures here:

n = l2_skeleton(id, refine=refine, drop_missing=drop_missing,
  progress=progress, dataset=dataset, **kwargs)

I get an error when passing in multiple ids but not for the same problem neuron by itself.

> sk=fp$flywire$l2_skeleton(ids[165:167], omit_failures = F, drop_missing = F)
                                                           Error in py_call_impl(callable, dots$args, dots$keywords) : 
  ValueError: Unable to create L2 skeleton: root ID 720575940452114071 consists of only a single L2 chunk.

Detailed traceback:
  File "/Users/jefferis/Library/r-miniconda/envs/r-reticulate/lib/python3.8/site-packages/fafbseg/flywire/l2.py", line 240, in l2_skeleton
    n = l2_skeleton(id, refine=refine, drop_missing=drop_missing,
  File "/Users/jefferis/Library/r-miniconda/envs/r-reticulate/lib/python3.8/site-packages/fafbseg/flywire/l2.py", line 263, in l2_skeleton
    raise ValueError(msg)
> sk=fp$flywire$l2_skeleton(ids[166], omit_failures = F, drop_missing = F)
WARNING : Unable to create L2 skeleton: root ID 720575940452114071 consists of only a single L2 chunk. (navis)

[schlegelp]

Oh man - how do I keep missing these things... Fixed now on Github and PyPI.

[schlegelp]

Closing as fixed.