Usability issues with the tracking annotator

[constantinpape]

The tracking annotator is fairly complex to use and it would be good to simplify it a bit and prevent easy to make user errors.

• [ ] Don't allow setting a track_id for a layer if that track has already ended (due to division). Currently this let's the user set a point, but nothing happens.
• [ ] Change track state via box annotation (shape layer) #23

(will add more points here)

cc @Sagnik700

[Sagnik700]

While tracking with micro-sam on 200x200 dimensional MiaPaCa-flat dataset threw one error in between the whole workflow which most probably happened due to wrong combination of prompts during segmenting a cell in division mode. Here is the entire error stack trace:

InvalidIndexError                         Traceback (most recent call last)
File ~\mambaforge\envs\micro-sam-env\lib\site-packages\pandas\core\series.py:1159, in Series.__setitem__(self=Series([], Name: label, dtype: object), key=[9], value='positive')
   1158 try:
-> 1159     self._set_with_engine(key, value)
        key = [9]
        self = Series([], Name: label, dtype: object)
        value = 'positive'
   1160 except KeyError:
   1161     # We have a scalar (or for MultiIndex or object-dtype, scalar-like)
   1162     #  key that is not present in self.index.

File ~\mambaforge\envs\micro-sam-env\lib\site-packages\pandas\core\series.py:1222, in Series._set_with_engine(self=Series([], Name: label, dtype: object), key=[9], value='positive')
   1221 def _set_with_engine(self, key, value) -> None:
-> 1222     loc = self.index.get_loc(key)
        key = [9]
        self = Series([], Name: label, dtype: object)
   1224     # this is equivalent to self._values[key] = value

File ~\mambaforge\envs\micro-sam-env\lib\site-packages\pandas\core\indexes\range.py:350, in RangeIndex.get_loc(self=RangeIndex(start=0, stop=0, step=1), key=[9])
    349     raise KeyError(key)
--> 350 self._check_indexing_error(key)
        key = [9]
        self = RangeIndex(start=0, stop=0, step=1)
    351 raise KeyError(key)

File ~\mambaforge\envs\micro-sam-env\lib\site-packages\pandas\core\indexes\base.py:5736, in Index._check_indexing_error(self=RangeIndex(start=0, stop=0, step=1), key=[9])
   5733 if not is_scalar(key):
   5734     # if key is not a scalar, directly raise an error (the code below
   5735     # would convert to numpy arrays and raise later any way) - GH29926
-> 5736     raise InvalidIndexError(key)
        key = [9]

InvalidIndexError: [9]

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
File ~\mambaforge\envs\micro-sam-env\lib\site-packages\psygnal\_signal.py:972, in SignalInstance._run_emit_loop(self=<SignalInstance 'changed' on ComboBox(value='division', annotation=None, name='')>, args=('division',))
    971 try:
--> 972     caller.cb(args)
        caller = <psygnal._weak_callback._StrongFunction object at 0x000001FD4E474280>
        args = ('division',)
    973 except Exception as e:

File ~\mambaforge\envs\micro-sam-env\lib\site-packages\psygnal\_weak_callback.py:268, in _StrongFunction.cb(self=<psygnal._weak_callback._StrongFunction object>, args=('division',))
    267     args = args[: self._max_args]
--> 268 self._f(*self._args, *args, **self._kwargs)
        args = ('division',)
        self._f = <function create_tracking_menu.<locals>.state_changed at 0x000001FD4E4B2320>
        self = <psygnal._weak_callback._StrongFunction object at 0x000001FD4E474280>
        self._args = ()
        self._kwargs = {}

File e:\stud.ip docs - m.sc. applied computer science\thesis\other project repos\micro-sam\micro_sam\sam_annotator\annotator_tracking.py:293, in create_tracking_menu.<locals>.state_changed(new_state='division')
    292 current_properties["state"] = np.array([new_state])
--> 293 points_layer.current_properties = current_properties
        current_properties = {'label': <class 'numpy.ndarray'> (1,) object, 'state': <class 'numpy.ndarray'> (1,) <U8, 'track_id': <class 'numpy.ndarray'> (1,) object}
        points_layer = <Points layer 'prompts' at 0x1fd4dcdfe80>
    294 points_layer.refresh_colors()

File ~\mambaforge\envs\micro-sam-env\lib\site-packages\napari\layers\points\points.py:698, in Points.current_properties(self=<Points layer 'prompts'>, current_properties={'label': <class 'numpy.ndarray'> (1,) object, 'state': <class 'numpy.ndarray'> (1,) <U8, 'track_id': <class 'numpy.ndarray'> (1,) object})
    697     update_indices = list(self.selected_data)
--> 698 self._feature_table.set_currents(
        self = <Points layer 'prompts' at 0x1fd4dcdfe80>
        self._feature_table = <napari.layers.utils.layer_utils._FeatureTable object at 0x000001FD4DE80970>
        update_indices = [9]
        current_properties = {'label': <class 'numpy.ndarray'> (1,) object, 'state': <class 'numpy.ndarray'> (1,) <U8, 'track_id': <class 'numpy.ndarray'> (1,) object}
    699     current_properties, update_indices=update_indices
    700 )
    701 current_properties = self.current_properties

File ~\mambaforge\envs\micro-sam-env\lib\site-packages\napari\layers\utils\layer_utils.py:866, in _FeatureTable.set_currents(self=<napari.layers.utils.layer_utils._FeatureTable object>, currents={'label': <class 'numpy.ndarray'> (1,) object, 'state': <class 'numpy.ndarray'> (1,) <U8, 'track_id': <class 'numpy.ndarray'> (1,) object}, update_indices=[9])
    865 for k in self._defaults:
--> 866     self._values[k][update_indices] = self._defaults[k][0]
        self._defaults =       label     state track_id
0  positive  division        1
        self = <napari.layers.utils.layer_utils._FeatureTable object at 0x000001FD4DE80970>
        update_indices = [9]
        k = 'label'
        self._values = Empty DataFrame
Columns: [label, state, track_id]
Index: []

File ~\mambaforge\envs\micro-sam-env\lib\site-packages\pandas\core\series.py:1216, in Series.__setitem__(self=Series([], Name: label, dtype: object), key=[9], value='positive')
   1215     else:
-> 1216         self._set_with(key, value)
        key = [9]
        self = Series([], Name: label, dtype: object)
        value = 'positive'
   1218 if cacher_needs_updating:

File ~\mambaforge\envs\micro-sam-env\lib\site-packages\pandas\core\series.py:1238, in Series._set_with(self=Series([], Name: label, dtype: object), key=[9], value='positive')
   1236 if not self.index._should_fallback_to_positional:
   1237     # Regardless of the key type, we're treating it as labels
-> 1238     self._set_labels(key, value)
        key = [9]
        value = 'positive'
        self = Series([], Name: label, dtype: object)
   1240 else:
   1241     # Note: key_type == "boolean" should not occur because that
   1242     #  should be caught by the is_bool_indexer check in __setitem__

File ~\mambaforge\envs\micro-sam-env\lib\site-packages\pandas\core\series.py:1255, in Series._set_labels(self=Series([], Name: label, dtype: object), key=<class 'numpy.ndarray'> (1,) int32, value='positive')
   1254 if mask.any():
-> 1255     raise KeyError(f"{key[mask]} not in index")
        key = <class 'numpy.ndarray'> (1,) int32
        mask = <class 'numpy.ndarray'> (1,) bool
   1256 self._set_values(indexer, value)

KeyError: '[9] not in index'

The above exception was the direct cause of the following exception:

EmitLoopError                             Traceback (most recent call last)
File ~\mambaforge\envs\micro-sam-env\lib\site-packages\magicgui\widgets\bases\_value_widget.py:65, in ValueWidget._on_value_change(self=ComboBox(value='division', annotation=None, name=''), value='division')
     63 if value is self.null_value and not self._nullable:
     64     return
---> 65 self.changed.emit(value)
        value = 'division'
        self.changed = <SignalInstance 'changed' on ComboBox(value='division', annotation=None, name='')>
        self = ComboBox(value='division', annotation=None, name='')

File ~\mambaforge\envs\micro-sam-env\lib\site-packages\psygnal\_signal.py:927, in SignalInstance.emit(self=<SignalInstance 'changed' on ComboBox(value='division', annotation=None, name='')>, check_nargs=False, check_types=False, asynchronous=False, *args=('division',))
    924     sd.start()
    925     return sd
--> 927 self._run_emit_loop(args)
        self = <SignalInstance 'changed' on ComboBox(value='division', annotation=None, name='')>
        args = ('division',)
    928 return None

File ~\mambaforge\envs\micro-sam-env\lib\site-packages\psygnal\_signal.py:974, in SignalInstance._run_emit_loop(self=<SignalInstance 'changed' on ComboBox(value='division', annotation=None, name='')>, args=('division',))
    972                 caller.cb(args)
    973             except Exception as e:
--> 974                 raise EmitLoopError(
        caller = <psygnal._weak_callback._StrongFunction object at 0x000001FD4E474280>
        args = ('division',)
    975                     slot_repr=repr(caller), args=args, exc=e
    976                 ) from e
    978 return None

EmitLoopError: calling <psygnal._weak_callback._StrongFunction object at 0x000001FD4E474280> with args=('division',) caused KeyError: '[9] not in index'.

[Sagnik700]

Testing micro-sam with 0.1.2.post1 version and with MiaPaCa200x200 dataset, daughter cell segmentation of a frame was visible in the previous frame where the parent cell was annotated as division in track_state. Screenshot 1(Frame 22) shows the parent cell segmentation in brown and the daughter cell segmentation in blue. The blue segmentation should not be visible in Screenshot 1 since it is appearing in Screenshot 2(Frame 23) onwards.

Steps to replicate the issue:

1. Segment the exact cell which is highlighted as division as track_state in frame 22
2. Segment the daughter cell as highlighted as track 2 as track_state in frame 23 and click on Track Object
3. Come back to frame 22 to notice the issue

[Sagnik700]

For some live cell datasets where frame rate is low(significant displacements and change of shape of the cells in subsequent frames), resolution is low and population density is high, it benefits highly if a cell is segmented in as many frames possible using box prompts before clicking on Track Object. If it is sparsely segmented, then after clicking Track Object, the track localization is not accurate and the corrections required will usually take more time.

Frames when a cell is segmented every 10th frame along its lineage and Track Object is clicked:

Frames when a cell is segmented every 4th frame along its lineage and Track Object is clicked:

[Sagnik700]

Clear annotations doesnt reset track_state and track_id leading to errors thrown when there was a division segmentation before clearing the annotations, creating multiple track_id values in the dropdown which were meaningless after clearing the annotations.

[Sagnik700]

Selecting point prompts of different sizes does not update the point size slider in the top left corner. In the screenshot below, point prompts of different sizes are registered, but after selecting the bigger prompt using the Point Selector the point size slider still shows the size of the previously selected smaller point