feature request: recomputing spot detections after users' manual edits

[jiyuuchc]

Background

Trackmate provides a rich set of UIs for users to modify the results from automatic algorithms. For example, it has keyboard shortcuts to either delete or add new spots. However, if the user manually added a spot by specifying a location, it typically is not annotated with as rich set of feature annotations. It would be nice if Trackmate could compute the same features for all the spots including manually added ones.

Feature request

For the detectors that support such a feature, Trackmate should call them to perform an update of spot detection. Trackmate should provide the additional annotations, e.g., the additional spot locations that the user had given. The detector should then fill in the missing information (radius, ROI) as needed.

Potential issues

The timing of the update needs to be discussed. The detector might be slow, so synchronous (blocking) update might be bad user experience. On the other hand, asynchronous update (esp if without UI indicator) can be counter-intuitive and confusing. A potential simple solution might be ask user to click a button to specifically request an update.

Relavance

I am the co-author of trackmate-lacss, a detector using the deep-learning model lacss as its back-end. We are interested in implementing such an "update" feature in our detector. In fact, the core functionality already exists in the back-end, but is not exposed to the user.

In addition, recently there are some research on a type of "segment-anything" models that are specifically designed to do object detection with a user-provided hint, e.g. a location. These models also benefit from this feature, if someone is to implement them in Trackmate.

[tinevez]

Hello @jiyuuchc I am not sure I get it right.

There is an action in the last panel called 'recompute features' that does what it says. Could it be what you are looking for?

[jiyuuchc]

Attention: This is an external email. Use caution responding, opening attachments or clicking on links.

User-supplied spots typically only has X-Y values (or XYZ in 3d), which is different from the spots generated from our cell detector, which has the full segmentation, from which TrackMate computes a lot of other features (area, circularity etc).

When we worked on analyses of BYTETrack tracking, we had to do a bit of heuristics on user-added spots, because we cannot get correct IOU values on these.

For us, a better solution is to ask the detector to re-segment based on the user-supplied hint ( the locations ). This would make the downstream task more consistent. However to do that in Trackmate, we need upstream support to add this capability.

Does that make sense?

---

From: Jean-Yves Tinevez @.***> Sent: Thursday, November 7, 2024 3:32 PM To: trackmate-sc/TrackMate Cc: Yu,Ji; Mention Subject: Re: [trackmate-sc/TrackMate] feature request: recomputing spot detections after users' manual edits (Issue #310)

Attention: This is an external email. Use caution responding, opening attachments or clicking on links.

Hello @jiyuuchchttps://urldefense.com/v3/__https://github.com/jiyuuchc__;!!Cn_UX_p3!lvM5xA7xaa3NUfoi8Ene9qog6vyrX_PWWAz2Pha7BLJ0NXzyYakQqX1a-osgOcCwo9JaJOZLaAHcLyJnWAg3Gw$ I am not sure I get it right.

There is an action in the last panel called 'recompute features' that does what it says. Could it be what you are looking for?

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/trackmate-sc/TrackMate/issues/310*issuecomment-2463158037__;Iw!!Cn_UX_p3!lvM5xA7xaa3NUfoi8Ene9qog6vyrX_PWWAz2Pha7BLJ0NXzyYakQqX1a-osgOcCwo9JaJOZLaAHcLyJRjQy_Sw$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AAKRPNS7AXZXLL4ZCW2HQSDZ7PE6ZAVCNFSM6AAAAABRL3AFLWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINRTGE2TQMBTG4__;!!Cn_UX_p3!lvM5xA7xaa3NUfoi8Ene9qog6vyrX_PWWAz2Pha7BLJ0NXzyYakQqX1a-osgOcCwo9JaJOZLaAHcLyKsX5p-JA$. You are receiving this because you were mentioned.Message ID: @.***>

[tinevez]

Yes thank you! It seems like a gigantic task however: We need to make it possible for a user to pick any existing algorithm to operate locally in neighborhoods around a set of discrete locations. So a new framework, UI, extension mechanism to do so in TrackMate.

In the forthcoming version 8, it will be possible for users to edit manually segmentation results. Would that work instead?

[jiyuuchc]

Obviously I am bias here, but I don't think it is that difficult to implement.

From the detector-writer point of view, all we need is that the TrackMate make a call at some point -- maybe when user clicked an update button:

if detectorFactory.canUpdate() {
    updatedSpots = detectorFactory.getDetector(interval, frame).updateSpots(currentSpots);
}

This is a bit exaggeration -- there are also globalDetector to worry about, and there are some post-processing to do -- but it is not that far from the truth.

That said, I understand if you think this feature doesn't bring enough value -- after all, currently I don't know any other detector except ours that is actually able to implement this function. And we can always perform the update outside TrackMate using the export file.