Similarity sort

[mzur]

• [x] Implement models for training proposal and annotation candidate feature vectors
• [x] Implement script to generate feature vectors
• [x] Implement API endpoints to return sorted proposal/candidate IDs
• [x] Implement UI to select reference patch and show sorted list
• [x] Update Python script to handle grayscale etc. images, too (the usual)
• [x] Append proposal/candidate IDs of which no feature vector exists to the list of sorted IDs
• [x] Show error message if no feature vector exists at all
• [x] Create issue to fix ellipse bounding box computation for feature vectors (https://github.com/biigle/maia/issues/129)

References https://github.com/biigle/maia/issues/27 References https://github.com/biigle/core/pull/670

[mzur]

@dlangenk @lehecht could you please give me feedback on the UI of the similarity sort feature? Currently it's like this: I can select a patch with a "pin" button as a "reference patch". This will load the new sorting of patches where the most similar patches to the selected reference are shown first. The reference patch itself is shown separately in the image grid. I chose to show it on the left (instead of on the top) because this wastes less space on widescreen monitors. Here is a video:

Screencast from 11.10.2023 10:32:53.webm

This doesn't feel really elegant, yet. I also thought to show the reference patch as part of the regular patch grid but "pinned" as the first patch on every page when you scroll down. But this felt wrong, too, and could lead to all sorts of issues when patch ranges should be selected (because the reference will suddenly be in the range).

What do you think? Any better ideas?

[mzur]

I've now managed to avoid the issues with patch ranges mentioned above and tested the other idea of the reference patch pinned to the first position in the grid:

Screencast from 11.10.2023 16:29:03.webm

This feels a little nicer IMO. @dlangenk @lehecht Any comments?

[dlangenk]

I've now managed to avoid the issues with patch ranges mentioned above and tested the other idea of the reference patch pinned to the first position in the grid:

Screencast.from.11.10.2023.16.29.03.webm This feels a little nicer IMO. @dlangenk @lehecht Any comments?

I like it this way. It wastes much less space and is clear enough i would say. It could be visually a little bit detached but it is also ok like this. You could for example set a bigger "colored" margin or something, but I'm not so sure about that.

[lehecht]

I like both versions. In your first version I understood more quickly what the reference is and what the sorting result was. But on the other hand it consumes a lot of space, like you already mentioned. The last version also looks good but I needed a little more time to recognize the pinned reference. It would be better to highlight the reference a bit more, as @dlangenk has already suggested.

[mzur]

Thanks for the feedback! I'll stick with the second example then. I'll increase the border size a bit but other than that I don't know how to highlight the patch even more. Maybe I've shown a bad example because the highlight color is similar to the patch color. It may be more salient for different kinds of images. But it's the highlight color that is used throughout all of BIIGLE so :shrug:

[dlangenk]

Thanks for the feedback! I'll stick with the second example then. I'll increase the border size a bit but other than that I don't know how to highlight the patch even more. Maybe I've shown a bad example because the highlight color is similar to the patch color. It may be more salient for different kinds of images. But it's the highlight color that is used throughout all of BIIGLE so 🤷

https://jackrugile.com/jrumble/ 🤣🤪 (Just kidding, please don't use that)

[dlangenk]

Sry I'm late to the party, but I have a question. If you pin a thumbnail like in your screencast, then you sort according to that thumbnail - What happens if you unpin it. Will the original order be restored? I actually would have liked to remain in that order. Since you probably want e.g. to pin another sea cucumber, which is similar but still a bit distinct to find further specimen and not that similar to the first one. An example sea cucumber id 5 is pinned and the next three ids after sorting are 212,213,214. But if you would have pinned 212 you would get 213,444,445 instead. If we would go back to the original order it's harder to find and pin 212. Hope you get what I mean. Otherwise I'm back on monday.

[mzur]

It's too late for this PR but we can always improve things later. I've just deployed all the stuff and prepared one MAIA job as an example:

Screencast from 13.10.2023 12:45:25.webm

It looks like it's working quite well :wink: And fast, too (the job has ~20k annotation candidates). You can try it here if you want: https://biigle.de/maia/665

Next, I'll compute the feature vectors for all MAIA jobs. Then it's on to support for all annotations in Largo.

[mzur]

I thought about your comment again abd maybe it's not an issue at all. You can directly pin another patch from the sorted view. You dont have to go "sorted"->"unsorted"->"sorted". But still if the pinned patch is unpinned, the original order is restored.