Triplet Loss and Multi-Modality

[miguel-arrf]

Hi!

I'm dealing with a challenge where I have images of multiple modalities (for instance different modalities of MRI scans). I want to learn a representation between those modalities. For instance, trying to understand if a representation that embodies the differences between these modalities will increase the performance of a given downstream task.

What I want is something like (for example for SimCLR), setting x_1 to be always of one modality and x_2 of another modality.

• What's the best way of doing this? Should I do it by adding a new method (https://solo-learn.readthedocs.io/en/latest/tutorials/add_new_method.html)?
• Also, is the Triplet Loss available within solo-learn?

Thank you for the help! :)

[vturrisi]

Hey,

Adding a new method would be the easiest. Not sure our tutorial is up-to-date, but it shouldn't be hard to implement it. You can start by copy-pasting another method and doing the needed modifications.

For your data, you will have issues since we only use a single dataloader from a single data source. You will need to modify that, but we don't have tutorials for that. You should probably modify the main file, load multiple datasets and then wrap them.

Triplet loss is also not implemented.