search

jamesdolezal / slideflow

Deep learning library for digital pathology, with both Tensorflow and PyTorch support.
https://slideflow.dev
GNU General Public License v3.0
230 stars 38 forks source link

multimodal data modeling in MIL #267

Open Meijian opened 1 year ago

Meijian commented 1 year ago

Feature

Not sure if it's already in the plan but it'd be great if we can incorporate multimodal data in MIL just like the regular train() dose.

Pitch

Incorporating multimodal data in image processing is increasingly important as clinical studies routinely collect multiple data types. It should help to improve the model's performance.

Alternatives

Additional context

jamesdolezal commented 1 year ago

Thank you, this is a great idea. I will add it to the feature list. It will get included in 2.1 or 2.2.

Meijian commented 1 year ago

Perfect! Thanks for considering it.

jamesdolezal commented 8 months ago

So we're not quite ready for multimodal integration with clinical variables, but in version 2.2 we did just release a multimodal MIL strategy that allows you to train models at multiple magnification levels simultaneously (see here). This has laid the groundwork for future expansion into other types of mode integration (such as clinical variables, gene expression etc). Multimodal integration with non-pathology modalities is our next priority - stay tuned!

Meijian commented 8 months ago

It's nice to see the progress. Thank you for your efforts, James!