search

facebookresearch / dino

PyTorch code for Vision Transformers training with the Self-Supervised learning method DINO
Apache License 2.0
6.04k stars 884 forks source link

hello what I would need to do to apply it to 3d medical imaging setting #210

Closed jakubMitura14 closed 1 year ago

jakubMitura14 commented 1 year ago

Hello, I would like to use your algorithm for the 3d setting (magnetic resonance imaging of the prostate gland). I have only image-level labels, and your algorithm seems very interesting. What would I need to do to adapt it for a 3-dimensional setting?

abebe9849 commented 1 year ago

you can use 2.5d model. pretrain:stack 3 adjacent slices→training DINO for 2d model fine tuniung: training MIL(e.g:https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/362607 ) with DINO-pretrained-backbone.

jakubMitura14 commented 1 year ago

Thanks @abebe9849 ! Hovewer in this case image level labels would not work as I do not know on which slice is the object of intrest - so setting label as for example prostate cancer present will be false for most slices and true only for those where it truly is present

abebe9849 commented 1 year ago

@jakubMitura14 RSNA2022 also did not provide per-slice annotations. Using MIL eliminates the need for slice-wise annotation. It seems that self-supervised learning can also be used for aggregation layers. (e.g: https://github.com/mahmoodlab/HIPT)

jakubMitura14 commented 1 year ago

thanks!

abebe9849 commented 1 year ago

https://github.com/yeerwen/DeSD seems a 3D version of DINO. @jakubMitura14

I hope this would help you.

jakubMitura14 commented 1 year ago

Perfect -Thank You !