How to extract nnUNetv2 features

[U-ma-s]

Hi, I want to solve the problem addressed by this issue #608 in nnunetv2 as well. Specifically, I want to get the feature maps extracted by Encoder and cluster the training dataset in some way.

Any tip on which part of the code I should be looking at?

Thanks in advance,

[GregorKoehler]

Hi @U-ma-s , depending on the nnUNet plans you're using, you get different network architectures (the class name of the U-Net architecture should be stored as network_class_name in the plans). Those network architectures are defined in the dynamic-network-architectures repository, e.g. here: https://github.com/MIC-DKFZ/dynamic-network-architectures/blob/main/dynamic_network_architectures/architectures/unet.py As you can when looking at the forward passes, in nnUNetv2 (and hence the dynamic-network-architectures repo as this was introduced with nnUNetv2), the encoder is typically defined separately from the decoder. Therefore, you can just use your trainer's self.network.encoder(inputs) to get features from the encoder. Keep in mind that, as can be seen here (https://github.com/MIC-DKFZ/dynamic-network-architectures/blob/main/dynamic_network_architectures/building_blocks/residual_encoders.py#L135-L145), you might have to select the correct entry of the list of stage representations which are usually returned if return_skips is True. I guess you're probably after the last element.

[U-ma-s]

Thank you for your helpful reply. I will try it!

[GregorKoehler]

Great, please let me know if you were successful!

[U-ma-s]

Hi @GregorKoehler , I'm having a hard time creating a program because I'm new to this. There are two main things I don't know.

1. Where should I add the program?
2. How to identify where the input image is located in the feature map

Any advice or sample code would be very helpful.