Whole slide inference with hooknet (Pytorch) - new links / methods?

[VolodymyrChapman]

Hi there, I've trained a hooknet model for a binary classification problem (positive or negative for presence of a pathological feature) using an internal dataset and I'd like to now conduct inference on whole slide images. Before hooknet, I would patch tissue areas with OpenSlide().read_region() and infer but I obviously need to preserve the pyramidal file structure for hooknet. Ideally, I'd get a binary mask at spacing of 8.1 (32x magnification) as output for downstream uses.

My config file:

    default:
        seed: 123
        yaml_source: data/inference_config/data.yml

        labels: 
            positive: 1

        batch_shape:
            batch_size: 2
            shape: [[600,600,3],[600,600,3], [600,600,3]]
            spacing: [1.01, 4.05, 8.102]
            y_shape: [3, 374, 374]

        sample_callbacks:
            - "*object": wholeslidedata.samplers.callbacks.CropSampleCallback
              output_shape: [374, 374]

I have searched for existing code for doing this with hooknet but dependencies appear to be outdated / files are missing:

1. The dependencies are outdated in hooknet/inference/apply_torch.py:

https://github.com/DIAGNijmegen/pathology-hooknet/blob/178865901e26ebca2a301dc27f40f82320a1e567/hooknet/inference/apply_torch.py#L17C6-L17C29 wholeslidedata no longer has a source directory etc.

2. Config files are no longer available from this issue: https://github.com/DIAGNijmegen/pathology-whole-slide-data/issues/23

This hints at the existence of updated / improved methods for inference over an entire whole slide image.

Could I please ask, what would be the advised way to infer over whole slide images with a trained hooknet model?

Thank you and my best wishes,

Volodymyr

[martvanrijthoven]

Dear Volodymyr Chapman,

My apologies for the inconvenience and confusion. To be honest, the torch model is still in experimental phase, and I have not put enough effort in updating the repo to make all the parts consistent with the PyTorch model. For an inference pipeline i need to write a new full example and I am not sure when I will have time for this. In the mean time you can look at the patch iterator. That should enable you to easy extract the data from the whole-slide images to be used for the inference. Stitching it together can be done with the MaskWriter. However i dont have a clear example for this. You can also have a look at the inference pipeline for the tensorflow model. The torch pipeline will be almost the same, except for the specific PyTorch model methods for inference.

Offtopic, i looked at you configuration file and noticed a couple of issues. The PyTorch model is developed to use two steps between magnifications: e.g., [0.25, 1.0, 4.0] or [0.5, 2.0, 8.0] or [1.0, 4.0, 16.0]. However you have [1.01, 4.05, 8.102] which does not make the model crash, but there will be pixel misalignment issues when the features are concatenated. Furthermore also the patch size you used causes pixel misalignment. You can use this script to check if an input size would result in a correct pixel alignment when the branches are concatenated. I ran it from 284 to 1244 input sizes and these are valid input sizes

284
316
348
380
412
444
476
508
540
572
604
636
668
700
732
764
796
828
860
892
924
956
988
1020
1052
1084
1116
1148
1180
1212
1244

I hope this helps a bit, again my apologies for the trouble. Please let me know if you any further questions and I hope i can work on a PyTorch inference example soon.

Best wishes, Mart

[VolodymyrChapman]

Dear Mart,

Thank you so much for your timely and detailed response! I apologise - I didn't mean the issue report to come across as a criticism. I understand how quickly your work moves and I thought I was missing something in the direction it had moved in :-) Thank you also for reviewing my config file. Superb - I will work on a solution using your guidance and link here in case anyone else has use for it. Thank you again and my best wishes, Volodymyr

[martvanrijthoven]

Dear Volodymyr,

Thanks for the understanding. No worries, It is great that you pointed out this issue and I will try to find some time to work on the pytorch inference pipeline. I will keep this issue open and will let you know when there is an update.

Best wishes, Mart

[mdeleeuw1]

Dear Volodymyr Chapman,

My apologies for the inconvenience and confusion. To be honest, the torch model is still in experimental phase, and I have not put enough effort in updating the repo to make all the parts consistent with the PyTorch model. For an inference pipeline i need to write a new full example and I am not sure when I will have time for this. In the mean time you can look at the patch iterator. That should enable you to easy extract the data from the whole-slide images to be used for the inference. Stitching it together can be done with the MaskWriter. However i dont have a clear example for this. You can also have a look at the inference pipeline for the tensorflow model. The torch pipeline will be almost the same, except for the specific PyTorch model methods for inference.

Offtopic, i looked at you configuration file and noticed a couple of issues. The PyTorch model is developed to use two steps between magnifications: e.g., [0.25, 1.0, 4.0] or [0.5, 2.0, 8.0] or [1.0, 4.0, 16.0]. However you have [1.01, 4.05, 8.102] which does not make the model crash, but there will be pixel misalignment issues when the features are concatenated. Furthermore also the patch size you used causes pixel misalignment. You can use this script to check if an input size would result in a correct pixel alignment when the branches are concatenated. I ran it from 284 to 1244 input sizes and these are valid input sizes

284
316
348
380
412
444
476
508
540
572
604
636
668
700
732
764
796
828
860
892
924
956
988
1020
1052
1084
1116
1148
1180
1212
1244

I hope this helps a bit, again my apologies for the trouble. Please let me know if you any further questions and I hope i can work on a PyTorch inference example soon.

Best wishes, Mart

Dear Mart,

Thank you for the extensive clarification on this topic. I have a question regarding the input resolutions of the PyTorch model you mentioned in this comment. Why do the input channels differ a factor 4 in resolution settings? When looking at the torchmodel.py script (specifically the hook_to_index and hook_from_index variables), it seems as there is only one layer of difference between the hooking layer of each input channel, which would mean the resolution differs by a factor of 2 instead of 4. Any clarification on why this is not the case would be highly appreciated! :)

Cheers,

Mike

[martvanrijthoven]

Dear Mike,

The indexes are a bit confusion indeed, but the same value for hook_to_index and hook_from_index are already a downfactor of 2. So a value hook_to_index and hook_from_index with a value of 1 differ already a factor of 2 with regard to the downsampling. Hence a hook_to_index=1 and hook_from_index=2 will differ a factor 4. Please check out this relative code: https://github.com/DIAGNijmegen/pathology-hooknet/blob/178865901e26ebca2a301dc27f40f82320a1e567/hooknet/models/torchmodel.py#L157

If it is still confusion, please let me know and I can elaborate more.

Best wishes, Mart

[mdeleeuw1]

Dear Mart,

I see, that makes sense. Thank you for clarifying!

Cheers, Mike