Consult about the resolution

ghost commented 3 years ago

Hi, thanks for your work and opensource code, I am confused by the resolution issues. The inputs of Hooknet is a pair image with different resolutions, do I need to prepare two kinds of images (with respect to two resolutions of one image context)? Or I just need to enlarge the context branch (more field) to target branch (more resolution), but in this way the resolutions will not change, how to deal with the the resolution problem, I am looking forward to your reply, thanks.

martvanrijthoven commented 3 years ago

Hi Longqiwf,

Thank you for your interest in HookNet!

The model indeed expects a pair of images (patches) with different resolutions (e.g, 0.5μm and 8.0μm). Both patches should have the same dimensions (e.g., both 284x284x3).

I hope this resolves your confusion? Please let me know if something is still unclear.

martvanrijthoven commented 3 years ago

To add:

The two patches should indeed belong to the same image context and are expected to be concentric (i.e., both patches should have the same center coordinate)

ghost commented 3 years ago

hi martvanrijthoven, thank you for your quickly response, for instance, I prepared a 2842843 image (belongs to context branch with more field), and I don't have the target branch, if I enlarge the concentric zones from the context image to 2842843, the resolutions will be decreased, which is opposite to your paper (in target branch the resolution is higher). So should I prepare another dataset for the target branch from the outset?

martvanrijthoven commented 3 years ago

I am not completely sure what you mean. But the model expects patches prepared like this: MFMR_DCIS

What do you mean with 'I don't have the target branch'? Is it possible to show me an example of your data, maybe I will understand your issue better?

ghost commented 3 years ago

Hi martvanrijthoven, Yeah, I know the form of the inputs, the problem is about resolutions issue, maybe I have a misunderstanding of resolutions, I am very happy if you could point me the correct understand. Such as your illustrated picture, it seems you magnify the center zone of the context branch to be a target branch, actually, you use another picture which is prepared for the target branch, rather than generating from the context branch, because if you magnify the center zone of the context branch to the same dimension of original context branch, its resolution will be decreased (looks more serrated and rough). But in your paper, the initial input of target branch has a higher resolution compared with the context branch. So I mean if it is necessary to parpare two datasets for the same image context, one is for context branch with lower resolution, another is specially for the target branch with higher resolution.

martvanrijthoven commented 3 years ago

Hi Longqiwf,

Whole slide images are stored in pyramidal data structures, containing multiple resolutions:

An-illustration-of-how-digital-slides-are-stored-in-a-pyramid-structure-In-this-example

Note that these whole slide images are > 50000x50000 in size at the highest resolutions.

If you use ASAP (https://github.com/computationalpathologygroup/ASAP/releases) or Openslide (https://openslide.org/), you can easily extract patches, with a specific center coordinate, shape, and resolution. You can prepare your dataset by extracting the patches from the resolutions that you would like.

Just for illustration, the following shows how this sort of works: crop-down

Would this answer your question?

ghost commented 3 years ago

Hi martvanrijthoven, Thanks for your patiently explained, it really help me a lot :)

martvanrijthoven commented 3 years ago

Hi Longqiwf,

Sure you are welcome, let me know if you have any questions left, and feel free to open another issue.

yuling-luo commented 1 year ago

Hi @martvanrijthoven Great work on the paper. I have a question about the patches as well. This might be a silly question. So you extract patches from the image, but only select the patches based on their class labels? "Patches were sampled with a particular tissue type, i.e., class label, at the center location of the sampled patch".

martvanrijthoven commented 1 year ago

Dear @yuling-luo, thank you for your interest in HookNet. Absolutely not a silly question, reading the sentence back now, I can imagine it is confusing. In the wholeslidedata documentation I have a bit more elaborated explanation how the sampling is done. I put the text here below for convenience. Please let me know if something is still unclear.

Data Preparation

To prepare the data for sampling, the following steps are taken:

Associations
- WSIs and their corresponding annotation files (WSAs) are matched and stored as "associations".
- Each association consists of a matched WSI file and WSA file.
- Each WSI file has an image backend.
- Each WSA file has an annotation parser.
Parsing annotations: All WSAs are parsed by their annotation parsers to extract a list of annotations (polygons, points).
Labels: All available labels are gathered from all annotations.
Annotation mapping: All annotations are mapped to their respective labels.

Sampling Strategy

The sampling strategy consists of three sampler components:

label_sampler: samples a label from the available labels.
annotation_sampler: samples an annotation from the list of annotations that corresponds to the label sampled in step 1.
point_sampler: samples a point within the annotation sampled in step 2.

Sampling Data

The data sampling consists of two sampler components:

patch_sampler: selects a patch with a center point based on the point sampled in step 3 of the sampling strategy.
patch_label_sampler: creates a label (classification, detection, or segmentation) based on the point sampled in step 3 of the sampling strategy.

By repeating the above steps, the BatchIterator generates patches and corresponding labels for use in machine learning models.

"""

yuling-luo commented 1 year ago

Thanks @martvanrijthoven. So instead of using all the divided patches from the image like below image shown. You are only selecting the patches within the annotation. And based on sampled points within the annotation, you can extract region from different resolutions and feed them into model. Is my understanding correct?

martvanrijthoven commented 1 year ago

@yuling-luo Yes you are correct, because only then a ground-truth mask can be generated to be used to train a segmentation network like HookNet.

Please note that for inference when applying the HookNet model, the "dividing into patches" method is used to predict on the full WSI. This "dividing into patches" can be done with the PatchIterator Furthermore the "dividing into patches" could also be used for multiple instance learning, but I have not worked on that topic myself.

I hope this clarifies it, please let me know if you have any other questions.

yuling-luo commented 1 year ago

@yuling-luo Yes you are correct, because only then a ground-truth mask can be generated to be used to train a segmentation network like HookNet.

Please note that for inference when applying the HookNet model, the "dividing into patches" method is used to predict on the full WSI. This "dividing into patches" can be done with the PatchIterator Furthermore the "dividing into patches" could also be used for multiple instance learning, but I have not worked on that topic myself.

I hope this clarifies it, please let me know if you have any other questions.

Thank you for your response. It's really helpful!

DIAGNijmegen / pathology-hooknet