why is class 0 treated differently ?

[joihn]

Hello, thanks for this repo,

cond_scale = 1.0 if (labels == 0).all().item() else cond_scale https://github.com/marcoaversa/diffinfinite/blob/4d2309606e5ce6190828b96f436cfa962ec5d1ec/dm.py#L433

Is seems you treat class 0 differently from the others, why ? I guess class 0 represent your background ? Or maybe it's a "unkown" class ?

[marcoaversa]

Hi, the conditioning approach is treated class-wise. For each step the model returns a noise output for each class in the segmentation mask. The 0 represents the 'unknown' class an it is treated differently. This is because in the classifier-free method, the model returns 2 noise outputs, one for the class and one for the null label. We use the classifier-free as semi-supervised method where the null label corresponds to the unknown class. For this reason it requires only to return one noise output. Hope this helps.

[joihn]

thanks for the quick reply, So if I understand correctly, the 5 class model are the following index_value:class_name

0: unknown
1: Carcinoma
2: Necrosis
3: Tumor Stroma
4: Others

(I deduced the list from main.ipynb )

What is the difference between unknown and others ? Why do you choose to split them in 2 class ?

[marcoaversa]

Yes, these are the classes considered. 'Unknown' are parts of the images that the pathologist didn't annotate, 'Others' are a mix of scarcely annotated classes compared to Carcinoma, Necrosis and Tumor stroma. We trained also a model with 10 classes considering other classes with a low ratio in the dataset. We decided to choose this configuration to improve the biological plausibility of the dataset. 'Others' contains some classes that are spatially distributed around the high represented classes. Even if the classes are mixed into 'Others', once the mask is generated, the diffusion model generates the most appropriate sub-class depending on the spatial mask distribution. In that way we are able to generate more diverse and faithful synthetic images.

[joihn]

Okay thanks for the helpfull reply !

[joihn]

It seems the special treatment of class 0 is only present during inference, and not during training. Do you think this could perturbate the network during its training? He will try to guess how class 0 looks like, but will never manage to accurately guess it (since there is too much variety), therefore he will constantly receive contradictory gradient about this class ?

[marcoaversa]

Class label is present during training with probability=0.5 we feed images with annotations and images without annotations. When we don't have annotations, the model is trained with the NULL label so it is equal to pass a mask full of zeros.

[joihn]

okay thanks :)