Instance[ParcelID] and Semantic[Target] questions

[GustavoMourao]

Hi @VSainteuf , how is it going?

I have been studying the code related in terms of training with different datasets. And one question that I have faced based on the data that you have used is about the labels created into semantic $TARGET{}.npy$ and $ParcelIDs{}.npy$. I have attached a figure of patch 10006:

As you can see, in instance annotation before mapping with semantic channel 0 ($ParcelIDs{}.npy$), we have a set IDs that isn't related with semantic annotations (in terms of background class, specially). Besides of that, in channel 2 of $TARGET{}.npy$ we have a set of regions, related with background class, that isn't labelled as in channels 0 and 1.

So, I would like to better understand what was the methodology applied in labelling instances ($ParcelIDs{}.npy$) and semantic label at channel 2 ($TARGET{}.npy$).

Thanks a lot!

[jhonjam]

Hello @VSainteuf , I am doing an analysis of your data set, especially $TARGET_{}.npy$. I performed tests to identify each channel, based on the scripts train_panoptic.py (from src.dataset import PASTIS_Dataset, data_loader) and metrics.py, there is the following function.

def add(self, predictions, target): ( target_heatmap, true_instances, zones, size, sem_obj, sem_pix, ) = target.split((1, 1, 1, 2, 1, 1), dim=-1)

based on pastis-benchmark/demo.ipynb sem_pix is to equals pixel_semantic_annotation

My questions are: 1 why only channel 0 of pixel_semantic_annotation (dataset.py) is used. Example if self.class_mapping is not None: pixel_semantic_annotation = self.class_mapping( pixel_semantic_annotation[0] ) else: pixel_semantic_annotation = pixel_semantic_annotation[0] 2 what happened to the other channel the pixel_semantic_annotation ( pixel_semantic_annotation = np.load( os.path.join( self.folder, "ANNOTATIONS", "TARGET_{}.npy".format(id_patch) ) ) )?
3 this channels are not necessary for train and test?

Could you please provide information on the three $TARGET_{}.npy$ channels, what does each of these refer to?

Thanks ...

[VSainteuf]

Hi @GustavoMourao and @jhonjam, So here are some elements that will hopefully help you:

the 'TARGET_xx.npy' arrays are composed of these 3 channels:

0. The semantic label of each pixel according the 20 class nomenclature of PASTIS.
1. The unique instance id of each pixel, ranging from 1 to N, with N the number of parcels in the patch.
2. The fine-grained semantic label of each pixel, according to the nomenclature used by the French subsidy allocating authority that has around 400 classes I think.

So for semantic seg, only the first channel is useful. For panoptic segmentation the second channel is also used as it provides the instance masks of parcels. For example, this second channel is used to compute de ground truth centerness heat maps that are contained in /INSTANCE_ANNOTATIONS. The third channel of 'TARGET_xx.npy' is not used as of now.

the 'ParcelIDs_xx.npy' array contains the instance ids of each pixel according to the numbering of the French Land Parcel Identification System (LPIS). I kept this array to keep the connection with the vector data of the LPIS but these instance IDs are not used in the experiments. These instance IDs cover the whole French territory so they take very large integer values (in the range of millions) with relatively small difference in value between neighbouring parcels. So on that point @GustavoMourao, plotting them with a default colormap will often make different values look the same ( as in your leftmost figure).

[VSainteuf]

And @GustavoMourao regarding you first question, I think it's a matter of colormaps. I plotted the same patch with a more divergeant colormap and everything seems fine:

There is only one case where there might be differences between ParcelIDs, and instance ids (second channel of TARGET). As detailed in the documentation we considered small or thin parcels as background, such parcels do have an id in ParcelID but are annotated with 0 in TARGET (both for semantic and instance). But the case you raised was not an occurence of this it seems.

[jhonjam]

Hello @VSainteuf , thanks for the answer.

At this moment, I am doing an analysis of the predictions by CNN based on the script paps.py returns the following dictionary

predictions = dict( center_mask=center_mask, saliency=saliency, heatmap=heatmap, semantic=sem, size=size, confidence=confidence, centerness=centerness, instance_masks=instance_masks, instance_boxes=instance_boxes, pano_instance=panoptic_instance, pano_semantic=panoptic_semantic, )

According to the documentation, pano_semantic returns a tensor with size BxKxHxW. I made the graphs of these tensors (k = 0,...,19) and each created image should contain the pixel predictions of each class, eg, k=1 corresponds to the "meadow" class.

By comparing the ground truth labels of a test image with the predicted pixels, the model predicts a large number of pixels corresponding to the "Sorghum", "Soft winter wheat" classes. Pixels that are not in the ground truth and thus increasing false positives. The same happens for other test images.

This can be seen in the figure below.

Based on the above, can it be said that the predictions of k= 2, 3, 7, 18 (Soft winter wheat, Corn, Sunflower, Sorghum respectively) are considered false positives? and that's the reason why the RQ metric is low?

In the official article in Figure 12: Qualitative Panoptic Segmentation Results, an image with the semantic segmentation predictions is shown.

1) How was this image constructed? 2) To create this image, the tensor pano_semantic (BxKxHxW) was joined?

could you please provide information about this. Thanks a lot!

[VSainteuf]

Hi @jhonjam, yes pixels that belonging to the background and predicted as a parcel of a given class count as false positives in the RQ metric. Just one precision: the RQ is computed at object level and not at pixel level.

Regarding Figure 12, the predictions shown in column (d) are those of a U-TAE semantic segmentation model. To plot the panoptic predictions on column (c) I used the findcountours method of OpenCV on the predicted instance masks contained in pano_instance, and used the predicted class for color coding. The repository does not contain this plotting method yet, but I will include it once I find some time to clean it!

[VSainteuf]

You can now find the plotting method for the panoptic predictions in the visualisations.ipynb notebook in the utae-paps repository.