Support for single-band rasters in segmentation

[Saareem]

Currently, single-channel rasters for segmentation are not supported. This is an artificial restriction as there are certainly some segmentation models which use single input band instead of an RGB, RGBA or composite band raster. The current implementation assumes the input raster to be any of the previously mentioned or, rather, having more than 1 standalone bands. I don't need this feature at the moment, so I won't implement it, but it should be noted. I know of some models which use a single-band probability mask as an input to perform post-processing for segmentation model outputs. One use case for single-band rasters as input could be segmentation of DTM, DSM or their derivatives and, additionally, radar images.

From input_channels_mapping_widget.py

        if number_of_image_bands == 1:
            # if there is one band, then there is probably more "bands" hidden in a more complex data type (e.g. RGBA)
            data_type = rlayer.dataProvider().dataType(1)
            if data_type in [Qgis.DataType.Byte, Qgis.DataType.UInt16, Qgis.DataType.Int16,
                             Qgis.DataType.Float32]:
                image_channel = ImageChannelStandaloneBand(
                    band_number=1,
                    name=rlayer.bandName(1))
                image_channels.append(image_channel)
            elif data_type == Qgis.DataType.ARGB32:
                # Alpha channel is at byte number 3, red is byte 2, ... - reversed order
                band_names = [
                    'Alpha (band 4)',
                    'Red (band 1)',
                    'Green (band 2)',
                    'Blue (band 3)',
                ]
                for i in [1, 2, 3, 0]:  # We want order of model inputs as 'RGB' first and then 'A'
                    image_channel = ImageChannelCompositeByte(
                        byte_number=3 - i,  # bytes are in reversed order
                        name=band_names[i])
                    image_channels.append(image_channel)
            else:
                raise Exception("Invalid input layer data type!")

I stumbled upon this on accident as I quit QGIS when I had a single-band raster set as the input. When QGIS loaded, it also checked the input channel mapping using this single-band raster previously written in the config and produced an error: "Invalid input layer type!".

[przemyslaw-aszkowski]

Thanks, good observation, we will be definitively looking into it soon to add support for 1 channel images

[adrianloy]

I have 4 input channels (RGB + height data). Would be great to allow arbitrary amount of input bands!

[bartoszptak]

Hi @adrianloy and @Saareem!

Could you share with me your models and provide example data for these features' development process? If they are confidential, please send them via mail (available on my GitHub profile).

[Saareem]

I have 4 input channels (RGB + height data). Would be great to allow arbitrary amount of input bands!

@adrianloy I have a similar model and amount of channels above 3 is certainly not a problem as per say, even with the original Segmentor class and its MapProcessorSegmentation processor. I could easily run a UNet model having 5 input channels (see. https://doi.org/10.3390/rs15174278 for the UNet model). However, the normalization, division by 255, is not good for height data. What I did is that I inherited the Segmentor class and override the preprocessing function. But making n+1 implementations of preprocessing method for each type of normalization does not make any sense.

As selection of normalization is based on the model and same normalization is expected at inference time as during the training, it would certainly be beneficial to have the possibility to run models without any normalization, assuming that the normalization is performed before input to the model, as a separate preprocessing step. Another option would be to change the architecture of the plugin so that different normalization routine (min/max, division by 255 for byte rasters, z-score, you name it) could be selected on input channel level but it complicates things significantly. In that way, it would be easier to run models which used some particular type of normalization during the training.

As an example, the UNet I mentioned

Standardizes RGB channels by

• subtracting dataset level mean value
• dividing by dataset level standard deviation value

However, for height information, such as DSM and DEM, this is not performed.

For DSM and DEM

• channels are standardized to standard deviation of 1 by dividing with the dataset level standard deviation value (i.e. the model should not learn in any way how absolute height is related to features)
• The DSM and DEM channels for a tile are concatenated and a mean is calculated
• The mean calculated in the previous step is subtracted from height channels (DSM and DEM)

For now, I'm performing the global, raster level operations, in a separate preprocessing step using BandMathX tool in OrfeoToolBox in QGIS and then input the prenormalized raster to Deepness. In the overridden preprocessing method I'm only calculating the tile-level means and making the subtraction of mean from tile for DSM and DEM.

[Saareem]

@adrianloy @bartoszptak
I could find this paper which discusses using DeepLabV3+ for DTM segmentation. It also discusses the issue of normalization for channels containing height. The solution is a bit different from what we have used but the point is that dividing by 255 is a bad idea. There's an implementation available here https://github.com/Bashirkazimi/semantic_segmentation and I could probably provide some DTM data, if you can not find any yourself. DeepLabV3+, afaik, is rather easy to modify for arbitrary number of input channels, including one.

[adrianloy]

What you say is correct in general, however in my specific case I am not running into the normalization issue, as I use the height data for some post-processing in raw format. So my custom computational graph will take care of it, but I need to be able to pass the band as input to it.

[Saareem]

What you say is correct in general, however in my specific case I am not running into the normalization issue, as I use the height data for some post-processing in raw format. So my custom computational graph will take care of it, but I need to be able to pass the band as input to it.

@adrianloy Are you using composite or multi-band raster? I'm using 5-channel multi-band raster without issues so it should be possible. It seems in composite rasters there is an assumption it will contain at most 4 channels.