Which transforms work on heatmaps (or depth)

[ApoorvaSuresh]

I get NaN losses when I use RGBD data for training. I split the RGBD as RGB (augment_images is applied on this) and D (augment_heatmap is appleid on this) and then concatenated.

I use fliplr and crop&pad. was wondering if these are not compatible with heatmap augmentation??

[aleju]

All augmentations are compatible with heatmaps, but only some augmentations will actually modify them (the others will return them without any changes). More specifically, only augmentations that affect the spatial locations of pixels have an effect. CropAndPad and Fliplr do that. The output of these augmenters should never be NaN, unless the input already contained NaNs. Anything else would be a bug. In case of CropAndPad it might be worth mentioning that the pad operation within that augmenter will always pad the heatmap array with 0 and not any other value. That could cause issues, depending on the intended value range of the heatmap.

[vassiliskrikonis]

@aleju can I ask why padding doesn't work on heatmaps and/or segmentation maps?

[aleju]

Padding does work, there just isn't a nice interface for choosing your own non-default value to pad non-image data with.

The pad values in the __init__ methods of augmenter classes are intended for image data and hence cannot simply be used for other datatypes (e.g. padding an uint8 image by 128 makes sense, but padding a heatmap with value range [0.0, 1.0] by 128 doesn't make sense). There are currently no separate pad parameters exposed for heatmaps and segmentation maps, mostly because it would start to bloat up the interface quite a lot (one parameter for the padding value, one for the padding mode, possibly one for the resize interpolation order -- and that times two for segmentation maps and heatmaps and possibly more in the future for other datatypes). It would also increase further the number of required unittests.

However, most (or maybe all) of the augmenters that change pixel locations have internal attributes that define the pad values and can be changed, e.g. aug = Affine(...); affine._cval_segmentation_maps = 2.

[lone17]

@aleju Sorry for digging this up. Would there be any potential problem if I np.dstack the image and the heatmap\segmetation map and then pass into the augmenter to achieve symmetric padding on the heatmap\segmetation.

And one more of topic question: what is the different between 'symmetric' and 'reflect' padding mode ?