RandomAffine throws on `C=0` tensor.

timleslie commented 1 year ago

Describe the bug

kornia.augmentation.RandomAffine doesn't handle the degenerate case of a C=0 tensor.

In [19]: import torch

In [20]: import kornia.augmentation

In [21]: my_fcn = kornia.augmentation.RandomAffine((-15., 20.))

In [22]: input = torch.rand(2, 0, 224, 224)

In [23]: out, transform = my_fcn(input)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[23], line 1
----> 1 out, transform = my_fcn(input)

File ~/src/rosa/.venv/lib/python3.8/site-packages/torch/nn/modules/module.py:1194, in Module._call_impl(self, *input, **kwargs)
   1190 # If we don't have any hooks, we want to skip the rest of the logic in
   1191 # this function, and just call forward.
   1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194     return forward_call(*input, **kwargs)
   1195 # Do not call functions when jit is used
   1196 full_backward_hooks, non_full_backward_hooks = [], []

File ~/src/rosa/.venv/lib/python3.8/site-packages/kornia/augmentation/base.py:212, in _BasicAugmentationBase.forward(self, input, params, **kwargs)
    208     params['batch_prob'] = tensor([True] * batch_shape[0])
    210 params, flags = self._process_kwargs_to_params_and_flags(params, self.flags, **kwargs)
--> 212 output = self.apply_func(in_tensor, params, flags)
    213 return self.transform_output_tensor(output, input_shape) if self.keepdim else output

File ~/src/rosa/.venv/lib/python3.8/site-packages/kornia/augmentation/_2d/base.py:126, in RigidAffineAugmentationBase2D.apply_func(self, in_tensor, params, flags)
    123     flags = self.flags
    125 trans_matrix = self.generate_transformation_matrix(in_tensor, params, flags)
--> 126 output = self.transform_inputs(in_tensor, params, flags, trans_matrix)
    127 self._transform_matrix = trans_matrix
    129 return output

File ~/src/rosa/.venv/lib/python3.8/site-packages/kornia/augmentation/base.py:263, in _AugmentationBase.transform_inputs(self, input, params, flags, transform, **kwargs)
    261 in_tensor = self.transform_tensor(input)
    262 if to_apply.all():
--> 263     output = self.apply_transform(in_tensor, params, flags, transform=transform)
    264 elif not to_apply.any():
    265     output = self.apply_non_transform(in_tensor, params, flags, transform=transform)

File ~/src/rosa/.venv/lib/python3.8/site-packages/kornia/augmentation/_2d/geometric/affine.py:113, in RandomAffine.apply_transform(self, input, params, flags, transform)
    110 if not isinstance(transform, Tensor):
    111     raise TypeError(f'Expected the `transform` be a Tensor. Got {type(transform)}.')
--> 113 return warp_affine(
    114     input,
    115     transform[:, :2, :],
    116     (height, width),
    117     flags["resample"].name.lower(),
    118     align_corners=flags["align_corners"],
    119     padding_mode=flags["padding_mode"].name.lower(),
    120 )

File ~/src/rosa/.venv/lib/python3.8/site-packages/kornia/geometry/transform/imgwarp.py:206, in warp_affine(src, M, dsize, mode, padding_mode, align_corners, fill_value)
    203 # src_norm_trans_dst_norm = torch.inverse(dst_norm_trans_src_norm)
    204 src_norm_trans_dst_norm = _torch_inverse_cast(dst_norm_trans_src_norm)
--> 206 grid = F.affine_grid(src_norm_trans_dst_norm[:, :2, :], [B, C, dsize[0], dsize[1]], align_corners=align_corners)
    208 if padding_mode == "fill":
    209     return _fill_and_warp(src, grid, align_corners=align_corners, mode=mode, fill_value=fill_value)

File ~/src/rosa/.venv/lib/python3.8/site-packages/torch/nn/functional.py:4330, in affine_grid(theta, size, align_corners)
   4323     warnings.warn(
   4324         "Since version 1.3.0, affine_grid behavior has changed "
   4325         "for unit-size grids when align_corners=True. "
   4326         "This is not an intended use case of affine_grid. "
   4327         "See the documentation of affine_grid for details."
   4328     )
   4329 elif min(size) <= 0:
-> 4330     raise ValueError("Expected non-zero, positive output size. Got {}".format(size))
   4332 return torch.affine_grid_generator(theta, size, align_corners)

ValueError: Expected non-zero, positive output size. Got [2, 0, 224, 224]

Reproduction steps

import torch
import kornia.augmentation
my_fcn = kornia.augmentation.RandomAffine((-15., 20.))
input = torch.rand(2, 0, 224, 224)
out, transform = my_fcn(input)

Expected behavior

I would expect the transform to succeed, with an output of the same shape as the input.

Environment

wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py

Collecting environment information... PyTorch version: 1.13.1 Is debug build: False CUDA used to build PyTorch: None ROCM used to build PyTorch: N/A

OS: macOS 13.1 (x86_64) GCC version: Could not collect Clang version: 14.0.0 (clang-1400.0.29.202) CMake version: Could not collect Libc version: N/A

Python version: 3.8.16 (default, Dec 16 2022, 11:04:25) [Clang 14.0.0 (clang-1400.0.29.202)] (64-bit runtime) Python platform: macOS-13.1-x86_64-i386-64bit Is CUDA available: False CUDA runtime version: No CUDA CUDA_MODULE_LOADING set to: N/A GPU models and configuration: No CUDA Nvidia driver version: No CUDA cuDNN version: No CUDA HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True

CPU: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz

Versions of relevant libraries: [pip3] efficientnet-pytorch==0.7.1 [pip3] mypy==1.1.1 [pip3] mypy-extensions==1.0.0 [pip3] numpy==1.23.5 [pip3] pytorch-lightning==1.8.6 [pip3] segmentation-models-pytorch==0.3.2 [pip3] torch==1.13.1 [pip3] torchmetrics==0.11.4 [pip3] torchvision==0.14.1 [conda] Could not collect



### Additional context

_No response_

edgarriba commented 1 year ago

@timleslie thanks to raise this issue. The raised error comes directly from Pytorch which doesn't handle this corner case and since we just map grid_sample, I don't think we should over engineer their functionality. /cc @johnnv1 @ducha-aiki

To understand better, what's you use cases for C=0 and what it represents ?

timleslie commented 1 year ago

Hi @edgarriba, thanks for taking a look at this!

We're working on a computer vision model, and the C in our (B, C, H, W) tensor represents the number of different positive findings in a batch of data. In general the data distribution is such that each batch contains at least one positive finding, but we can't guarantee that, and we occasionally hit the edge case of no positive samples in our batch, at which point this particular transform fails.

Our full transform includes RandomHorizontalFlip, RandomVerticalFlip as well, both of which handle the C=0 case as expected.

edgarriba commented 1 year ago

I see -- tensors with a zero dimension has an empty data storage, i believe in your case you should skip that batch.

>>> torch.rand(1,1,1,0).storage().nbytes() == 0
True

kornia / kornia