How to use the hybrid kernel from the paper?

benemer commented 2 years ago

In Sec. 5.1 of the main paper "4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks", hybrid kernels are proposed to reduce the number of parameters by using a hypercubic kernel for the spatial and a hypercross kernel for the temporal dimension. However, it seems like the current version of the Minkowski Engine (0.5.4) does no longer support hybrid kernels. How can I use them?

import MinkowskiEngine as ME

kernel_size = 3
dimension = 4
list_region_types = [ME.RegionType.HYPER_CUBE, ME.RegionType.HYPER_CROSS]
for region_type in list_region_types:
    kernel_generator = ME.KernelGenerator(
        kernel_size=kernel_size,
        stride=1,
        dilation=1,
        is_transpose=False,
        region_type=region_type,
        region_offsets=None,
        expand_coordinates=False,
        axis_types=None,
        dimension=dimension,
    )
    print(
        "Kernel volume with region type {} is {}".format(
            str(region_type), kernel_generator.kernel_volume
        )
    )

axis_types = [
    ME.RegionType.HYPER_CUBE,
    ME.RegionType.HYPER_CUBE,
    ME.RegionType.HYPER_CUBE,
    ME.RegionType.HYPER_CROSS,
]
kernel_generator = ME.KernelGenerator(
    kernel_size=kernel_size,
    stride=1,
    dilation=1,
    is_transpose=False,
    region_type=ME.RegionType.HYBRID,
    region_offsets=None,
    expand_coordinates=False,
    axis_types=axis_types,
    dimension=dimension,
)
print("Kernel volume with hybrid kernel is {}.".format(kernel_generator.kernel_volume))

Output:

Kernel volume with region type RegionType.HYPER_CUBE is 81
Kernel volume with region type RegionType.HYPER_CROSS is 9
Traceback (most recent call last):
  File "test.py", line 35, in <module>
    region_type=ME.RegionType.HYBRID,
AttributeError: type object 'MinkowskiEngineBackend._C.RegionType' has no attribute 'HYBRID'

Expected behavior As shown in the example code above, I would like to specify different kernel shapes for the spatial and temporal axis. According to the docs, this can be achieved by passing a list of RegionType as axis_types.

However, this results in

AssertionError: Axis types must be None when region_type is given

It is not possible to pass region_type=ME.RegionType.HYBRID since the hybrid RegionType is no longer available in Minkowski Engine 0.5.4 (see above).

How is the hybrid kernel of the paper implemented in the latest Minkowski Engine version?

Desktop:

OS: Ubuntu 20.04
Python version: 3.8.10
Pytorch version: 1.10
CUDA version: 10.1
NVIDIA Driver version: 470.86
Minkowski Engine version 0.5.4

suyunzzz commented 2 years ago

hello, have you solved this problem? it seems that there are three RegionType in https://nvidia.github.io/MinkowskiEngine/common.html

i want to know if i use dim=3, i do not need use RegionType=HYBRID? HYBRID just for dim=4?

benemer commented 2 years ago

Hi, I was able to use the RegionType=HYBRID with ME version 4.3. However, a lot of other functionalities are not available here, therefore I would like to use the hybrid kernel with the latest version.

Regarding your question: The RegionType has nothing to do with the dimensions. The region type determines the shape of the kernel, see Fig. 3 of the paper for an example with dim=3, dim=2, and different kernel shapes. I think that the third region type CUSTOM allows to build a hybrid kernel of hypercross and hypercube sub-kernels. However, it seems like the CUSTOM region type is not yet available:

The custom region has not been ported to v0.5 yet. Sorry for the delay. I'll try to finish it in a few weeks. Originally posted by @chrischoy in https://github.com/NVIDIA/MinkowskiEngine/issues/355#issuecomment-849933126

Ltwicke commented 2 years ago

Hello,

I too ran into the problem described in

I feel like the main problem lies in he fact, that they explain nowhere, what the region_offset is exactly and how i can construct it. Does anyone know anything about that?

izav-hub commented 1 year ago

I'm having the same problems... Could anyone shed some light on the changes that need to be made to https://github.com/chrischoy/SpatioTemporalSegmentation in order to make it compatible with MinkowskiEngine v0.5? @chrischoy

NVIDIA / MinkowskiEngine

How to use the hybrid kernel from the paper? #424