Custom Image Decoder Implementation

[wms2537]

Description

Is it possible to implement a custom operator in C++ for mxnet which takes in a buffer as input instead of ndarray? I think it would be good to be able to implement custom image decoders that straight away decode data into ndarray.

I tried following the instructions here but it seems to require ndarray type for the data input.

An example is the jpeg2dct project which decodes jpeg buffer into numpy/tensorflow data types. This extracts the dct coefficients from the jpeg image and load it into numpy array or tensorflow tensor.

References

• https://github.com/uber-research/jpeg2dct

[github-actions[bot]]

Welcome to Apache MXNet (incubating)! We are on a mission to democratize AI, and we are glad that you are contributing to it by opening this issue. Please make sure to include all the relevant context, and one of the @apache/mxnet-committers will be here shortly. If you are interested in contributing to our project, let us know! Also, be sure to check out our guide on contributing to MXNet and our development guides wiki.

[bgawrych]

NDArray is object describing array, but under the hood it contains normal memory. You can access this memory by calling i.e:

int16_t* in_ptr = in_data.dptr<int16_t>();

(bsaed on example) All offsets of different dimensions you can deduce by calling proper ndarray functions.

You can also utilize numpy array to create mxnet's ndarray

from jpeg2dct.numpy import load, loads

import mxnet as mx
#read from a file
jpeg_file = 'test.jpg'
dct_y, dct_cb, dct_cr = load(jpeg_file)
print ("Y component DCT shape {} and type {}".format(dct_y.shape, dct_y.dtype))
print ("Cb component DCT shape {} and type {}".format(dct_cb.shape, dct_cb.dtype))
print ("Cr component DCT shape {} and type {}".format(dct_cr.shape, dct_cr.dtype))

print(type(dct_cr))
print(dct_cr)
mxnet_array = mx.nd.array(dct_cr)

print(mxnet_array.shape)
print(type(mxnet_array))
print(mxnet_array)

However if you wish to load file directly and operate on bytes you can check this operator: https://github.com/apache/incubator-mxnet/blob/9de2a48fabf1b8a60eb539640dea4c0637b5522b/src/io/image_io.cc#L371

[wms2537]

Is the type int_16 usable? I tried

NDArray(mshadow::Shape3(out_shape), Context::CPU(), false, mshadow::kInt16);

but got the following error

MXNetError: Unknown type enum 8

[bgawrych]

Looks like there is no support for int16 in NDArray interface. @szha is this known issue?

>>> mx.nd.ones((3,2), dtype='int16')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/bg/incubator-mxnet/python/mxnet/ndarray/ndarray.py", line 3301, in ones
    return _internal._ones(shape=shape, ctx=ctx, dtype=dtype, **kwargs)
  File "<string>", line 39, in _ones
  File "/home/bg/incubator-mxnet/python/mxnet/_ctypes/ndarray.py", line 82, in _imperative_invoke
    check_call(_LIB.MXImperativeInvokeEx(
  File "/home/bg/incubator-mxnet/python/mxnet/base.py", line 246, in check_call
    raise get_last_ffi_error()
mxnet.base.MXNetError: MXNetError: Invalid Input: 'int16', valid values are: {'bfloat16', 'bool', 'float16', 'float32', 'float64', 'int32', 'int64', 'int8', 'uint8'}, in operator _ones(name="", dtype="int16", ctx="cpu(0)", shape="(3, 2)")

[szha]

Yes. I think we should support it.

[bgawrych]

@wms2537 Would you like to contribute and add such support? I think adding new dtype into mshadow library shouldn't be problematic

[wms2537]

Should I also add in the cudnnDataType_t? As int32 is not available there.

typedef enum {
    CUDNN_DATA_FLOAT    = 0,
    CUDNN_DATA_DOUBLE   = 1,
    CUDNN_DATA_HALF     = 2,
    CUDNN_DATA_INT8     = 3,
    CUDNN_DATA_INT32    = 4,
    CUDNN_DATA_INT8x4   = 5,
    CUDNN_DATA_UINT8    = 6,
    CUDNN_DATA_UINT8x4  = 7,
    CUDNN_DATA_INT8x32  = 8,
    CUDNN_DATA_BFLOAT16 = 9,
    CUDNN_DATA_INT64    = 10,
} cudnnDataType_t;

[bgawrych]

I'm not sure, @MoisesHer @szha maybe you can confirm

[szha]

cudnnDataType_t doesn't come from mxnet and it doesn't support int16. See cudnn manual: https://docs.nvidia.com/deeplearning/cudnn/api/index.html#cudnnDataType_t

I haven't looked into the exact request, so the following comment is about adding new dtype in general. Adding a new dtype involves several steps, including:

• Adding the dtype definition: https://github.com/apache/incubator-mxnet/blob/5722f8b38af58c5a296e46ca695bfaf7cff85040/3rdparty/mshadow/mshadow/base.h#L352-L366
• Adding support in the desired operator. Lots of the operators in MXNet use macros for supporting multiple data types: https://github.com/apache/incubator-mxnet/blob/5722f8b38af58c5a296e46ca695bfaf7cff85040/src/operator/mxnet_op.h#L198-L241
• Adding option for operators that take explicit dtype. https://github.com/apache/incubator-mxnet/blob/66a65924f03e6e62ca0619afb02e2a674fcccbfd/src/operator/random/sample_op.h#L103-L105
• Adding frontend support for the dtype: https://github.com/apache/incubator-mxnet/blob/5722f8b38af58c5a296e46ca695bfaf7cff85040/python/mxnet/numpy/utils.py#L24-L55 https://github.com/apache/incubator-mxnet/blob/5722f8b38af58c5a296e46ca695bfaf7cff85040/python/mxnet/ndarray/ndarray.py#L65-L112
• Adding dlpack support. https://github.com/apache/incubator-mxnet/blob/5722f8b38af58c5a296e46ca695bfaf7cff85040/python/mxnet/dlpack.py#L46-L60
• Test for the new dtype. https://github.com/apache/incubator-mxnet/blob/5722f8b38af58c5a296e46ca695bfaf7cff85040/tests/python/unittest/test_numpy_ndarray.py#L38-L53

[wms2537]

I tried adding but got the following error when compiling, it seems that atomicAdd dows not support int16. How to fix this?

../src/operator/numpy/np_bincount_op.cu(46): error: no instance of overloaded function "atomicAdd" matches the argument list
            argument types are: (uint16_t *, const uint16_t)
          detected during:
            instantiation of "void mxnet::op::BincountFusedKernel::Map(int, const DType *, const OType *, OType *) [with DType=float, OType=uint16_t]" 
../src/operator/numpy/./../mxnet_op.h(1213): here
            instantiation of "void mxnet::op::mxnet_op::mxnet_generic_kernel<OP,Args...>(int, Args...) [with OP=mxnet::op::BincountFusedKernel, Args=<float *, uint16_t *, uint16_t *>]" 
../src/operator/numpy/./../mxnet_op.h(1234): here
            instantiation of "void mxnet::op::mxnet_op::Kernel<OP, mxnet::gpu>::Launch(mshadow::Stream<mshadow::gpu> *, int, Args...) [with OP=mxnet::op::BincountFusedKernel, Args=<float *, uint16_t *, uint16_t *>]" 

[bgawrych]

https://github.com/apache/incubator-mxnet/blob/5722f8b38af58c5a296e46ca695bfaf7cff85040/src/operator/numpy/np_bincount_op.cu#L103-L111

https://github.com/apache/incubator-mxnet/blob/5722f8b38af58c5a296e46ca695bfaf7cff85040/src/operator/numpy/np_bincount_op.cu#L146-L152

Template mechanism here create Kernel with int16 data type, because it can be returned by MSHADOW_TYPE_SWITCH - as cudnn doesn't support int16 you can define new macro (or search for existing one) which doesn't support int16 e.g ( MSHADOW_TYPE_SWITCH_NO_INT16) and replace it (I believe this problem will occurs in other places as well). However to avoid potential bugs for GPU implementations should be check whether input/output is int16 datatype and if it is, there should be fallback mechanism implemented to CPU implementation. Something like: https://github.com/apache/incubator-mxnet/blob/66a65924f03e6e62ca0619afb02e2a674fcccbfd/src/operator/tensor/elemwise_sum.cc#L117-L121