Support Sparse Input/Tensor

[shane-huang]

Neural nets are not only used for dense inputs such as image and audio. Sparse inputs should also be supported.

Now there's a trend to use NN for recommendation (e.g. wide&deep recommendation, ncf, rnn-based recommender). Many recommender works on dataframes (columns of features) and in such cases, sparse inputs are quite common because of heavily used one-hot encoding.

We need to support Sparse Tensor as input for mostly used layers. As for which layers we need to support for Sparse Input, I think for now we can just support the layers that are used in Wide&Deep recommender. Then we can add support for new layers per request from customers.

[sperlingxx]

so....how about the progress of "Sparse Input/Tensor"? Is there any schedule for this issue?

[qiuxin2012]

@sperlingxx We have implemented the Wide&Deep model with SparseTensor. Now we are doing more tests on the SparseTensor. Do you have any requirement for the SparseTensor?

[sperlingxx]

@qiuxin2012 是的，我们在尝试用bigDL做推荐/搜索相关的工作，基于Spark On ODPS

[qiuxin2012]

@sperlingxx 我们计划在下个月的v0.3.0的发布中加入SparseTensor的支持。 如果方便的话，最好可以告诉我们你需要SparseTensor的哪些功能，或者希望能支持一个什么样的模型。

[sperlingxx]

@qiuxin2012 哇~那太棒了~ 我们会试一些最新的paper里提出的模型，差不多就是deep&wide的变种吧。 所以希望有类似 torch.nn.SparseLinear/IndexLinear 的Layer~

[michellemay]

Have you tried with an optimizer? Looks like it just don't work with simple linear models..

java.lang.UnsupportedOperationException: SparseTensor: Unimplemented method
    at com.intel.analytics.bigdl.tensor.SparseTensor.stride(SparseTensor.scala:124)
    at com.intel.analytics.bigdl.tensor.DenseTensorMath$.addmm(DenseTensorMath.scala:565)
    at com.intel.analytics.bigdl.tensor.DenseTensor.addmm(DenseTensor.scala:1230)
    at com.intel.analytics.bigdl.nn.Linear.updateOutput(Linear.scala:108)
    at com.intel.analytics.bigdl.nn.Linear.updateOutput(Linear.scala:45)
    at com.intel.analytics.bigdl.nn.abstractnn.AbstractModule.forward(AbstractModule.scala:268)
    at com.intel.analytics.bigdl.nn.Sequential.updateOutput(Sequential.scala:39)
    at com.intel.analytics.bigdl.nn.abstractnn.AbstractModule.forward(AbstractModule.scala:268)
    at com.intel.analytics.bigdl.optim.DistriOptimizer$$anonfun$9$$anonfun$10$$anonfun$apply$2.apply$mcI$sp(DistriOptimizer.scala:234)
    at com.intel.analytics.bigdl.optim.DistriOptimizer$$anonfun$9$$anonfun$10$$anonfun$apply$2.apply(DistriOptimizer.scala:227)
    at com.intel.analytics.bigdl.optim.DistriOptimizer$$anonfun$9$$anonfun$10$$anonfun$apply$2.apply(DistriOptimizer.scala:227)
    at com.intel.analytics.bigdl.utils.ThreadPool$$anonfun$1$$anon$4.call(ThreadPool.scala:112)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

As well as the Sample object will throw on the isContinuous requirements with MethodNotImplemented error

  def apply[T: ClassTag](
        featureTensor: Tensor[T],
        labelTensor: Tensor[T])(implicit ev: TensorNumeric[T]) : Sample[T] = {
    require(featureTensor.isContiguous(), "featureTensor is not contiguous")
    require(labelTensor.isContiguous(), "labelTensor is not contiguous")
    if (featureTensor.getTensorType == DenseType) {
      ArraySample(featureTensor, labelTensor)
    } else {
      TensorSample(featureTensor, labelTensor)
    }
  }

[qiuxin2012]

@michellemay I'm not sure how you create the SparseTensor. We have a Wide&Deep example written in Python. And SparseTensor should works with Sparse Layers, see SparseTensor and Sparse Layers for help.

[yiheng]

It's better to have a introduction document under the Programming Guide section in our document website

[michellemay]

My remark is about scala/spark. I haven't tried in python as it's not my use case. What I say is that it can't work (release 0.4.0) in scala because there are requirement checks and unsupported method calls when I try to bypass theses checks.

[qiuxin2012]

@michellemay Em, I see. Could you put your sparse tensor into an Array to work around this? I will fix this bug in 0.5.0-snapshot.

def apply[T: ClassTag](
        featureTensors: Array[Tensor[T]],
        labelTensor: Tensor[T])(implicit ev: TensorNumeric[T]) : Sample[T] = {
    if (featureTensors.exists(_.getTensorType == SparseType)) {
      TensorSample(featureTensors, labelTensor)
    } else {
      ArraySample(featureTensors, labelTensor)
    }
  }

I think this one will works fine.