Unit tests for distributions methods

[lostella]

Currently there are no unit tests for distributions methods like .mean, .stddev, or .variance. These methods are covered in other tests, see here. However, we should avoid using untested .mean, .stddev, or .variance in the middle of testing other properties.

See also the discussion in #42 for some context.

[rlucas7]

If no one has taken this yet, I can work on this one. Not sure if tests should go in a pre-existing file though, all the existing filenames under test/ don't look like they apply?

[lostella]

@rlucas7 sounds great! Yes, you can initiate a new test module test/distribution/test_distribution_methods.py for this.

Ideally we should test that mean, stddev, variance, prob, log_prob, cdf (but the latter may not be implemented in all cases) return the expected tensor given the constructor arguments (and method argument x in some cases). But not all of them need to be done at once, it’s up to you.

[rlucas7]

@lostella they cannot all be done at once, I'm getting some mxnet errors from _ctype when I do: reproducing example:

from gluonts.distribution import StudentT
import mxnet as mx
s=StudentT(mu=mx.nd.zeros(shape=(3, 4, 5)), sigma=mx.nd.ones(shape=(3, 4, 5)), nu=mx.nd.array([4.2, 3.0]))
s.mean

gives me:

>>> s.mean
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/rlucas/gluon-ts/src/gluonts/distribution/student_t.py", line 71, in mean
    return self.F.where(self.nu > 1.0, self.mu, nans_like(self.mu))
  File "<string>", line 62, in where
  File "/Users/rlucas/anaconda3/envs/gluon-ts/lib/python3.6/site-packages/mxnet/_ctypes/ndarray.py", line 92, in _imperative_invoke
    ctypes.byref(out_stypes)))
  File "/Users/rlucas/anaconda3/envs/gluon-ts/lib/python3.6/site-packages/mxnet/base.py", line 252, in check_call
    raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [21:39:40] src/operator/tensor/./control_flow_op.h:191: Check failed: (*in_attrs)[0].Size() == static_cast<size_t>(tshape[0]) (2 vs. 3) 

Stack trace returned 8 entries:
[bt] (0) 0   libmxnet.so                         0x0000000112565c90 std::__1::__tree<std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, mxnet::NDArrayFunctionReg*>, std::__1::__map_value_compare<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, mxnet::NDArrayFunctionReg*>, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::allocator<std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, mxnet::NDArrayFunctionReg*> > >::destroy(std::__1::__tree_node<std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, mxnet::NDArrayFunctionReg*>, void*>*) + 2736
[bt] (1) 1   libmxnet.so                         0x0000000112565a3f std::__1::__tree<std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, mxnet::NDArrayFunctionReg*>, std::__1::__map_value_compare<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, mxnet::NDArrayFunctionReg*>, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::allocator<std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, mxnet::NDArrayFunctionReg*> > >::destroy(std::__1::__tree_node<std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, mxnet::NDArrayFunctionReg*>, void*>*) + 2143
[bt] (2) 2   libmxnet.so                         0x0000000112bcdf24 void mxnet::op::FillZerosRspImpl<mshadow::cpu>(mshadow::Stream<mshadow::cpu>*, mxnet::NDArray const&) + 10612
[bt] (3) 3   libmxnet.so                         0x0000000113c005a9 mxnet::imperative::SetShapeType(mxnet::Context const&, nnvm::NodeAttrs const&, std::__1::vector<mxnet::NDArray*, std::__1::allocator<mxnet::NDArray*> > const&, std::__1::vector<mxnet::NDArray*, std::__1::allocator<mxnet::NDArray*> > const&, mxnet::DispatchMode*) + 1577
[bt] (4) 4   libmxnet.so                         0x0000000113bfef06 mxnet::Imperative::Invoke(mxnet::Context const&, nnvm::NodeAttrs const&, std::__1::vector<mxnet::NDArray*, std::__1::allocator<mxnet::NDArray*> > const&, std::__1::vector<mxnet::NDArray*, std::__1::allocator<mxnet::NDArray*> > const&) + 742
[bt] (5) 5   libmxnet.so                         0x0000000113b4ad9e SetNDInputsOutputs(nnvm::Op const*, std::__1::vector<mxnet::NDArray*, std::__1::allocator<mxnet::NDArray*> >*, std::__1::vector<mxnet::NDArray*, std::__1::allocator<mxnet::NDArray*> >*, int, void* const*, int*, int, int, void***) + 1774
[bt] (6) 6   libmxnet.so                         0x0000000113b4bac0 MXImperativeInvokeEx + 176
[bt] (7) 7   libffi.6.dylib                      0x000000010e8ed884 ffi_call_unix64 + 76

If you you or someone on the team can confirm the error on your end would be great. Workaround can be to use distribution lists for the different methods, e.g. DISTRIBUTIONS_WITH_WORKING_MEANS etc. until that issue is fixed?

[lostella]

@rlucas7 nu should have the same shape as mu and sigma. Also, note that nu should be > 1 (component-wise) in order for the mean to be well defined.

[rlucas7]

@lostella I see:

 s=StudentT(mu=mx.nd.zeros(shape=(3, 1)), sigma=mx.nd.ones(shape=(3, 1)), nu=mx.nd.array([[4.2, 3.0, 3.0]]).T)
>>> s.mean

[[0.]
 [0.]
 [0.]]
<NDArray 3x1 @cpu(0)>
>>> s.variance

[[1.909091]
 [3.      ]
 [3.      ]]
<NDArray 3x1 @cpu(0)>
>>> s.stddev

[[1.3816986]
 [1.7320508]
 [1.7320508]]
<NDArray 3x1 @cpu(0)>

I see now the NDArrays need to be same dimension. There should probably be an error for that with an informative message to the user?

[lostella]

@rlucas7 yes, shape assertion would be a good idea in general. One caveat is that one can really assert shapes in F=mx.nd mode, but that's fine. I would deal with this in a separate issue, since it's potentially something that can be used in several other places (e.g. in the models networks).

[lostella]

See #184

[rlucas7]

182 partially addressed this but there are 3 outstanding issues blocking closing this issue.

1. stddev needs to be implemented for a couple of distributions.
2. prob, log_prob, cdf need to be tested
3. looks like cdf will need to be implemented in some distributions

For these I'll need to take a closer look at the multivariate and matrix valued distributions. When the output is array-like (e.g. the mxnet ndarray) and more than 2 dimensions I'm not clear along which 2 dimensions the stddev implementation would need to perform cholesky decompositions. If someone else is clear on that and can comment here to clarify that would be helpful.