search

dmlc / mxnet-memonger

Sublinear memory optimization for deep learning, reduce GPU memory cost to train deeper nets
Apache License 2.0
308 stars 63 forks source link

memonger does not work on 0.10.1 #6

Closed taineleau-zz closed 7 years ago

taineleau-zz commented 7 years ago

Hi,

I tried to put the model in a training script, but I found I couldn't make it work using memonger.

If I feed the planned net to mx.FeedForward() it would report:

 [23:14:29] src/operator/./pooling-inl.h:166: Check failed: dshape.ndim() >= 3U (0 vs. 3) Pooling: Input data should be  3D in (batch, channel, x) Or 4D in (batch, channel, y, x)  Or 5D in (batch, channel, d, y, x)

while everything works well if I don't use memonger.

further information:

infer_shape error. Arguments:
Traceback (most recent call last):
  File "train_densenet.py", line 240, in <module>
    main()
  File "train_densenet.py", line 161, in main
    net_planned = memonger.search_plan(symbol)
  File "/home/taineleau/densenet.mxnet/memonger.py", line 140, in search_plan
    sym = make_mirror_plan(sym, threshold=threshold, plan_info=info, **kwargs)
  File "/home/taineleau/densenet.mxnet/memonger.py", line 62, in make_mirror_plan
    _, out_shapes, _ = internals.infer_shape(**kwargs)
  File "/home/taineleau/.local/lib/python2.7/site-packages/mxnet-0.10.1-py2.7.egg/mxnet/symbol.py", line 878, in infer_shape
    res = self._infer_shape_impl(False, *args, **kwargs)
  File "/home/taineleau/.local/lib/python2.7/site-packages/mxnet-0.10.1-py2.7.egg/mxnet/symbol.py", line 1004, in _infer_shape_impl
    ctypes.byref(complete)))
  File "/home/taineleau/.local/lib/python2.7/site-packages/mxnet-0.10.1-py2.7.egg/mxnet/base.py", line 85, in check_call
    raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: Error in operator TBstage1_pool1: [23:14:29] src/operator/./pooling-inl.h:166: Check failed: dshape.ndim() >= 3U (0 vs. 3) Pooling: Input data should be  3D in (batch, channel, x) Or 4D in (batch, channel, y, x)  Or 5D in (batch, channel, d, y, x)