apache / mxnet

Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more
https://mxnet.apache.org
Apache License 2.0
20.77k stars 6.79k forks source link

Set range in NDArray from Python list fails #16647

Closed fhieber closed 4 years ago

fhieber commented 4 years ago

Description

Setting a range in an NDArray using a Python list fails with an obscure error about broadcasting when the range to set values in is equal to the size of the respective dimension in the target array. Bug occurs in latest master (not in MXNet 1.5.1). When another NDArray is used as data source, no error is thrown.

Error Message

ValueError                                Traceback (most recent call last)
<ipython-input-32-7d7d8c3fa8f4> in <module>
----> 1 dst[0, :len(src), 0] = src

~/anaconda3/lib/python3.7/site-packages/mxnet/ndarray/ndarray.py in __setitem__(self, key, value)
    523             indexing_dispatch_code = get_indexing_dispatch_code(slc_key)
    524             if indexing_dispatch_code == _NDARRAY_BASIC_INDEXING:
--> 525                 self._set_nd_basic_indexing(key, value)
    526             elif indexing_dispatch_code == _NDARRAY_ADVANCED_INDEXING:
    527                 self._set_nd_advanced_indexing(key, value)

~/anaconda3/lib/python3.7/site-packages/mxnet/ndarray/ndarray.py in _set_nd_basic_indexing(self, key, value)
    958                 # Other array-like
    959                 value_nd = self._prepare_value_nd(
--> 960                     value, bcast_shape=self.shape
    961                 )
    962                 value_nd.copyto(self)

~/anaconda3/lib/python3.7/site-packages/mxnet/ndarray/ndarray.py in _prepare_value_nd(self, value, bcast_shape, squeeze_axes)
    755                 value_nd = value_nd.reshape(bcast_shape)
    756             else:
--> 757                 value_nd = value_nd.broadcast_to(bcast_shape)
    758         return value_nd
    759

~/anaconda3/lib/python3.7/site-packages/mxnet/ndarray/ndarray.py in broadcast_to(self, shape)
   2275         broadcasting_axes = np.nonzero(cur_shape_arr != np.array(shape))
   2276         if (cur_shape_arr[broadcasting_axes] != 1).any():
-> 2277             raise ValueError(err_str)
   2278         if cur_shape != self.shape:
   2279             return op.broadcast_to(self.reshape(cur_shape), shape=shape)

ValueError: operands could not be broadcast together with remapped shapes[original->remapped]: (3,) and requested shape (1, 3, 1)

To Reproduce

dst = mx.nd.zeros((1, 3, 1))  # destination array
src = [1, 2, 3]
dst[0, :len(src), 0] = src

Using an NDArray as source data works

dst = mx.nd.zeros((1, 3, 1))  # destination array
src = [1, 2, 3]
dst[0, :len(src), 0] = mx.nd.array(src)

And also any range that is < dim2 of dst

dst = mx.nd.zeros((1, 3, 1))  # destination array
src = [1, 2]
dst[0, :len(src), 0] = src

Environment

----------Python Info----------
Version      : 3.7.1
Compiler     : Clang 4.0.1 (tags/RELEASE_401/final)
Build        : ('default', 'Dec 14 2018 13:28:58')
Arch         : ('64bit', '')
------------Pip Info-----------
Version      : 18.1
Directory    : /Users/fhieber/anaconda3/lib/python3.7/site-packages/pip
----------MXNet Info-----------
Version      : 1.6.0
Directory    : /Users/fhieber/anaconda3/lib/python3.7/site-packages/mxnet
Num GPUs     : 0
Commit Hash   : 78627387d41f284067bec164063ae53dbc32adf3
----------System Info----------
Platform     : Darwin-19.0.0-x86_64-i386-64bit
system       : Darwin
node         : -----
release      : 19.0.0
version      : Darwin Kernel Version 19.0.0: Wed Sep 25 20:18:50 PDT 2019; root:xnu-6153.11.26~2/RELEASE_X86_64
----------Hardware Info----------
machine      : x86_64
processor    : i386
b'machdep.cpu.brand_string: Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz'
b'machdep.cpu.features: FPU VME DE PSE TSC MSR PAE MCE CX8 APIC SEP MTRR PGE MCA CMOV PAT PSE36 CLFSH DS ACPI MMX FXSR SSE SSE2 SS HTT TM PBE SSE3 PCLMULQDQ DTES64 MON DSCPL VMX SMX EST TM2 SSSE3 FMA CX16 TPR PDCM SSE4.1 SSE4.2 x2APIC MOVBE POPCNT AES PCID XSAVE OSXSAVE SEGLIM64 TSCTMR AVX1.0 RDRAND F16C'
b'machdep.cpu.leaf7_features: RDWRFSGS TSC_THREAD_OFFSET SGX BMI1 HLE AVX2 SMEP BMI2 ERMS INVPCID RTM FPU_CSDS MPX RDSEED ADX SMAP CLFSOPT IPT MDCLEAR TSXFA IBRS STIBP L1DF SSBD'
b'machdep.cpu.extfeatures: SYSCALL XD 1GBPAGE EM64T LAHF LZCNT PREFETCHW RDTSCP TSCI'
----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.1085 sec, LOAD: 0.8791 sec.
Timing for GluonNLP GitHub: https://github.com/dmlc/gluon-nlp, DNS: 0.0012 sec, LOAD: 0.6442 sec.
Timing for GluonNLP: http://gluon-nlp.mxnet.io, DNS: 0.0397 sec, LOAD: 0.2313 sec.
Timing for D2L: http://d2l.ai, DNS: 0.1149 sec, LOAD: 0.3263 sec.
Timing for D2L (zh-cn): http://zh.d2l.ai, DNS: 0.0340 sec, LOAD: 0.4763 sec.
Timing for FashionMNIST: https://repo.mxnet.io/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0398 sec, LOAD: 0.8817 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.1047 sec, LOAD: 0.6946 sec.
Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0200 sec, LOAD: 0.2118 sec.
szha commented 4 years ago

This should be a release blocker.

cc @ptrendx @reminisce

samskalicky commented 4 years ago

@lanking520 assign @szha