unit tests fail on RaspBerry pi with MxNet 1.0

tianweiz07 commented 6 years ago

Description

Install MxNet 1.0 on RaspBerry pi using docker_multiarch. When running uni tests, some of them fail

Environment info (Required)

----------Python Info---------- ('Version :', '2.7.13') ('Compiler :', 'GCC 6.3.0 20170516') ('Build :', ('default', 'Nov 24 2017 17:33:09')) ('Arch :', ('32bit', 'ELF')) ------------Pip Info----------- ('Version :', '9.0.1') ('Directory :', '/usr/lib/python2.7/dist-packages/pip') ----------MXNet Info----------- Version : 1.0.0 Directory : ./usr/local/lib/python2.7/dist-packages/mxnet-1.0.0-py2.7.egg/mxnet Commit Hash : 790ed30f834668c673c03d8b5ea0e46a6069576d ----------System Info---------- ('Platform :', 'Linux-4.9.59-v7+-armv7l-with-debian-9.3') ('system :', 'Linux') ('node :', 'raspberrypi') ('release :', '4.9.59-v7+') ('version :', '#1047 SMP Sun Oct 29 12:19:23 GMT 2017') ----------Hardware Info---------- ('machine :', 'armv7l') ('processor :', '') Architecture: armv7l Byte Order: Little Endian CPU(s): 4 On-line CPU(s) list: 0-3 Thread(s) per core: 1 Core(s) per socket: 4 Socket(s): 1 Model: 4 Model name: ARMv7 Processor rev 4 (v7l) CPU max MHz: 1200.0000 CPU min MHz: 600.0000 BogoMIPS: 76.80 Flags: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32 ----------Network Test---------- Setting timeout: 10 Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0180 sec, LOAD: 0.4988 sec. Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0154 sec, LOAD: 0.1196 sec. Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0256 sec, LOAD: 0.1110 sec. Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0168 sec, LOAD: 0.0623 sec. Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.0273 sec, LOAD: 0.0753 sec. Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.2554 sec, LOAD: 0.1503 sec.

Package used (Python/R/Scala/Julia): I'm using Python

Build info (Required if built from source)

Compiler (gcc/clang/mingw/visual studio): gcc

MXNet commit hash: 790ed30f834668c673c03d8b5ea0e46a6069576d

Build config: export CC = gcc export CXX = g++ export NVCC = nvcc

DMLC_LOG_STACK_TRACE=0

DEV = 0

DEBUG = 0

USE_PROFILER =

USE_SIGNAL_HANDLER =

ADD_LDFLAGS =-L/home/pi/deepgreen/mxnet/ext/lib /home/pi/deepgreen/mxnet/ext/lib/libopenblas.a

ADD_CFLAGS = -I/home/pi/deepgreen/mxnet/ext/include

USE_CUDA = 0

USE_CUDA_PATH = NONE

ENABLE_CUDA_RTC = 1

USE_CUDNN = 0

USE_NCCL = 0 USE_NCCL_PATH = NONE

USE_OPENCV = 0

USE_LIBJPEG_TURBO = 0 USE_LIBJPEG_TURBO_PATH = NONE

USE_OPENMP = 0

MKLML_ROOT=/usr/local

USE_MKL2017 = 0

USE_MKL2017_EXPERIMENTAL = 0

USE_NNPACK = 0

USE_BLAS = openblas

USE_LAPACK = 1

USE_LAPACK_PATH = /home/local/ANT/yunusko/IOT/deepgreen/ext/lib

USE_INTEL_PATH = NONE

ifeq ($(USE_BLAS), mkl) USE_STATIC_MKL = 1 else USE_STATIC_MKL = NONE endif

ARCH := armv7l ifneq (,$(filter $(ARCH), armv6l armv7l powerpc64le ppc64le aarch64)) USE_SSE=0 else USE_SSE=1 endif

USE_DIST_KVSTORE = 0

USE_HDFS = 0

LIBJVM=$(JAVA_HOME)/jre/lib/amd64/server

USE_S3 = 0

USE_OPERATOR_TUNING = 1

USE_GPERFTOOLS = 0

USE_JEMALLOC = 1

EXTRA_OPERATORS =

USE_CPP_PACKAGE = 0

Error Message:

test_executor.test_dot: sometimes passed sometimes segmentation fault.
test_gluon.test_conv: sometimes passed sometimes segmentation fault.
test_gluon.test_dtype: sometimes passed sometimes segmentation fault.
test_gluon.test_export: sometimes passed sometimes segmentation fault.
test_gluon_rnn.test_rnn_cells: sometimes passed sometimes segmentation fault.
test_ndarry.test_order: terminate called after throwing on instance of 'std::bad_alloc' what(): std::bad_alloc
test_optimizer.test_sgd: memory error Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/home/pi/ml@edge/build/mxnet/mxnet/tests/python/unittest1/test_optimizer.py", line 196, in test_sgd compare_optimizer(opt1(kwarg), opt2(kwarg), big_shape, np.float32) File "/home/pi/ml@edge/build/mxnet/mxnet/tests/python/unittest1/test_optimizer.py", line 56, in compare_optimizer compare_ndarray_tuple(state1, state2) File "/home/pi/ml@edge/build/mxnet/mxnet/tests/python/unittest1/test_optimizer.py", line 33, in compare_ndarray_tuple assert_almost_equal(t1.asnumpy(), t2.asnumpy(), rtol=rtol, atol=atol) File "/home/pi/ml@edge/build/mxnet/mxnet/python/mxnet/ndarray/ndarray.py", line 1789, in asnumpy data = np.empty(self.shape, dtype=self.dtype) MemoryError

Steps to reproduce

cd mxnet/tests/python/unittest
nosetests --verbosity=3

marcoabreu commented 6 years ago

Hello @tianweiz07, does this also happen on MXNet 1.1?

tianweiz07 commented 6 years ago

Hi @marcoabreu , I did not try the MxNet 1.1 version. Currently we are just planning to use MxNet 1.0 for our RaspBerry Pi devices.

lebeg commented 6 years ago

I see you are using your own lapack library:

USE_LAPACK_PATH = /home/local/ANT/yunusko/IOT/deepgreen/ext/lib

How did you build it?

lebeg commented 6 years ago

You seem to have a custom version of OpenBLAS as well

ADD_LDFLAGS =-L/home/pi/deepgreen/mxnet/ext/lib /home/pi/deepgreen/mxnet/ext/lib/libopenblas.a

ADD_CFLAGS = -I/home/pi/deepgreen/mxnet/ext/include

Is this desired? I think you can install it by just

apt install libopenblas-dev

This will give you lapack also.

piyushghai commented 6 years ago

@tianweiz07 Are you still facing the installation issue on Raspberry Pi ?

piyushghai commented 5 years ago

@tianweiz07 Can you confirm if you're not facing the issues you mentioned before ?

apache / mxnet