apache / mxnet

Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more
https://mxnet.apache.org
Apache License 2.0
20.77k stars 6.8k forks source link

Crashing test test_laop6 #15975

Open marcoabreu opened 5 years ago

marcoabreu commented 5 years ago
test_operator_gpu.test_laop_6 ... 

Error running tests, python exited with status code C0000005

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fwindows-gpu/detail/PR-15972/1/pipeline/

mxnet-label-bot commented 5 years ago

Hey, this is the MXNet Label Bot. Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it. Here are my recommended labels: Test

ChaiBapchya commented 5 years ago

Faced this issue atleast 5-6 times and had mentioned about it in this issue https://github.com/apache/incubator-mxnet/issues/11720

marcoabreu commented 5 years ago

Awesome, thanks! But lets keep the tests separate.

ChaiBapchya commented 5 years ago
======================================================================
ERROR: test_operator_gpu.test_laop_6
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python37\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "C:\Python37\lib\site-packages\nose\util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "C:\jenkins_slave\workspace\ut-python-gpu\tests\python\gpu\../unittest\common.py", line 177, in test_new
    orig_test(*args, **kwargs)
  File "C:\jenkins_slave\workspace\ut-python-gpu\tests\python\gpu\../unittest\test_operator.py", line 6899, in test_laop_6
    check_grad(test_logabsdet, [a])
  File "C:\jenkins_slave\workspace\ut-python-gpu\tests\python\gpu\../unittest\test_operator.py", line 6869, in <lambda>
    atol=atol_bw, dtype=dtype)
  File "C:\jenkins_slave\workspace\ut-python-gpu\windows_package\python\mxnet\test_utils.py", line 992, in check_numeric_gradient
    symbolic_grads = {k:executor.grad_dict[k].asnumpy() for k in grad_nodes}
  File "C:\jenkins_slave\workspace\ut-python-gpu\windows_package\python\mxnet\test_utils.py", line 992, in <dictcomp>
    symbolic_grads = {k:executor.grad_dict[k].asnumpy() for k in grad_nodes}
  File "C:\jenkins_slave\workspace\ut-python-gpu\windows_package\python\mxnet\ndarray\ndarray.py", line 2406, in asnumpy
    ctypes.c_size_t(data.size)))
  File "C:\jenkins_slave\workspace\ut-python-gpu\windows_package\python\mxnet\base.py", line 254, in check_call
    raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [00:28:11] c:\jenkins_slave\workspace\build-gpu\include\mxnet\tuple.h:380: Check failed: ndim >= -1 (-42302925 vs. -1) : shape ndim must be >= -1, while received 

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fwindows-gpu/detail/PR-15980/4/pipeline

PR #15980