search

apache / mxnet

Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more
https://mxnet.apache.org
Apache License 2.0
20.77k stars 6.79k forks source link

Flaky test_gluon.test_conv causes other tests to fail on Windows #11506

Closed marcoabreu closed 5 years ago

marcoabreu commented 6 years ago

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/master/1106/pipeline

test_operator_gpu.test_ndarray_crop ... ok

test_operator_gpu.test_cell_fill_shape ... ok

test_operator_gpu.test_conv ... [23:56:08] c:\jenkins_slave\workspace\build-gpu@2\src\operator\nn\cudnn\./cudnn_algoreg-inl.h:107: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)

[23:56:08] C:/jenkins_slave/workspace/build-gpu@2/src/operator/nn/convolution.cu:148: This convolution is not supported by cudnn, MXNET convolution is applied.

[23:56:08] C:/jenkins_slave/workspace/build-gpu@2/src/operator/nn/convolution.cu:227: This convolution is not supported by cudnn, MXNET convolution is applied.

[23:56:08] C:/jenkins_slave/workspace/build-gpu@2/src/operator/nn/convolution.cu:148: This convolution is not supported by cudnn, MXNET convolution is applied.

[23:56:08] C:/jenkins_slave/workspace/build-gpu@2/src/operator/nn/convolution.cu:227: This convolution is not supported by cudnn, MXNET convolution is applied.

[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=270901180 to reproduce.

[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1177804468 to reproduce.

ERROR

test_operator_gpu.test_layer_fill_shape ... ERROR

test_operator_gpu.test_ndarray_concatenate ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=820959189 to reproduce.

ERROR

test_operator_gpu.test_normal_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1440948114 to reproduce.

ERROR

test_operator_gpu.test_sparse_nd_transpose ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=479442568 to reproduce.

ERROR

test_operator_gpu.test_slice_channel ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=952262734 to reproduce.

ERROR

test_operator_gpu.test_sparse_nd_storage_fallback ... [23:56:09] c:\jenkins_slave\workspace\build-gpu@2\src\operator\../common/utils.h:447: 

Storage type fallback detected:

operator = broadcast_add

input storage types = [default, default, ]

output storage types = [csr, ]

params = {}

context.dev_mask = gpu

The operator with default storage type will be dispatched for execution. You're seeing this warning message because the operator above is unable to process the given ndarrays with specified storage types, context and parameter. Temporary dense ndarrays are generated in order to execute the operator. This does not affect the correctness of the programme. You can set environment variable MXNET_STORAGE_FALLBACK_LOG_VERBOSE to 0 to suppress this warning.

[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=2072600123 to reproduce.

ERROR

test_operator_gpu.test_clip ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1670877169 to reproduce.

ERROR

test_operator_gpu.test_convolution_with_type ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1234 to reproduce.

ERROR

test_operator_gpu.test_deconv ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=77155738 to reproduce.

[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=542068512 to reproduce.

ERROR

test_operator_gpu.test_dot ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1533114250 to reproduce.

ERROR

test_operator_gpu.test_uniform_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1203830416 to reproduce.

ERROR

test sparse random operator on cpu ... ok

test_operator_gpu.test_nag ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=0 to reproduce.

ERROR

test_operator_gpu.test_gamma_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=579518551 to reproduce.

ERROR

test regression operator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1630735113 to reproduce.

ERROR

test_operator_gpu.test_sparse_nd_astype ... ok

test_operator_gpu.test_exponential_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=493650054 to reproduce.

ERROR

test_operator_gpu.test_sparse_nd_astype_copy ... ok

test_operator_gpu.test_convolution_options ... SKIP: test fails intermittently. temporarily disabled till it gets fixed. tracked at https://github.com/apache/incubator-mxnet/issues/10141

test_operator_gpu.test_reduce ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1367744731 to reproduce.

ERROR

test_operator_gpu.test_pool ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1525342399 to reproduce.

[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=785623715 to reproduce.

ERROR

test_operator_gpu.test_poisson_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1266754562 to reproduce.

ERROR

test_operator_gpu.test_negative_binomial_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=372547776 to reproduce.

ERROR

test_operator_gpu.test_sparse_nd_pickle ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=0 to reproduce.

ERROR

test_operator_gpu.test_broadcast ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=526949765 to reproduce.

ERROR

test_operator_gpu.test_ftml ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=0 to reproduce.

ERROR

test_operator_gpu.test_batchnorm ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1425658102 to reproduce.

[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1562292807 to reproduce.

ERROR

test_operator_gpu.test_sparse_nd_save_load ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=0 to reproduce.

ERROR

test_operator_gpu.test_instancenorm ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=832404596 to reproduce.

[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1764718120 to reproduce.

ERROR

test_operator_gpu.test_multinomial_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1990187257 to reproduce.

ERROR

test_operator_gpu.test_layernorm ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=803289623 to reproduce.

[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1460168409 to reproduce.

ERROR

test_operator_gpu.test_reflectionpad ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=897856965 to reproduce.

[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1138894034 to reproduce.

ERROR

test_operator_gpu.test_convolution_versions ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1576481502 to reproduce.

ERROR

test_operator_gpu.test_broadcast_binary ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1660279093 to reproduce.

ERROR

test_operator_gpu.test_reshape ... ok

test_operator_gpu.test_sparse_nd_unsupported ... ok

test_operator_gpu.test_slice ... ok

test_operator_gpu.test_at ... ok

test_operator_gpu.test_create_csr ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=106598815 to reproduce.

ERROR

test_operator_gpu.test_with_random_seed ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1495502410 to reproduce.

ERROR

test_operator_gpu.test_elemwise_csr_same_zeros ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=583265934 to reproduce.

ERROR

test_operator_gpu.test_moveaxis ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=197256063 to reproduce.

ERROR

test_operator_gpu.test_deferred_init ... ok

test_operator_gpu.test_deconvolution_with_type ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=838817885 to reproduce.

ERROR

test_operator_gpu.test_arange ... [23:56:09] c:\jenkins_slave\workspace\build-gpu@2\src\common\../operator/mxnet_op.h:576: Check failed: (err) == (cudaSuccess) Name: mxnet_generic_kernel ErrStr:unspecified launch failure

(py3) C:\jenkins_slave\workspace\ut-python-gpu>IF -1073740791 NEQ 0 exit /b 1
marcoabreu commented 6 years ago

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/master/1107/pipeline/ contains more details

marcoabreu commented 6 years ago

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/master/1106/pipeline