apache / mxnet

Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more
https://mxnet.apache.org
Apache License 2.0
20.73k stars 6.81k forks source link

[Flaky Test] Python 3 TensorRT GPU : test_cvnets.test_tensorrt_on_cifar_resnets #14482

Open piyushghai opened 5 years ago

piyushghai commented 5 years ago

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-gpu/detail/PR-14269/4/pipeline/274

======================================================================
ERROR: test_cvnets.test_tensorrt_on_cifar_resnets
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/work/mxnet/tests/python/tensorrt/test_cvnets.py", line 152, in test_tensorrt_on_cifar_resnets
    speedup, acc_diff = run_experiment_for(model, batch_size, num_workers)
  File "/work/mxnet/tests/python/tensorrt/test_cvnets.py", line 114, in run_experiment_for
    num_workers=num_workers, use_tensorrt=False)
  File "/work/mxnet/tests/python/tensorrt/test_cvnets.py", line 56, in cifar10_infer
    executor = get_classif_model(model_name, use_tensorrt, ctx, batch_size)
  File "/work/mxnet/tests/python/tensorrt/test_cvnets.py", line 32, in get_classif_model
    net = gluoncv.model_zoo.get_model(model_name, pretrained=True)
  File "/usr/local/lib/python3.6/dist-packages/gluoncv/model_zoo/model_zoo.py", line 105, in get_model
    net = models[name](**kwargs)
  File "/usr/local/lib/python3.6/dist-packages/gluoncv/model_zoo/cifarresnet.py", line 349, in cifar_resnet56_v2
    return get_cifar_resnet(2, 56, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/gluoncv/model_zoo/cifarresnet.py", line 273, in get_cifar_resnet
    root=root), ctx=ctx)
  File "/usr/local/lib/python3.6/dist-packages/gluoncv/model_zoo/model_store.py", line 96, in get_model_file
    overwrite=True)
  File "/usr/local/lib/python3.6/dist-packages/gluoncv/utils/download.py", line 54, in download
    unit='KB', unit_scale=False, dynamic_ncols=True):
  File "/usr/local/lib/python3.6/dist-packages/tqdm/_tqdm.py", line 1022, in __iter__
    for obj in iterable:
  File "/usr/local/lib/python3.6/dist-packages/requests/models.py", line 748, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
mxnet-label-bot commented 5 years ago

Hey, this is the MXNet Label Bot. Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it. Here are my recommended labels: Test, Flaky

zachgk commented 5 years ago

@mxnet-label-bot add [Test, Flaky, Python]