Open marcoabreu opened 6 years ago
Happens here too: http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-11127/11/pipeline . Please prioritize fixing this may become a blocker for 1.2.1 release.
I broke my Mac's Test, currently working on this...
I see this on the master branch, is it braking on the 1.2.0 branch as well?
@nswamy havent seen this yet on 1.2 but need to merge PRs on master before we cherry-pick for 1.2
This is due to the Spark tests that were newly merged https://github.com/apache/incubator-mxnet/pull/10462. I would like to disable so that we don't block the pipeline, we will activate these tests later. @CodingCat hope its ok with you
please go ahead and merge
but according to "terminate called without an active exception", it looks like the docker instance is assigned with too little memory so it is killed in the middle (does anyone how much memory we allocate to docker?)
@nswamy when you disable the test, would you please only disable https://github.com/apache/incubator-mxnet/pull/10462/files#diff-b9835287795c57e48fb9da87d39d06a8R64 (LeNet) for now (I believe LeNet is much more memory-intensive ) and leave MLP to guard the correctness in Spark side,
if MLP also behaves flaky, let's go ahead and kill it (I think it will not)
I don't think we assign any limits to Docker. But why does a unittest require so many resources?
@nswamy wouldn't these kind of heavy tests would be more suitable for nightly?
may be i haven't looked how long it takes to run these tests. We need to have some tests for training as well in the regular pipeline, may be we can run it on GPUs and keep MNIST only on CPU tests.
Currently all test I can see from CI build failure comes from CPU test. There is no difference in configuration running on CPU and GPU for the test on Spark currently. I think it might come from the diff between CPU/GPU build. Please correct me if this assumption is wrong.
@yzhliu
As the PR merged yesterday, let's keep focus if there are more CI died because of Scala. Please close this issue after some days if there is no further crashes.
This is resolved lets open new if there is an issue
I meant this is not blocking the pipeline, i'll keep it reopen and edit the Issue title/remove the Flaky label
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-11246/1/pipeline/
Exception in thread "Thread-113" java.lang.IllegalArgumentException: requirement failed: Failed to start ps scheduler process with exit code 134