search

JuliaLang / julia

The Julia Programming Language
https://julialang.org/
MIT License
45.74k stars 5.48k forks source link

FreeBSD buildbots failing in testset Distributed #33451

Closed mbauman closed 4 years ago

mbauman commented 5 years ago

I don't think we have an issue for this, and it seems to be consistently happening these days. E.g., https://build.julialang.org/#/builders/29/builds/4925/steps/2/logs/stdio:

Error in testset Distributed:
Error During Test at /usr/home/julia/buildbot/worker-tabularasa/tester_freebsd64/build/share/julia/test/testdefs.jl:20
  Got exception outside of a @test
  LoadError: Distributed test failed, cmd : `/usr/home/julia/buildbot/worker-tabularasa/tester_freebsd64/build/bin/julia -Cnative -J/usr/home/julia/buildbot/worker-tabularasa/tester_freebsd64/build/lib/julia/sys.so -g1 --check-bounds=yes --startup-file=no --depwarn=error /usr/home/julia/buildbot/worker-tabularasa/tester_freebsd64/build/share/julia/stdlib/v1.4/Distributed/test/distributed_exec.jl`
  Stacktrace:
   [1] error(::String) at ./error.jl:33
   [2] top-level scope at /usr/home/julia/buildbot/worker-tabularasa/tester_freebsd64/build/share/julia/stdlib/v1.4/Distributed/test/runtests.jl:11
   ...
  in expression starting at /usr/home/julia/buildbot/worker-tabularasa/tester_freebsd64/build/share/julia/stdlib/v1.4/Distributed/test/runtests.jl:10

ERROR: LoadError: Test run finished with errors
in expression starting at /usr/home/julia/buildbot/worker-tabularasa/tester_freebsd64/build/share/julia/test/runtests.jl:62
ERROR: A test has failed. Please submit a bug report (https://github.com/JuliaLang/julia/issues)

Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] #runtests#719(::Int64, ::Bool, ::Nothing, ::typeof(Base.runtests), ::Array{String,1}) at ./util.jl:818
 [3] (::Base.var"#kw##runtests")(::NamedTuple{(:ncores,),Tuple{Int64}}, ::typeof(Base.runtests), ::Array{String,1}) at ./none:0
 [4] top-level scope at none:3
caused by [exception 1]
failed process: Process(setenv(`/usr/home/julia/buildbot/worker-tabularasa/tester_freebsd64/build/bin/julia -Cnative -J/usr/home/julia/buildbot/worker-tabularasa/tester_freebsd64/build/lib/julia/sys.so -g1 /usr/home/julia/buildbot/worker-tabularasa/tester_freebsd64/build/bin/../share/julia/test/runtests.jl ambiguous compiler/inference compiler/validation compiler/ssair compiler/irpasses compiler/codegen compiler/inline compiler/contextual subarray strings/basic strings/search strings/util strings/io strings/types unicode/utf8 core worlds keywordargs numbers subtype char triplequote intrinsics dict hashing iobuffer staged offsetarray arrayops tuple reduce reducedim abstractarray intfuncs simdloop vecelement rational bitarray copy math fastmath functional iterators operators path ccall parse loading bigint sorting spawn backtrace exceptions file read version namedtuple mpfr broadcast complex floatapprox reflection regex float16 combinatorics sysinfo env rounding ranges mod2pi euler show client errorshow sets goto llvmcall llvmcall2 grisu some meta stacktraces docs misc threads stress enums cmdlineargs int interpreter checked bitset floatfuncs precompile boundscheck error cartesian osutils channels iostream secretbuffer specificity reinterpretarray syntax logging missing asyncmap SparseArrays/higherorderfns SparseArrays/sparse SparseArrays/sparsevector Pkg/resolve LinearAlgebra/triangular LinearAlgebra/qr LinearAlgebra/dense LinearAlgebra/matmul LinearAlgebra/schur LinearAlgebra/special LinearAlgebra/eigen LinearAlgebra/bunchkaufman LinearAlgebra/svd LinearAlgebra/lapack LinearAlgebra/tridiag LinearAlgebra/bidiag LinearAlgebra/diagonal LinearAlgebra/cholesky LinearAlgebra/lu LinearAlgebra/symmetric LinearAlgebra/generic LinearAlgebra/uniformscaling LinearAlgebra/lq LinearAlgebra/hessenberg LinearAlgebra/blas LinearAlgebra/adjtrans LinearAlgebra/pinv LinearAlgebra/givens LinearAlgebra/structuredbroadcast LinearAlgebra/addmul LibGit2/libgit2 Dates/accessors Dates/adjusters Dates/query Dates/periods Dates/ranges Dates/rounding Dates/types Dates/io Dates/arithmetic Dates/conversions Base64 CRC32c DelimitedFiles Distributed FileWatching Future InteractiveUtils Libdl Logging Markdown Mmap Printf Profile REPL Random SHA Serialization SharedArrays Sockets Statistics SuiteSparse Test UUIDs Unicode LibGit2/online download`,["PATH=/usr/home/julia/bin:/usr/home/julia/local/bin:/usr/home/julia/.local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/julia/bin", "PWD=/usr/home/julia/buildbot/worker-tabularasa/tester_freebsd64/build", "SUDO_GID=0", "SHELL=/usr/local/bin/bash", "MAIL=/var/mail/julia", "LOGNAME=julia", "SUDO_USER=root", "JULIA_CPU_THREADS=8", "SUDO_UID=0", "SUDO_COMMAND=/usr/home/julia/.local/bin/buildbot-worker start .", "USER=julia", "HOME=/home/julia", "TERM=unknown", "OPENBLAS_MAIN_FREE=1"]), ProcessExited(1)) [1]
ararslan commented 5 years ago

I wonder whether this might be something specific to the buildbot environment; running the Distributed tests repeatedly on my FreeBSD 11.3 machine with current Julia master, they consistently pass.

vtjnash commented 4 years ago

Error text was printed a bit earlier:

      From worker 9:    ERROR: LoadError: TaskFailedException:
      From worker 9:    Timed out waiting to read host:port string from worker.
      From worker 9:    Stacktrace:
      From worker 9:     [1] worker_from_id(::Distributed.ProcessGroup, ::Int64) at /usr/home/julia/buildbot/worker/package_freebsd64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:1059
      From worker 9:     [2] worker_from_id at /usr/home/julia/buildbot/worker/package_freebsd64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:1056 [inlined]
      From worker 9:     [3] #remote_do#156 at /usr/home/julia/buildbot/worker/package_freebsd64/build/usr/share/julia/stdlib/v1.4/Distributed/src/remotecall.jl:482 [inlined]
      From worker 9:     [4] remote_do at /usr/home/julia/buildbot/worker/package_freebsd64/build/usr/share/julia/stdlib/v1.4/Distributed/src/remotecall.jl:482 [inlined]
      From worker 9:     [5] kill at /usr/home/julia/buildbot/worker/package_freebsd64/build/usr/share/julia/stdlib/v1.4/Distributed/src/managers.jl:534 [inlined]
      From worker 9:     [6] create_worker(::Distributed.LocalManager, ::WorkerConfig) at /usr/home/julia/buildbot/worker/package_freebsd64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:581
      From worker 9:     [7] setup_launched_worker(::Distributed.LocalManager, ::WorkerConfig, ::Array{Int64,1}) at /usr/home/julia/buildbot/worker/package_freebsd64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:523
      From worker 9:     [8] (::Distributed.var"##43#46"{Distributed.LocalManager,Array{Int64,1},WorkerConfig})() at ./task.jl:333
      From worker 9:    Stacktrace:
      From worker 9:     [1] sync_end(::Array{Any,1}) at ./task.jl:300
      From worker 9:     [2] macro expansion at ./task.jl:319 [inlined]
      From worker 9:     [3] #addprocs_locked#40(::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol},NamedTuple{(:exename, :exeflags),Tuple{String,Cmd}}}, ::typeof(Distributed.addprocs_locked), ::Distributed.LocalManager) at /usr/home/julia/buildbot/worker/package_freebsd64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:477
      From worker 9:     [4] #addprocs_locked at ./none:0 [inlined]
      From worker 9:     [5] #addprocs#39(::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol},NamedTuple{(:exename, :exeflags),Tuple{String,Cmd}}}, ::typeof(addprocs), ::Distributed.LocalManager) at /usr/home/julia/buildbot/worker/package_freebsd64/build/usr/share/julia/stdlib/v1.4/Distributed/src/cluster.jl:441
      From worker 9:     [6] #addprocs at ./none:0 [inlined]
      From worker 9:     [7] #addprocs#247 at /usr/home/julia/buildbot/worker/package_freebsd64/build/usr/share/julia/stdlib/v1.4/Distributed/src/managers.jl:316 [inlined]
      From worker 9:     [8] #addprocs at ./none:0 [inlined]
      From worker 9:     [9] #addprocs_with_testenv#4 at /usr/home/julia/buildbot/worker-tabularasa/tester_freebsd64/build/share/julia/test/testenv.jl:29 [inlined]
      From worker 9:     [10] addprocs_with_testenv(::Int64) at /usr/home/julia/buildbot/worker-tabularasa/tester_freebsd64/build/share/julia/test/testenv.jl:29
      From worker 9:     [11] top-level scope at /usr/home/julia/buildbot/worker-tabularasa/tester_freebsd64/build/share/julia/stdlib/v1.4/Distributed/test/distributed_exec.jl:1079
      From worker 9:     [12] include at ./boot.jl:328 [inlined]
      From worker 9:     [13] include_relative(::Module, ::String) at ./loading.jl:1105
      From worker 9:     [14] include(::Module, ::String) at ./Base.jl:31
      From worker 9:     [15] exec_options(::Base.JLOptions) at ./client.jl:295
      From worker 9:     [16] _start() at ./client.jl:468
      From worker 9:    in expression starting at /usr/home/julia/buildbot/worker-tabularasa/tester_freebsd64/build/share/julia/stdlib/v1.4/Distributed/test/distributed_exec.jl:1078