beacon-biosignals / K8sClusterManagers.jl

A Julia cluster manager for Kubernetes
Other
31 stars 5 forks source link

`error: couldn't find any field with path "{.metadata.labels.worker-id}"` #62

Open omus opened 3 years ago

omus commented 3 years ago

Occurred in CI for README only change: #59

[ Info: Waiting for test-multi-addprocs job. This could take up to 4 minutes...
error: couldn't find any field with path "{.metadata.labels.worker-id}" in the list of objects
test-multi-addprocs: Error During Test at /home/runner/work/K8sClusterManagers.jl/K8sClusterManagers.jl/test/cluster.jl:234
  Got exception outside of a @test
  failed process: Process(`/home/runner/.julia/artifacts/e549ab3a763d3b31e726aa6336c6dbb75ee90a05/bin/kubectl get pods -l manager=test-multi-addprocs-wlpv5 '-o=jsonpath={range .items[*]}{.metadata.name}{"\n"}{end}' '--sort-by={.metadata.labels.worker-id}'`, ProcessExited(1)) [1]

  Stacktrace:
    [1] pipeline_error
      @ ./process.jl:525 [inlined]
    [2] read(cmd::Cmd)
      @ Base ./process.jl:412
    [3] read(cmd::Cmd, #unused#::Type{String})
      @ Base ./process.jl:421
    [4] readchomp
      @ ./io.jl:923 [inlined]
    [5] (::var"#19#21"{Cmd, String, String})(exe::String)
      @ Main ~/work/K8sClusterManagers.jl/K8sClusterManagers.jl/test/utils.jl:49
    [6] (::JLLWrappers.var"#2#3"{var"#19#21"{Cmd, String, String}, String})()
      @ JLLWrappers ~/.julia/packages/JLLWrappers/bkwIo/src/runtime.jl:49
    [7] withenv(::JLLWrappers.var"#2#3"{var"#19#21"{Cmd, String, String}, String}, ::Pair{String, String}, ::Vararg{Pair{String, String}, N} where N)
      @ Base ./env.jl:161
    [8] withenv_executable_wrapper(f::Function, executable_path::String, PATH::String, LIBPATH::String, adjust_PATH::Bool, adjust_LIBPATH::Bool)
      @ JLLWrappers ~/.julia/packages/JLLWrappers/bkwIo/src/runtime.jl:48
    [9] #invokelatest#2
      @ ./essentials.jl:708 [inlined]
   [10] invokelatest
      @ ./essentials.jl:706 [inlined]
   [11] #kubectl#7
      @ ~/.julia/packages/JLLWrappers/bkwIo/src/products/executable_generators.jl:7 [inlined]
   [12] kubectl
      @ ~/.julia/packages/JLLWrappers/bkwIo/src/products/executable_generators.jl:7 [inlined]
   [13] pod_names(labels::Pair{String, SubString{String}}; sort_by::String)
      @ Main ~/work/K8sClusterManagers.jl/K8sClusterManagers.jl/test/utils.jl:48
   [14] macro expansion
      @ ~/work/K8sClusterManagers.jl/K8sClusterManagers.jl/test/cluster.jl:265 [inlined]
   [15] macro expansion
      @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
   [16] top-level scope
      @ ~/work/K8sClusterManagers.jl/K8sClusterManagers.jl/test/cluster.jl:235
   [17] include(fname::String)
      @ Base.MainInclude ./client.jl:444
   [18] macro expansion
      @ ~/work/K8sClusterManagers.jl/K8sClusterManagers.jl/test/runtests.jl:27 [inlined]
   [19] macro expansion
      @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
   [20] macro expansion
      @ ~/work/K8sClusterManagers.jl/K8sClusterManagers.jl/test/runtests.jl:27 [inlined]
   [21] macro expansion
      @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
   [22] top-level scope
      @ ~/work/K8sClusterManagers.jl/K8sClusterManagers.jl/test/runtests.jl:20
   [23] include(fname::String)
      @ Base.MainInclude ./client.jl:444
   [24] top-level scope
      @ none:6
   [25] eval
      @ ./boot.jl:360 [inlined]
   [26] exec_options(opts::Base.JLOptions)
      @ Base ./client.jl:261
   [27] _start()
      @ Base ./client.jl:485

https://github.com/beacon-biosignals/K8sClusterManagers.jl/pull/59/checks?check_run_id=2470070043

omus commented 3 years ago

Note that this is the only cluster test to fail and this is also the only cluster test to make use of --sort-by. I suspect what is going on is that we've waited for the manager to running but we haven't guaranteed that the manager labeled the worker pods, or even ensured that the worker pods are running.

As this looks to be a test only issue I won't block the v0.1.0 release over this.

omus commented 3 years ago

Seems to be introduced with #48