search

beacon-biosignals / K8sClusterManagers.jl

A Julia cluster manager for Kubernetes
Other
31 stars 5 forks source link

Improve error messaging when using an incorrect namespace #12

Closed omus closed 3 years ago

omus commented 3 years ago

When specifying an incorrect namespace you get the following error:

julia> pids = K8sClusterManagers.addprocs_pod(2; namespace="ondawerks")
[ Info: unsupported crd.k8s.amazonaws.com/v1alpha1
[ Info: unsupported vpcresources.k8s.aws/v1beta1
ERROR: Swagger.ApiException(403, "Forbidden", HTTP.Messages.Response:
"""
HTTP/1.1 403 Forbidden
Audit-Id: dd0173b7-7fe9-4ffc-811f-0eca71fdc943
Cache-Control: no-cache, private
Content-Length: 392
Content-Type: application/json
Date: Fri, 19 Mar 2021 18:04:26 GMT
X-Content-Type-Options: nosniff
Connection: close

{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"driver-2021-03-19--17-58-47-kpqnt\" is forbidden: User \"system:serviceaccount:project-ondawerks:ondawerks-service-account\" cannot get resource \"pods\" in API group \"\" in the namespace \"ondawerks\"","reason":"Forbidden","details":{"name":"driver-2021-03-19--17-58-47-kpqnt","kind":"pods"},"code":403}
""")
Stacktrace:
 [1] exec(::Swagger.Ctx) at /root/.julia/packages/Swagger/4eKnR/src/client.jl:213
 [2] readCoreV1NamespacedPod(::Kuber.Kubernetes.CoreV1Api, ::String, ::String; pretty::Nothing, exact::Nothing, __export__::Nothing, _mediaType::Nothing) at /root/.julia/packages/Kuber/UfXri/src/api/api_CoreV1Api.jl:3912
 [3] #readNamespacedPod#735 at /root/.julia/packages/Kuber/UfXri/src/apialiases.jl:2026 [inlined]
 [4] readNamespacedPod(::Kuber.Kubernetes.CoreV1Api, ::String, ::String) at /root/.julia/packages/Kuber/UfXri/src/apialiases.jl:2026
 [5] macro expansion at /root/.julia/packages/Kuber/UfXri/src/simpleapi.jl:79 [inlined]
 [6] macro expansion at /root/.julia/packages/Retry/vS1bg/src/repeat_try.jl:192 [inlined]
 [7] get(::Kuber.KuberContext, ::Symbol, ::String, ::Nothing; max_tries::Int64, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /root/.julia/packages/Kuber/UfXri/src/simpleapi.jl:78
 [8] get at /root/.julia/packages/Kuber/UfXri/src/simpleapi.jl:62 [inlined] (repeats 2 times)
 [9] self_pod at /root/.julia/packages/K8sClusterManagers/eAv58/src/native_driver.jl:19 [inlined]
 [10] default_pod(::Kuber.KuberContext, ::Int64, ::Cmd, ::String; image::Nothing, memory::String, cpu::String, serviceAccountName::Nothing, base_obj::Kuber.Kubernetes.IoK8sApiCoreV1Pod, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /root/.julia/packages/K8sClusterManagers/eAv58/src/native_driver.jl:35
 [11] (::K8sClusterManagers.var"#3#4"{typeof(identity),String,Cmd,Base.Iterators.Pairs{Symbol,Any,NTuple{4,Symbol},NamedTuple{(:image, :memory, :cpu, :serviceAccountName),Tuple{Nothing,String,String,Nothing}}},Kuber.KuberContext})(::Int64) at ./none:0
 [12] iterate at ./generator.jl:47 [inlined]
 [13] _all at ./reduce.jl:827 [inlined]
 [14] all at ./reduce.jl:823 [inlined]
 [15] Dict(::Base.Generator{UnitRange{Int64},K8sClusterManagers.var"#3#4"{typeof(identity),String,Cmd,Base.Iterators.Pairs{Symbol,Any,NTuple{4,Symbol},NamedTuple{(:image, :memory, :cpu, :serviceAccountName),Tuple{Nothing,String,String,Nothing}}},Kuber.KuberContext}}) at ./dict.jl:130
 [16] #default_pods_and_context#2 at /root/.julia/packages/K8sClusterManagers/eAv58/src/native_driver.jl:67 [inlined]
 [17] K8sNativeManager(::UnitRange{Int64}, ::String, ::Cmd; configure::Function, namespace::String, retry_seconds::Int64, kwargs::Base.Iterators.Pairs{Symbol,Any,NTuple{4,Symbol},NamedTuple{(:image, :memory, :cpu, :serviceAccountName),Tuple{Nothing,String,String,Nothing}}}) at /root/.julia/packages/K8sClusterManagers/eAv58/src/native_driver.jl:82
 [18] addprocs_pod(::Int64; driver_name::String, configure::Function, namespace::String, image::Nothing, serviceAccountName::Nothing, memory::String, cpu::String, retry_seconds::Int64, exename::Cmd, exeflags::Cmd, params::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /root/.julia/packages/K8sClusterManagers/eAv58/src/native_driver.jl:193
 [19] top-level scope at REPL[4]:1
caused by [exception 3]
UndefVarError: readPod not defined
Stacktrace:
 [1] top-level scope
 [2] eval at ./boot.jl:331 [inlined]
 [3] eval at /root/.julia/packages/Kuber/UfXri/src/Kuber.jl:1 [inlined]
 [4] get(::Kuber.KuberContext, ::Symbol, ::String, ::Nothing; max_tries::Int64, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /root/.julia/packages/Kuber/UfXri/src/simpleapi.jl:66
 [5] get at /root/.julia/packages/Kuber/UfXri/src/simpleapi.jl:62 [inlined] (repeats 2 times)
 [6] self_pod at /root/.julia/packages/K8sClusterManagers/eAv58/src/native_driver.jl:19 [inlined]
 [7] default_pod(::Kuber.KuberContext, ::Int64, ::Cmd, ::String; image::Nothing, memory::String, cpu::String, serviceAccountName::Nothing, base_obj::Kuber.Kubernetes.IoK8sApiCoreV1Pod, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /root/.julia/packages/K8sClusterManagers/eAv58/src/native_driver.jl:35
 [8] (::K8sClusterManagers.var"#3#4"{typeof(identity),String,Cmd,Base.Iterators.Pairs{Symbol,Any,NTuple{4,Symbol},NamedTuple{(:image, :memory, :cpu, :serviceAccountName),Tuple{Nothing,String,String,Nothing}}},Kuber.KuberContext})(::Int64) at ./none:0
 [9] iterate at ./generator.jl:47 [inlined]
 [10] _all at ./reduce.jl:827 [inlined]
 [11] all at ./reduce.jl:823 [inlined]
 [12] Dict(::Base.Generator{UnitRange{Int64},K8sClusterManagers.var"#3#4"{typeof(identity),String,Cmd,Base.Iterators.Pairs{Symbol,Any,NTuple{4,Symbol},NamedTuple{(:image, :memory, :cpu, :serviceAccountName),Tuple{Nothing,String,String,Nothing}}},Kuber.KuberContext}}) at ./dict.jl:130
 [13] #default_pods_and_context#2 at /root/.julia/packages/K8sClusterManagers/eAv58/src/native_driver.jl:67 [inlined]
 [14] K8sNativeManager(::UnitRange{Int64}, ::String, ::Cmd; configure::Function, namespace::String, retry_seconds::Int64, kwargs::Base.Iterators.Pairs{Symbol,Any,NTuple{4,Symbol},NamedTuple{(:image, :memory, :cpu, :serviceAccountName),Tuple{Nothing,String,String,Nothing}}}) at /root/.julia/packages/K8sClusterManagers/eAv58/src/native_driver.jl:82
 [15] addprocs_pod(::Int64; driver_name::String, configure::Function, namespace::String, image::Nothing, serviceAccountName::Nothing, memory::String, cpu::String, retry_seconds::Int64, exename::Cmd, exeflags::Cmd, params::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /root/.julia/packages/K8sClusterManagers/eAv58/src/native_driver.jl:193
 [16] top-level scope at REPL[4]:1
caused by [exception 2]
Swagger.ApiException(403, "Forbidden", HTTP.Messages.Response:
"""
HTTP/1.1 403 Forbidden
Audit-Id: 2f0534e1-0734-4736-b5ef-2236b7f7c670
Cache-Control: no-cache, private
Content-Length: 392
Content-Type: application/json
Date: Fri, 19 Mar 2021 18:04:26 GMT
X-Content-Type-Options: nosniff
Connection: close

{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"driver-2021-03-19--17-58-47-kpqnt\" is forbidden: User \"system:serviceaccount:project-ondawerks:ondawerks-service-account\" cannot get resource \"pods\" in API group \"\" in the namespace \"ondawerks\"","reason":"Forbidden","details":{"name":"driver-2021-03-19--17-58-47-kpqnt","kind":"pods"},"code":403}
""")
Stacktrace:
 [1] exec(::Swagger.Ctx) at /root/.julia/packages/Swagger/4eKnR/src/client.jl:213
 [2] readCoreV1NamespacedPod(::Kuber.Kubernetes.CoreV1Api, ::String, ::String; pretty::Nothing, exact::Nothing, __export__::Nothing, _mediaType::Nothing) at /root/.julia/packages/Kuber/UfXri/src/api/api_CoreV1Api.jl:3912
 [3] #readNamespacedPod#735 at /root/.julia/packages/Kuber/UfXri/src/apialiases.jl:2026 [inlined]
 [4] readNamespacedPod(::Kuber.Kubernetes.CoreV1Api, ::String, ::String) at /root/.julia/packages/Kuber/UfXri/src/apialiases.jl:2026
 [5] macro expansion at /root/.julia/packages/Kuber/UfXri/src/simpleapi.jl:79 [inlined]
 [6] macro expansion at /root/.julia/packages/Retry/vS1bg/src/repeat_try.jl:192 [inlined]
 [7] get(::Kuber.KuberContext, ::Symbol, ::String, ::Nothing; max_tries::Int64, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /root/.julia/packages/Kuber/UfXri/src/simpleapi.jl:78
 [8] get at /root/.julia/packages/Kuber/UfXri/src/simpleapi.jl:62 [inlined] (repeats 2 times)
 [9] self_pod at /root/.julia/packages/K8sClusterManagers/eAv58/src/native_driver.jl:19 [inlined]
 [10] default_pod(::Kuber.KuberContext, ::Int64, ::Cmd, ::String; image::Nothing, memory::String, cpu::String, serviceAccountName::Nothing, base_obj::Kuber.Kubernetes.IoK8sApiCoreV1Pod, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /root/.julia/packages/K8sClusterManagers/eAv58/src/native_driver.jl:35
 [11] (::K8sClusterManagers.var"#3#4"{typeof(identity),String,Cmd,Base.Iterators.Pairs{Symbol,Any,NTuple{4,Symbol},NamedTuple{(:image, :memory, :cpu, :serviceAccountName),Tuple{Nothing,String,String,Nothing}}},Kuber.KuberContext})(::Int64) at ./none:0
 [12] iterate at ./generator.jl:47 [inlined]
 [13] grow_to!(::Dict{Any,Any}, ::Base.Generator{UnitRange{Int64},K8sClusterManagers.var"#3#4"{typeof(identity),String,Cmd,Base.Iterators.Pairs{Symbol,Any,NTuple{4,Symbol},NamedTuple{(:image, :memory, :cpu, :serviceAccountName),Tuple{Nothing,String,String,Nothing}}},Kuber.KuberContext}}) at ./dict.jl:139
 [14] dict_with_eltype at ./abstractdict.jl:540 [inlined]
 [15] Dict(::Base.Generator{UnitRange{Int64},K8sClusterManagers.var"#3#4"{typeof(identity),String,Cmd,Base.Iterators.Pairs{Symbol,Any,NTuple{4,Symbol},NamedTuple{(:image, :memory, :cpu, :serviceAccountName),Tuple{Nothing,String,String,Nothing}}},Kuber.KuberContext}}) at ./dict.jl:128
 [16] #default_pods_and_context#2 at /root/.julia/packages/K8sClusterManagers/eAv58/src/native_driver.jl:67 [inlined]
 [17] K8sNativeManager(::UnitRange{Int64}, ::String, ::Cmd; configure::Function, namespace::String, retry_seconds::Int64, kwargs::Base.Iterators.Pairs{Symbol,Any,NTuple{4,Symbol},NamedTuple{(:image, :memory, :cpu, :serviceAccountName),Tuple{Nothing,String,String,Nothing}}}) at /root/.julia/packages/K8sClusterManagers/eAv58/src/native_driver.jl:82
 [18] addprocs_pod(::Int64; driver_name::String, configure::Function, namespace::String, image::Nothing, serviceAccountName::Nothing, memory::String, cpu::String, retry_seconds::Int64, exename::Cmd, exeflags::Cmd, params::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /root/.julia/packages/K8sClusterManagers/eAv58/src/native_driver.jl:193
 [19] top-level scope at REPL[4]:1
caused by [exception 1]
UndefVarError: readPod not defined
Stacktrace:
 [1] top-level scope
 [2] eval at ./boot.jl:331 [inlined]
 [3] eval at /root/.julia/packages/Kuber/UfXri/src/Kuber.jl:1 [inlined]
 [4] get(::Kuber.KuberContext, ::Symbol, ::String, ::Nothing; max_tries::Int64, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /root/.julia/packages/Kuber/UfXri/src/simpleapi.jl:66
 [5] get at /root/.julia/packages/Kuber/UfXri/src/simpleapi.jl:62 [inlined] (repeats 2 times)
 [6] self_pod at /root/.julia/packages/K8sClusterManagers/eAv58/src/native_driver.jl:19 [inlined]
 [7] default_pod(::Kuber.KuberContext, ::Int64, ::Cmd, ::String; image::Nothing, memory::String, cpu::String, serviceAccountName::Nothing, base_obj::Kuber.Kubernetes.IoK8sApiCoreV1Pod, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /root/.julia/packages/K8sClusterManagers/eAv58/src/native_driver.jl:35
 [8] (::K8sClusterManagers.var"#3#4"{typeof(identity),String,Cmd,Base.Iterators.Pairs{Symbol,Any,NTuple{4,Symbol},NamedTuple{(:image, :memory, :cpu, :serviceAccountName),Tuple{Nothing,String,String,Nothing}}},Kuber.KuberContext})(::Int64) at ./none:0
 [9] iterate at ./generator.jl:47 [inlined]
 [10] grow_to!(::Dict{Any,Any}, ::Base.Generator{UnitRange{Int64},K8sClusterManagers.var"#3#4"{typeof(identity),String,Cmd,Base.Iterators.Pairs{Symbol,Any,NTuple{4,Symbol},NamedTuple{(:image, :memory, :cpu, :serviceAccountName),Tuple{Nothing,String,String,Nothing}}},Kuber.KuberContext}}) at ./dict.jl:139
 [11] dict_with_eltype at ./abstractdict.jl:540 [inlined]
 [12] Dict(::Base.Generator{UnitRange{Int64},K8sClusterManagers.var"#3#4"{typeof(identity),String,Cmd,Base.Iterators.Pairs{Symbol,Any,NTuple{4,Symbol},NamedTuple{(:image, :memory, :cpu, :serviceAccountName),Tuple{Nothing,String,String,Nothing}}},Kuber.KuberContext}}) at ./dict.jl:128
 [13] #default_pods_and_context#2 at /root/.julia/packages/K8sClusterManagers/eAv58/src/native_driver.jl:67 [inlined]
 [14] K8sNativeManager(::UnitRange{Int64}, ::String, ::Cmd; configure::Function, namespace::String, retry_seconds::Int64, kwargs::Base.Iterators.Pairs{Symbol,Any,NTuple{4,Symbol},NamedTuple{(:image, :memory, :cpu, :serviceAccountName),Tuple{Nothing,String,String,Nothing}}}) at /root/.julia/packages/K8sClusterManagers/eAv58/src/native_driver.jl:82
 [15] addprocs_pod(::Int64; driver_name::String, configure::Function, namespace::String, image::Nothing, serviceAccountName::Nothing, memory::String, cpu::String, retry_seconds::Int64, exename::Cmd, exeflags::Cmd, params::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /root/.julia/packages/K8sClusterManagers/eAv58/src/native_driver.jl:193
 [16] top-level scope at REPL[4]:1

Setting the namespace to "project-ondawerks" works properly

soulshake commented 3 years ago

This isn't a K8sClusterManagers.jl error, it's from Kubernetes:

{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"driver-2021-03-19--17-58-47-kpqnt\" 
is forbidden: User \"system:serviceaccount:project-ondawerks:ondawerks-service-account\" cannot get resource \"pods\" 
in API group \"\" in the namespace \"ondawerks\"","reason":"Forbidden","details":
{"name":"driver-2021-03-19--17-58-47-kpqnt","kind":"pods"},"code":403}

Though I suppose K8sClusterManagers.jl could go out of its way to check that the namespace actually exists, if some interested party wanted to add such a check :)

Edit: Sorry, I'm so used to visually drilling into stack traces that I forget :sweat_smile: I suppose what you're probably suggesting is to merely handle the exception by surfacing the error from k8s itself, which would probably be a lot easier for users to digest indeed.

omus commented 3 years ago

There are a couple of things that can be done here to improve the situation:

  1. Fix UndefVarError: readPod not defined. This error makes it appear that there is an internal failure happening
  2. Use pretty printing on the JSON output to make the actual error message easier to discover and read
  3. Fix the double stack trace. I'm not sure why the stack trace was emitted twice here but it's also just adding noise

Maybe (probably not) we could have K8sClusterManagers.jl catch the exception and reformat the exception as a proper Julia exception but that seems overkill at this point.

omus commented 3 years ago

Issue has been resolved. There were several parts to this:

  1. The readPod error message was from Kuber.jl which was dropped in #34
  2. Pretty printing the JSON message is no longer required as this was also from Kuber.jl which is no longer used.
  3. The double stack trace was addressed in #18.

Additionally, we now use kubectl_jll for interacting with Kubernetes and with our internal wrappers we get fairly nice to read KubeErrors. Finally, specifying the namespace is no longer required as the current namespace is determined automatically (#22)