kube-state-metrics support for oidc authentication providers

doktoroblivion commented 6 years ago

/kind feature

What happened: Attempting to deploy kube-state-metrics container into our environment the function threw and error, which at first I took to be a BUG, but am thinking its probably a FEATURE request now after going through much of the code. The message received when launching the container:

$ kube-state-metrics --port=8080 --telemetry-port=8081 --kubeconfig=/home/egriffin/.bluemix/plugins/container-service/clusters/apiconnect-dev-1/kube-config-dal10-apiconnect-dev-1.yml
I0430 13:18:38.117467   17878 main.go:225] Using default collectors
I0430 13:18:38.117574   17878 main.go:239] Using all namespace
F0430 13:18:38.120900   17878 main.go:248] Failed to create client: No Auth Provider found for name "oidc"

What you expected to happen: I expected the container to just launch, but I see there is no reference to the client-go/plugin/pkg/client/auth/oidc plugin in the code (main.go) or called routines within client-go.

How to reproduce it (as minimally and precisely as possible): Using a Bluemix or similar environment build and deploy the container within it using the provided configuration file, which will contain something like the following:

apiVersion: v1
clusters:
- cluster:
    certificate-authority: ca-dal10-apiconnect-dev-1.pem
    server: https://***.**.***.***:22998
  name: apiconnect-dev-1
contexts:
- context:
    cluster: apiconnect-dev-1
    namespace: default
    user: *******@us.ibm.com
  name: apiconnect-dev-1
current-context: apiconnect-dev-1
kind: Config
preferences: {}
users:
- name: *******@us.ibm.com
  user:
    auth-provider:
      config:
        client-id: bx
        client-secret: bx
        id-token: ******************
        idp-issuer-url: https://iam.ng.bluemix.net/kubernetes
        refresh-token: ********************
      name: oidc

Anything else we need to know?:

Environment:

Kubernetes version (use kubectl version):

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:55:54Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.11-2+fa6873d3e386d7", GitCommit:"fa6873d3e386d7ead42923b24aea3b76e74395a3", GitTreeState:"clean", BuildDate:"2018-04-17T08:10:40Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Kube-state-metrics image version
```
v1.3.0 / 2018-04-04
```

brancz commented 6 years ago

Bluemix is Kubernetes based, no? Why don't you use in cluster mode?

Other than that I wonder why this happens, as we use the same code as Kubernetes for this, can you confirm that that kubeconfig actually works for your local client?

doktoroblivion commented 6 years ago

@brancz Yes, the kubeconfig does work for all other clients (containers) and kubectl commands. I think this is an issue with getting the plugin for iodc to work with the main.go module. I see that when I try to make the container I get an error. Do I have to build the iodc plugin first, then update the Makefile for kube-state-metrics to pull it in? The following occurs if I just try to use the following import in main.go:

package main

import (
    "flag"
    "fmt"
    "log"
    "net"
    "net/http"
    "net/http/pprof"
    "os"
    "sort"
    "strconv"
    "strings"

    "github.com/golang/glog"
    "github.com/openshift/origin/pkg/util/proc"
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
    "github.com/spf13/pflag"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    clientset "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/tools/clientcmd"
    _ "k8s.io/client-go/plugin/pkg/client/auth"

    kcollectors "k8s.io/kube-state-metrics/collectors"
    "k8s.io/kube-state-metrics/version"
)

Then building container:
$ make container
docker run --rm -v "$PWD":/go/src/k8s.io/kube-state-metrics -w /go/src/k8s.io/kube-state-metrics -e GOOS=linux -e GOARCH=amd64 -e CGO_ENABLED=0 golang:1.10.1 go build -ldflags "-s -w -X k8s.io/kube-state-metrics/version.Release=v1.3.0 -X k8s.io/kube-state-metrics/version.Commit=ee86ca3 -X k8s.io/kube-state-metrics/version.BuildDate=2018-05-02T12:15:25Z" -o kube-state-metrics
main.go:39:5: cannot find package "k8s.io/client-go/plugin/pkg/client/auth" in any of:
    /go/src/k8s.io/kube-state-metrics/vendor/k8s.io/client-go/plugin/pkg/client/auth (vendor tree)
    /usr/local/go/src/k8s.io/client-go/plugin/pkg/client/auth (from $GOROOT)
    /go/src/k8s.io/client-go/plugin/pkg/client/auth (from $GOPATH)
Makefile:52: die Regel für Ziel „.container-amd64“ scheiterte
make: *** [.container-amd64] Fehler 1

My GOPATH looks like the following:

egriffin@oblivion:~/velox/go$ echo $GOPATH
/home/egriffin/velox/go
egriffin@oblivion:~/velox/go$ ls src
github.com  gopkg.in  k8s.io
egriffin@oblivion:~/velox/go$ ls src/k8s.io/
api  apimachinery  client-go  kube-openapi  kube-state-metrics

brancz commented 6 years ago

@andyxning this seems like we have a problem with the packages vendored. Could you have a look? I am at KubeCon the entire week.

doktoroblivion commented 6 years ago

Some additional information and now a different issue with compliation.

I decided, per the readme, to try and use dependency management (dep - the official one, not godep as in the readme). I ran the following:

$ dep init
Importing configuration from godep. These are only initial constraints, and are further refined during the solve process.
Detected godep configuration files...
Converting from Godeps.json ...
  Using ^1.0.0 as initial constraint for imported dep github.com/PuerkitoBio/purell
  Trying v1.0.0 (8a29053) as initial lock for imported dep github.com/PuerkitoBio/purell
  Trying * (5bd2802) as initial lock for imported dep github.com/PuerkitoBio/urlesc
  Trying * (4c0e845) as initial lock for imported dep github.com/beorn7/perks
  Trying * (5215b55) as initial lock for imported dep github.com/davecgh/go-spew
  Trying * (89ef8af) as initial lock for imported dep github.com/emicklei/go-restful
  Trying * (73d445a) as initial lock for imported dep github.com/ghodss/yaml
  Trying * (46af16f) as initial lock for imported dep github.com/go-openapi/jsonpointer
  Trying * (13c6e35) as initial lock for imported dep github.com/go-openapi/jsonreference
  Trying * (2433d2f) as initial lock for imported dep github.com/go-openapi/spec
  Trying * (0e04f5e) as initial lock for imported dep github.com/go-openapi/swag
  Trying * (c0656ed) as initial lock for imported dep github.com/gogo/protobuf
  Trying * (44145f0) as initial lock for imported dep github.com/golang/glog
  Trying * (7390af9) as initial lock for imported dep github.com/golang/protobuf
  Trying * (316fb6d) as initial lock for imported dep github.com/google/btree
  Trying * (bbcb9da) as initial lock for imported dep github.com/google/gofuzz
  Using ^0.1.0 as initial constraint for imported dep github.com/googleapis/gnostic
  Trying v0.1.0 (ee43cbb) as initial lock for imported dep github.com/googleapis/gnostic
  Trying * (c1f8028) as initial lock for imported dep github.com/gregjones/httpcache
  Trying * (0a025b7) as initial lock for imported dep github.com/hashicorp/golang-lru
  Trying * (26c6e11) as initial lock for imported dep github.com/howeyc/gopass
  Trying * (6633656) as initial lock for imported dep github.com/imdario/mergo
  Using ^1.0.3 as initial constraint for imported dep github.com/json-iterator/go
  Trying 1.0.3 (6240e1e) as initial lock for imported dep github.com/json-iterator/go
  Using ^1.0.0 as initial constraint for imported dep github.com/juju/ratelimit
  Trying 1.0 (5b9ff86) as initial lock for imported dep github.com/juju/ratelimit
  Trying * (e978125) as initial lock for imported dep github.com/mailru/easyjson
  Trying master (c12348c) as initial lock for imported dep github.com/matttproud/golang_protobuf_extensions
  Trying * (8f127d7) as initial lock for imported dep github.com/openshift/origin
  Using ^2.0.1 as initial constraint for imported dep github.com/peterbourgon/diskv
  Trying v2.0.1 (5f041e8) as initial lock for imported dep github.com/peterbourgon/diskv
  Trying * (8aae34f) as initial lock for imported dep github.com/prometheus/client_golang
  Trying * (fa8ad6f) as initial lock for imported dep github.com/prometheus/client_model
  Trying * (ebdfc6d) as initial lock for imported dep github.com/prometheus/common
  Trying * (abf152e) as initial lock for imported dep github.com/prometheus/procfs
  Trying * (736158d) as initial lock for imported dep github.com/robfig/cron
  Trying * (9ff6c69) as initial lock for imported dep github.com/spf13/pflag
  Trying * (8e06e8d) as initial lock for imported dep golang.org/x/crypto
  Trying * (e90d6d0) as initial lock for imported dep golang.org/x/net
  Trying * (30de6d1) as initial lock for imported dep golang.org/x/sys
  Trying * (1e65e9b) as initial lock for imported dep golang.org/x/text
  Using ^0.9.0 as initial constraint for imported dep gopkg.in/inf.v0
  Trying v0.9.0 (3887ee9) as initial lock for imported dep gopkg.in/inf.v0
  Trying * (53feefa) as initial lock for imported dep gopkg.in/yaml.v2
  Trying kubernetes-1.9.1-beta.0 (af4bc15) as initial lock for imported dep k8s.io/api
  Trying kubernetes-1.8.0-rc.1 (9d38e20) as initial lock for imported dep k8s.io/apimachinery
  Using ^6.0.0 as initial constraint for imported dep k8s.io/client-go
  Trying v6.0.0 (78700de) as initial lock for imported dep k8s.io/client-go
  Trying * (abfc5fb) as initial lock for imported dep k8s.io/kube-openapi
  Using ^1.9.0 as initial constraint for imported dep k8s.io/kubernetes
  Trying v1.9.0 (925c127) as initial lock for imported dep k8s.io/kubernetes
  Using kubernetes-1.9.1-beta.0 as constraint for direct dep k8s.io/api
  Locking in kubernetes-1.9.1-beta.0 (af4bc15) for direct dep k8s.io/api
  Locking in v1.1 (e790cca) for transitive dep github.com/pborman/uuid
  Locking in v1.0.5 (c155da1) for transitive dep github.com/Sirupsen/logrus
  Locking in  (fa8ad6f) for direct dep github.com/prometheus/client_model
  Locking in master (44e46d2) for transitive dep github.com/fatih/camelcase
  Locking in master (761b3ff) for transitive dep k8s.io/apiserver
  Locking in  (8aae34f) for direct dep github.com/prometheus/client_golang
  Locking in master (6881fee) for transitive dep golang.org/x/oauth2
  Locking in  (e90d6d0) for direct dep golang.org/x/net
  Locking in v2.0.0 (cadec56) for transitive dep github.com/russross/blackfriday
  Locking in master (d6e3b33) for transitive dep github.com/Azure/go-ansiterm
  Locking in v0.21.0 (29f476f) for transitive dep cloud.google.com/go
  Locking in master (53be0d3) for transitive dep github.com/petar/GoLLRB
  Locking in v1.0 (76626ae) for transitive dep github.com/inconshreveable/mousetrap
  Locking in master (962c31a) for transitive dep k8s.io/apiextensions-apiserver
  Locking in master (e9091a2) for transitive dep github.com/MakeNowJust/heredoc
  Locking in master (242a24b) for transitive dep github.com/gophercloud/gophercloud
  Locking in  (0b12d6b5) for transitive dep github.com/jmespath/go-jmespath
  Locking in v0.1.2 (ec4a0fe) for transitive dep gopkg.in/warnings.v0
  Locking in v3.2.0 (06ea103) for transitive dep github.com/dgrijalva/jwt-go
  Locking in  (ebdfc6d) for direct dep github.com/prometheus/common
  Locking in v3.0.0 (afac545) for transitive dep github.com/evanphx/json-patch
  Locking in v10.8.0 (8d642cd) for transitive dep github.com/Azure/go-autorest
  Locking in v2.6.2 (48294d9) for transitive dep github.com/docker/distribution
  Using ^2.9.0 as constraint for direct dep k8s.io/helm
  Locking in v2.9.0 (f6025bb) for direct dep k8s.io/helm
  Locking in master (256737a) for transitive dep vbom.ml/util
  Locking in v0.0.2 (a1f051b) for transitive dep github.com/spf13/cobra
  Using kubernetes-1.8.0-rc.1 as constraint for direct dep k8s.io/apimachinery
  Locking in kubernetes-1.8.0-rc.1 (9d38e20) for direct dep k8s.io/apimachinery
  Locking in v1.13.39 (236a3fc) for transitive dep github.com/aws/aws-sdk-go
  Locking in v1.2.2 (f02745a) for transitive dep gopkg.in/gcfg.v1
  Locking in v1.13.1 (092cba3) for transitive dep github.com/docker/docker
  Locking in v1.36.0 (6529cf7) for transitive dep github.com/go-ini/ini
  Locking in master (66deaeb) for transitive dep github.com/golang/groupcache
  Locking in v0.3.0 (3ede32e) for transitive dep github.com/docker/go-connections
  Locking in  (44145f0) for direct dep github.com/golang/glog
  Locking in v0.3.3 (47565b4) for transitive dep github.com/docker/go-units
  Locking in master (d6023ce) for transitive dep github.com/exponent-io/jsonpath
  Locking in master (bc6354c) for transitive dep github.com/docker/spdystream
  Locking in  (736158d) for direct dep github.com/robfig/cron
  Locking in v1.0.0 (150dc57) for transitive dep google.golang.org/appengine
  Locking in master (86672fc) for transitive dep github.com/shurcooL/sanitized_anchor_name
  Locking in master (258e2a2) for transitive dep k8s.io/utils
  Locking in master (ad45545) for transitive dep github.com/mitchellh/go-wordwrap
Old vendor backed up to /home/egriffin/go/src/k8s.io/kube-state-metrics/_vendor-20180502155517

This appeared to generate the appropriate toml file:

Gopkg.lock     Gopkg.toml

Then I attempted to make the container once again and ran into essentially the same type of errors.

$ make container
docker run --rm -v "$PWD":/go/src/k8s.io/kube-state-metrics -w /go/src/k8s.io/kube-state-metrics -e GOOS=linux -e GOARCH=amd64 -e CGO_ENABLED=0 golang:1.10.1 go build -ldflags "-s -w -X k8s.io/kube-state-metrics/version.Release=v1.3.0 -X k8s.io/kube-state-metrics/version.Commit=1e935b7 -X k8s.io/kube-state-metrics/version.BuildDate=2018-05-02T20:13:46Z" -o kube-state-metrics
main.go:32:2: cannot find package "github.com/openshift/origin/pkg/util/proc" in any of:
    /go/src/k8s.io/kube-state-metrics/vendor/github.com/openshift/origin/pkg/util/proc (vendor tree)
    /usr/local/go/src/github.com/openshift/origin/pkg/util/proc (from $GOROOT)
    /go/src/github.com/openshift/origin/pkg/util/proc (from $GOPATH)
main.go:34:2: cannot find package "github.com/prometheus/client_golang/prometheus/promhttp" in any of:
    /go/src/k8s.io/kube-state-metrics/vendor/github.com/prometheus/client_golang/prometheus/promhttp (vendor tree)
    /usr/local/go/src/github.com/prometheus/client_golang/prometheus/promhttp (from $GOROOT)
    /go/src/github.com/prometheus/client_golang/prometheus/promhttp (from $GOPATH)
Makefile:52: die Regel für Ziel „.container-amd64“ scheiterte
make: *** [.container-amd64] Fehler 1

andyxning commented 6 years ago

Will take a look at this later.

doktoroblivion commented 6 years ago

I have finally got this to work, but had to do the following which is not part of the documentation and the executable get's placed into the src directory instead of the $GOPATH/bin directory, where I think it is expected to go.


- install godep in /usr/local/bin or /usr/local/go/bin
- go get golang.org/x/oauth2
- from ~/go/src/k8s.io/kube-state-metrics
  - godep restore
  - edit main.go, add '_ "k8s.io/client-go/plugin/pkg/client/auth/oidc"' to import
  - rm -fr Godeps/
  - godep save ./...
  - make container
  - ./kube-state-metrics --port=8080 --telemetry-port=8081 --kubeconfig=<config file>

doktoroblivion commented 6 years ago

I think if we agree to doc this somehow you can close this issue.

andyxning commented 6 years ago

I think if we agree to doc this somehow you can close this issue.

Actually, this is the normal procedure to add new dependencies to go projects using godep. :) If you still think that should be documented somewhere in the readme. PR is welcomed. :)

kube-state-metrics should add support for all possible auth providers. I.e., we should add _ "k8s.io/client-go/plugin/pkg/client/auth to the main.go file and update the godeps.

@doktoroblivion Do you have some time to help us with this?

The change is somewhat like https://github.com/appscode/stash/pull/331

andyxning commented 6 years ago

@brancz

Bluemix is Kubernetes based, no? Why don't you use in cluster mode? Other than that I wonder why this happens, as we use the same code as Kubernetes for this, can you confirm that that kubeconfig actually works for your local client?

It seems that we do not import k8s.io/client-go/plugin/pkg/client/auth directly or indirectly in kube-state-metrics. Thus it does not support auth providers. I am not quite sure whether in cluster running can help kube-state-metrics avoid this problem. But even it is for in cluster running, we should also add support for all auth providers for kube-state-metrics running outside of a Kubernetes cluster.

doktoroblivion commented 6 years ago

@andyxning sure thing. However, I confirm that the change proposed in apscode/stash#331 will not work at the moment due to a problem with the azure package, so I am just PR'ing the oidc auth package. it throws the following error which I don't have time to research or fix.

$ make container
docker run --rm -v "$PWD":/go/src/k8s.io/kube-state-metrics -w /go/src/k8s.io/kube-state-metrics -e GOOS=linux -e GOARCH=amd64 -e CGO_ENABLED=0 golang:1.10.1 go build -ldflags "-s -w -X k8s.io/kube-state-metrics/pkg/version.Release=v1.3.0 -X k8s.io/kube-state-metrics/pkg/version.Commit=db75549 -X k8s.io/kube-state-metrics/pkg/version.BuildDate=2018-05-04T17:43:09Z" -o kube-state-metrics
# k8s.io/kube-state-metrics/vendor/k8s.io/client-go/plugin/pkg/client/auth/azure
vendor/k8s.io/client-go/plugin/pkg/client/auth/azure/azure.go:300:8: cannot use spt.Token (type func() adal.Token) as type adal.Token in field value
Makefile:52: recipe for target '.container-amd64' failed
make: *** [.container-amd64] Error 2

main.go will have to be updated and perhaps a compile or linter check put in to make sure plugin packages work and follow the rules (what ever they are supposed to be).

doktoroblivion commented 6 years ago

PR is opened and delivered. Again, it only pulls in oidc, since azure has an issue and I cannot verify the others, oidc does work.

kubernetes / kube-state-metrics

kube-state-metrics support for oidc authentication providers #449