search

openshift / origin

Conformance test suite for OpenShift
http://www.openshift.org
Apache License 2.0
8.48k stars 4.7k forks source link

oc cluster up --service-catalog --version=latest failing after 1.7 rebase: 'no kind is registered for the type v1.Endpoints' #15381

Closed jwmatthews closed 7 years ago

jwmatthews commented 7 years ago

After the rebase to 1.7 I'm unable to do: oc cluster up --service-catalog --version=latest Running from latest master source built oc/images

Assuming error related to:

Could not construct reference to: '&v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"service-catalog-controller-manager", GenerateName:"", Namespace:"service-catalog", SelfLink:"/api/v1/namespaces/service-catalog/endpoints/service-catalog-controller-manager", UID:"68abb3ae-6d7c-11e7-83f1-64006a559e97", ResourceVersion:"1460", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{sec:63636173475, nsec:0, loc:(*time.Location)(0x282f6a0)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{"control-plane.alpha.kubernetes.io/leader":"{\"holderIdentity\":\"controller-manager-1059462124-tkpr6-external-service-catalog-controller\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2017-07-20T18:51:15Z\",\"renewTime\":\"2017-07-20T18:51:48Z\",\"leaderTransitions\":0}"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:""}, Subsets:[]v1.EndpointSubset(nil)}' due to: 'no kind is registered for the type v1.Endpoints'.
Version

$ ./oc version oc v3.6.0-rc.0+7a48deb-163 kubernetes v1.7.0+695f48a16f features: Basic-Auth

Server https://127.0.0.1:8443 kubernetes v1.7.0+695f48a16f

Steps To Reproduce
  1. checkout master
  2. make build ; hack/build-local-images.py origin;
  3. cd _output/local/bin/linux/amd64/
  4. ./oc cluster up --service-catalog --version=latest
Current Result

./oc cluster up --service-catalog --version=latest Starting OpenShift using openshift/origin:latest ... -- Checking OpenShift client ... OK -- Checking Docker client ... OK -- Checking Docker version ... OK -- Checking for existing OpenShift container ... OK -- Checking for openshift/origin:latest image ... OK -- Checking Docker daemon configuration ... OK -- Checking for available ports ... WARNING: Binding DNS on port 8053 instead of 53, which may not be resolvable from all clients. -- Checking type of volume mount ... Using nsenter mounter for OpenShift volumes -- Creating host directories ... OK -- Finding server IP ... Using 127.0.0.1 as the server IP -- Checking service catalog version requirements ... OK -- Starting OpenShift container ... Creating initial OpenShift configuration Starting OpenShift using container 'origin' Waiting for API server to start listening OpenShift server started -- Adding default OAuthClient redirect URIs ... OK -- Installing registry ... OK -- Installing router ... OK -- Importing image streams ... OK -- Importing templates ... OK -- Installing service catalog ... FAIL Error: failed to register broker with service catalog: timed out waiting for the condition

Expected Result
Additional Information

$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE openshift/origin latest 223e695b1288 9 minutes ago 1.43 GB docker.io/openshift/origin-deployer latest 04f5bf1f6310 About an hour ago 1.051 GB docker.io/openshift/origin-docker-registry latest e2d4ad82c4c6 About an hour ago 1.135 GB docker.io/openshift/origin-haproxy-router latest cb3ae0c60b2a About an hour ago 1.072 GB docker.io/openshift/origin latest d1fb1ef10689 About an hour ago 1.051 GB docker.io/openshift/origin-pod latest e1070d845e3f About an hour ago 213.2 MB quay.io/kubernetes-service-catalog/apiserver canary 1d49ecc2e4f1 14 hours ago 166 MB quay.io/kubernetes-service-catalog/controller-manager canary d0716d69eeae 14 hours ago 164.1 MB quay.io/coreos/etcd latest 498ffffcfd05 24 hours ago 35.74 MB registry.ops.openshift.com/openshift3/ose-docker-registry v3.6 da5986fb25fb 2 days ago 1.088 GB registry.ops.openshift.com/openshift3/ose-pod v3.6 4b5dbee32711 2 days ago 205.8 MB registry.access.stage.redhat.com/openshift3/ansible-service-broker latest a3bc6dfb8ca7 9 days ago 595.5 MB registry.access.redhat.com/rhel7/etcd latest 1c65d1efd289 3 weeks ago 233.4 MB

$ oc get pods NAME READY STATUS RESTARTS AGE apiserver-644637603-r7pfk 2/2 Running 0 2m controller-manager-1059462124-tkpr6 1/1 Running 1 2m

$ oc logs controller-manager-1059462124-tkpr6 I0720 18:51:48.495708 1 controller_manager.go:100] Building k8s kubeconfig I0720 18:51:48.497368 1 controller_manager.go:128] Building service-catalog kubeconfig for url: http://172.30.1.2:80 I0720 18:51:48.497448 1 controller_manager.go:147] Starting http server and mux I0720 18:51:48.497460 1 controller_manager.go:176] Creating event broadcaster I0720 18:51:48.497540 1 controller_manager.go:216] Using namespace service-catalog for leader election lock I0720 18:51:48.497554 1 leaderelection.go:179] attempting to acquire leader lease... E0720 18:51:48.542617 1 event.go:259] Could not construct reference to: '&v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"service-catalog-controller-manager", GenerateName:"", Namespace:"service-catalog", SelfLink:"/api/v1/namespaces/service-catalog/endpoints/service-catalog-controller-manager", UID:"68abb3ae-6d7c-11e7-83f1-64006a559e97", ResourceVersion:"1460", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{sec:63636173475, nsec:0, loc:(time.Location)(0x282f6a0)}}, DeletionTimestamp:(v1.Time)(nil), DeletionGracePeriodSeconds:(int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{"control-plane.alpha.kubernetes.io/leader":"{\"holderIdentity\":\"controller-manager-1059462124-tkpr6-external-service-catalog-controller\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2017-07-20T18:51:15Z\",\"renewTime\":\"2017-07-20T18:51:48Z\",\"leaderTransitions\":0}"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:""}, Subsets:[]v1.EndpointSubset(nil)}' due to: 'no kind is registered for the type v1.Endpoints'. Will not report event: 'Normal' 'LeaderElection' 'controller-manager-1059462124-tkpr6-external-service-catalog-controller became leader' I0720 18:51:48.542707 1 leaderelection.go:189] successfully acquired lease service-catalog/service-catalog-controller-manager I0720 18:51:48.542786 1 controller_manager.go:301] Getting available resources I0720 18:51:48.549397 1 controller_manager.go:262] Created client for discovery I0720 18:51:48.551237 1 request.go:1190] body was not decodable (unable to check for Status): Object 'Kind' is missing in '{ "paths": [ "/apis", "/apis/servicecatalog.k8s.io", "/apis/servicecatalog.k8s.io/v1alpha1", "/healthz", "/healthz/etcd", "/healthz/ping", "/healthz/poststarthook/start-service-catalog-apiserver-informers", "/metrics", "/swaggerapi/", "/version" ] }' I0720 18:51:48.553146 1 controller_manager.go:279] Resource: &v1.APIResourceList{TypeMeta:v1.TypeMeta{Kind:"APIResourceList", APIVersion:"v1"}, GroupVersion:"servicecatalog.k8s.io/v1alpha1", APIResources:[]v1.APIResource{v1.APIResource{Name:"bindings", Namespaced:true, Kind:"Binding", Verbs:v1.Verbs{"create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"}, ShortNames:[]string(nil)}, v1.APIResource{Name:"bindings/status", Namespaced:true, Kind:"Binding", Verbs:v1.Verbs{"create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"}, ShortNames:[]string(nil)}, v1.APIResource{Name:"brokers", Namespaced:false, Kind:"Broker", Verbs:v1.Verbs{"create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"}, ShortNames:[]string(nil)}, v1.APIResource{Name:"brokers/status", Namespaced:false, Kind:"Broker", Verbs:v1.Verbs{"create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"}, ShortNames:[]string(nil)}, v1.APIResource{Name:"instances", Namespaced:true, Kind:"Instance", Verbs:v1.Verbs{"create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"}, ShortNames:[]string(nil)}, v1.APIResource{Name:"instances/status", Namespaced:true, Kind:"Instance", Verbs:v1.Verbs{"create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"}, ShortNames:[]string(nil)}, v1.APIResource{Name:"serviceclasses", Namespaced:false, Kind:"ServiceClass", Verbs:v1.Verbs{"create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"}, ShortNames:[]string(nil)}}} I0720 18:51:48.553642 1 controller_manager.go:315] Creating shared informers; resync interval: 5m0s I0720 18:51:48.553672 1 controller_manager.go:325] Creating controller; broker relist interval: 5m0s I0720 18:51:48.553921 1 controller_manager.go:342] Running controller I0720 18:51:48.553930 1 controller_manager.go:345] Starting shared informers I0720 18:51:48.554203 1 reflector.go:198] Starting reflector v1alpha1.Binding (5m0s) from github.com/kubernetes-incubator/service-catalog/pkg/client/informers_generated/externalversions/factory.go:61 I0720 18:51:48.554229 1 reflector.go:236] Listing and watching v1alpha1.Binding from github.com/kubernetes-incubator/service-catalog/pkg/client/informers_generated/externalversions/factory.go:61 I0720 18:51:48.554244 1 reflector.go:198] Starting reflector v1alpha1.ServiceClass (5m0s) from github.com/kubernetes-incubator/service-catalog/pkg/client/informers_generated/externalversions/factory.go:61 I0720 18:51:48.554262 1 reflector.go:236] Listing and watching v1alpha1.ServiceClass from github.com/kubernetes-incubator/service-catalog/pkg/client/informers_generated/externalversions/factory.go:61 I0720 18:51:48.554570 1 reflector.go:198] Starting reflector v1alpha1.Instance (5m0s) from github.com/kubernetes-incubator/service-catalog/pkg/client/informers_generated/externalversions/factory.go:61 I0720 18:51:48.554603 1 controller.go:152] Starting service-catalog controller I0720 18:51:48.554605 1 reflector.go:236] Listing and watching v1alpha1.Instance from github.com/kubernetes-incubator/service-catalog/pkg/client/informers_generated/externalversions/factory.go:61 I0720 18:51:48.554858 1 reflector.go:198] Starting reflector v1alpha1.Broker (5m0s) from github.com/kubernetes-incubator/service-catalog/pkg/client/informers_generated/externalversions/factory.go:61 I0720 18:51:48.554878 1 reflector.go:236] Listing and watching *v1alpha1.Broker from github.com/kubernetes-incubator/service-catalog/pkg/client/informers_generated/externalversions/factory.go:61 I0720 18:51:50.550907 1 leaderelection.go:204] succesfully renewed lease service-catalog/service-catalog-controller-manager I0720 18:51:52.558803 1 leaderelection.go:204] succesfully renewed lease service-catalog/service-catalog-controller-manager

oc logs apiserver-644637603-r7pfk -c apiserver https://gist.github.com/jwmatthews/16969850abdd9c451ab74d6ffe154758

jwmatthews commented 7 years ago

@bparees FYI

bparees commented 7 years ago

WARNING: Binding DNS on port 8053 instead of 53, which may not be resolvable from all clients.

what happens if you fix that?

bparees commented 7 years ago

also cc @deads2k

bparees commented 7 years ago

also cc @pmorie since the issue here is in the SC image trying to call openshift apis.

csrwng commented 7 years ago

what happens if you fix that?

I don't get that message on my Mac, but still fails the same way. That's not related. The main issue seems to be that the controller manager thinks that 'no kind is registered for the type v1.Endpoints'

bparees commented 7 years ago

yeah i reproduced it, haven't gotten a chance to debug into it, but @pmorie and @deads2k should know what is going on just by seeing the logs i would guess.

csrwng commented 7 years ago

I do have a related fix, though ... on the Mac socat was dying even before cluster up got the part where it was waiting for the service catalog. https://github.com/openshift/origin/pull/15386

bparees commented 7 years ago

I see endpoints under /api/v1

{
      "name": "endpoints",
      "singularName": "",
      "namespaced": true,
      "kind": "Endpoints",
      "verbs": [
        "create",
        "delete",
        "deletecollection",
        "get",
        "list",
        "patch",
        "update",
        "watch"
      ],
      "shortNames": [
        "ep"
      ]
    },

so this is either broken aggregation, broken compatibility in 3.7, or broken discovery in the SC controller-manager image.

deads2k commented 7 years ago

Just so I'm understanding the source of the error. The servicecatalog's controller has started failing on an inability to create an object reference.

"due to: 'no kind is registered for the type v1.Endpoints'. " suggests that its a scheme registration problem, not an API compatibility one. The rebase didn't touch any code under cmd/service-catalog.

bparees commented 7 years ago

@deads2k i'm able to oc cluster up using the current "latest" docker.io/openshift/origin-service-catalog:latest image on a 3.6-rc.0 cluster.

I am not able to do so on a 3.7 cluster (same SC image).

bparees commented 7 years ago

that said, on 3.7 the issue is slightly different with the latest image than what is seen above. We are failing to register the template broker w/ the servicecatalg.k8s.io api endpoint:

I0721 13:41:22.099758   23441 request.go:991] Request Body: {"apiVersion":"servi
cecatalog.k8s.io/v1alpha1","kind":"Broker","metadata":{"name":"template-broker"}
,"spec":{"url":"https://kubernetes.default.svc:443/brokers/template.openshift.io
"}}
I0721 13:41:22.099834   23441 round_trippers.go:383] POST https://127.0.0.1:8443
/apis/servicecatalog.k8s.io/v1alpha1/brokers
I0721 13:41:22.099846   23441 round_trippers.go:390] Request Headers:
I0721 13:41:22.099856   23441 round_trippers.go:393]     Accept: application/jso
n
I0721 13:41:22.099866   23441 round_trippers.go:393]     Content-Type: applicati
on/json
I0721 13:41:22.099875   23441 round_trippers.go:393]     User-Agent: oc/v1.7.0+6
95f48a16f (linux/amd64) kubernetes/695b561
I0721 13:41:22.100778   23441 round_trippers.go:408] Response Status: 404 Not Fo
und in 0 milliseconds
I0721 13:41:22.100798   23441 round_trippers.go:411] Response Headers:
I0721 13:41:22.100807   23441 round_trippers.go:414]     Content-Type: text/plai
n; charset=utf-8
I0721 13:41:22.100816   23441 round_trippers.go:414]     X-Content-Type-Options:
 nosniff
I0721 13:41:22.100824   23441 round_trippers.go:414]     Content-Length: 19
I0721 13:41:22.100832   23441 round_trippers.go:414]     Date: Fri, 21 Jul 2017 
17:41:22 GMT
I0721 13:41:22.100840   23441 round_trippers.go:414]     Cache-Control: no-store
I0721 13:41:22.100863   23441 request.go:991] Response Body: 404 page not found
I0721 13:41:22.100897   23441 servicecatalog.go:179] retrying registration after error the server could not find the requested resource (post brokers.servicecatalog.k8s.io)

The service catalog api server is having some permissions issues:

I0721 17:40:52.119497       1 request.go:991] Request Body: {"kind":"SubjectAcce
ssReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimes
tamp":null},"spec":{"nonResourceAttributes":{"path":"/","verb":"get"},"user":"sy
stem:anonymous","group":["system:unauthenticated"]},"status":{"allowed":false}}

I0721 17:40:52.126922       1 request.go:991] Response Body: {"kind":"SubjectAcc
essReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTime
stamp":null},"spec":{"nonResourceAttributes":{"path":"/","verb":"get"},"user":"s
ystem:anonymous","group":["system:unauthenticated"]},"status":{"allowed":false,"
reason":"User \"system:anonymous\" cannot \"get\" on \"/\""}}
bparees commented 7 years ago

sent you guys a log zip

bparees commented 7 years ago

curl -k https://localhost:8443/apis/ shows the SC api:

   {
      "name": "servicecatalog.k8s.io",
      "versions": [
        {
          "groupVersion": "servicecatalog.k8s.io/v1alpha1",
          "version": "v1alpha1"
        }
      ],
      "preferredVersion": {
        "groupVersion": "servicecatalog.k8s.io/v1alpha1",
        "version": "v1alpha1"
      },
      "serverAddressByClientCIDRs": null
    }

but i can't curl that api:

$ curl -k https://localhost:8443/apis/servicecatalog.k8s.io
404 page not found

I can curl other apis:

$ curl -k https://localhost:8443/apis/user.openshift.io
{
  "kind": "APIGroup",
  "apiVersion": "v1",
  "name": "user.openshift.io",
  "versions": [
    {
      "groupVersion": "user.openshift.io/v1",
      "version": "v1"
    }
  ],
  "preferredVersion": {
    "groupVersion": "user.openshift.io/v1",
    "version": "v1"
  },
  "serverAddressByClientCIDRs": null
}

@deads2k is this an issue w/ enabling v1alpha1 apis?

deads2k commented 7 years ago

Looks like an aggregator issue post-rebase.

deads2k commented 7 years ago

Reminder for myself. Looks like I've got a missing role in kube-system which would stop proper authentication at the UAS.