search

paralin / openshift-under-kubernetes

A full system for deploying and managing Openshift Origin under an existing Kubernetes cluster.
Apache License 2.0
16 stars 5 forks source link

OpenShift doesn't run properly #4

Open dmytroleonenko opened 8 years ago

dmytroleonenko commented 8 years ago

AWS. CoreOS kubernetes cluster with k8s 1.4.0 created using kube-aws. 3 workers in 3 AZs.

[09:55]$ openshift-under-kubernetes --config config/kubeconfig --secure deploy  --create-volume --load-balancer --server-key credentials/apiserver-key.pem
Loading kube config...
Checking connectivity...
Currently there are 2 namespaces in the cluster.
Everything looks good, proceeding.
()
Collecting some initial cluster info...
list index out of range
It looks like we do not have the openshift-origin namespace, and do not have a working ReplicationController.
Currently I do not consider this a complete OpenShift deployment.
()
Preparing to execute deploy...
Deploy temp dir: /var/folders/pj/clwv46wd5bj8d26by56nvwhw0000gn/T/tmpz2BW3l
Creating openshift-deploy namespace...
Creating openshift-origin namespace...
Will use load balancer type service.
Creating 'openshift' service...
Waiting for service load balancer IP to be allocated...
External OpenShift IP: xxx.us-east-1.elb.amazonaws.com
Internal OpenShift IP: 10.3.0.203
Creating create-config-script secret...
Creating kubeconfig secret...
Generating openshift config via cluster...
Creating config generation pod...
Waiting for generate-config pod to finish...
Checking logs...
Retreived config bundle successfully.
Generated updated master-config.yaml.
Do you want to edit master-config.yaml? [y/N]:
Creating openshift-config secret...
Creating etcd service...
Creating etcd controller...
Waiting for etcd pod to be created...
Waiting for etcd-dcstc pod to start...
Creating openshift replication controller...
Waiting for openshift pod to be created...
Waiting for openshift-qiswa pod to start...
()
 == OpenShift Deployed ==
External IP: xxx.us-east-1.elb.amazonaws.com
Currently there are 4 namespaces in the cluster.
Cleaning up old openshift-deploy namespace.
Deleting openshift-deploy namespace...
Waiting for openshift-deploy to terminate...

OpenShift image tag: latest (not v.1.4.0-alpha0) but all of them produce the same result

$ kubectl --kubeconfig=kubeconfig --namespace=openshift-origin logs -f openshift-qiswa
W1013 06:56:32.328305       1 start_master.go:276] Warning: assetConfig.loggingPublicURL: Invalid value: "": required to view aggregated container logs in the console, master start will continue.
W1013 06:56:32.328378       1 start_master.go:276] Warning: assetConfig.metricsPublicURL: Invalid value: "": required to view cluster metrics in the console, master start will continue.
E1013 06:56:32.344206       1 reflector.go:203] github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/limitranger/admission.go:154: Failed to list *api.LimitRange: Get https://10.3.0.203:443/api/v1/limitranges?resourceVersion=0: dial tcp 10.3.0.203:443: getsockopt: connection refused
E1013 06:56:32.353336       1 reflector.go:214] github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/storageclass/default/admission.go:62: Failed to list *storage.StorageClass: Get https://10.3.0.203:443/apis/storage.k8s.io/v1beta1/storageclasses?resourceVersion=0: dial tcp 10.3.0.203:443: getsockopt: connection refused
E1013 06:56:32.357131       1 reflector.go:214] github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/serviceaccount/admission.go:119: Failed to list *api.Secret: Get https://10.3.0.203:443/api/v1/secrets?fieldSelector=type%3Dkubernetes.io%2Fservice-account-token&resourceVersion=0: dial tcp 10.3.0.203:443: getsockopt: connection refused
E1013 06:56:32.357294       1 reflector.go:203] github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/limitranger/admission.go:154: Failed to list *api.LimitRange: Get https://10.3.0.203:443/api/v1/limitranges?resourceVersion=0: dial tcp 10.3.0.203:443: getsockopt: connection refused
E1013 06:56:32.357447       1 reflector.go:214] github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/serviceaccount/admission.go:103: Failed to list *api.ServiceAccount: Get https://10.3.0.203:443/api/v1/serviceaccounts?resourceVersion=0: dial tcp 10.3.0.203:443: getsockopt: connection refused
E1013 06:56:32.357553       1 reflector.go:214] github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/resourcequota/resource_access.go:83: Failed to list *api.ResourceQuota: Get https://10.3.0.203:443/api/v1/resourcequotas?resourceVersion=0: dial tcp 10.3.0.203:443: getsockopt: connection refused
I1013 06:56:32.360205       1 start_master.go:393] Starting master on 0.0.0.0:443 (v1.4.0-alpha.0+c728fe9)
I1013 06:56:32.360228       1 start_master.go:394] Public master address is https://xxx.us-east-1.elb.amazonaws.com:443
I1013 06:56:32.360267       1 start_master.go:398] Using images from "openshift/origin-<component>:v1.4.0-alpha.0"
I1013 06:56:32.362768       1 run_components.go:227] Using default project node label selector:
E1013 06:56:32.366265       1 reflector.go:203] github.com/openshift/origin/pkg/project/cache/cache.go:95: Failed to list *api.Namespace: Get https://10.3.0.203:443/api/v1/namespaces?resourceVersion=0: dial tcp 10.3.0.203:443: getsockopt: connection refused
E1013 06:56:32.366446       1 reflector.go:214] github.com/openshift/origin/pkg/controller/shared/shared_informer.go:104: Failed to list *api.ClusterResourceQuota: Get https://10.3.0.203:443/oapi/v1/clusterresourcequotas?resourceVersion=0: dial tcp 10.3.0.203:443: getsockopt: connection refused
W1013 06:56:32.497280       1 swagger.go:32] No API exists for predefined swagger description /api/v1
W1013 06:56:32.511151       1 swagger.go:32] No API exists for predefined swagger description /api/v1
I1013 06:56:32.525278       1 master.go:369] Started Kubernetes proxy at 0.0.0.0:443/api/
I1013 06:56:32.525295       1 master.go:369] Started Origin API at 0.0.0.0:443/oapi/v1
I1013 06:56:32.525301       1 master.go:369] Started OAuth2 API at 0.0.0.0:443/oauth
I1013 06:56:32.525305       1 master.go:369] Started Web Console 0.0.0.0:443/console/
I1013 06:56:32.525310       1 master.go:369] Started Swagger Schema API at 0.0.0.0:443/swaggerapi/
I1013 06:56:32.525315       1 master.go:369] Started OpenAPI Schema at 0.0.0.0:443/swagger.json
I1013 06:56:32.680808       1 ensure.go:224] No cluster policy found.  Creating bootstrap policy based on: /config/policy.json
E1013 06:56:32.684183       1 reflector.go:214] github.com/openshift/origin/pkg/controller/shared/shared_informer.go:92: Failed to list *api.Namespace: User "system:openshift-master" cannot list all namespaces in the cluster
E1013 06:56:32.705647       1 reflector.go:214] github.com/openshift/origin/pkg/controller/shared/shared_informer.go:92: Failed to list *api.LimitRange: User "system:openshift-master" cannot list all limitranges in the cluster
E1013 06:56:32.716174       1 reflector.go:214] github.com/openshift/origin/pkg/controller/shared/shared_informer.go:92: Failed to list *api.SecurityContextConstraints: User "system:openshift-master" cannot list all securitycontextconstraints in the cluster
E1013 06:56:33.350326       1 reflector.go:203] github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/limitranger/admission.go:154: Failed to list *api.LimitRange: User "system:openshift-master" cannot list all limitranges in the cluster
E1013 06:56:33.361096       1 reflector.go:203] github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/limitranger/admission.go:154: Failed to list *api.LimitRange: User "system:openshift-master" cannot list all limitranges in the cluster
E1013 06:56:33.361385       1 reflector.go:214] github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/resourcequota/resource_access.go:83: Failed to list *api.ResourceQuota: User "system:openshift-master" cannot list all resourcequotas in the cluster
E1013 06:56:33.363610       1 reflector.go:214] github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/serviceaccount/admission.go:103: Failed to list *api.ServiceAccount: User "system:openshift-master" cannot list all serviceaccounts in the cluster
E1013 06:56:33.363814       1 reflector.go:214] github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/storageclass/default/admission.go:62: Failed to list *storage.StorageClass: User "system:openshift-master" cannot list all storage.k8s.io.storageclasses in the cluster
E1013 06:56:33.364437       1 reflector.go:214] github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/serviceaccount/admission.go:119: Failed to list *api.Secret: User "system:openshift-master" cannot list all secrets in the cluster
E1013 06:56:33.390786       1 reflector.go:214] github.com/openshift/origin/pkg/controller/shared/shared_informer.go:104: Failed to list *api.ClusterResourceQuota: User "system:openshift-master" cannot list all clusterresourcequotas in the cluster
E1013 06:56:33.391052       1 reflector.go:203] github.com/openshift/origin/pkg/project/cache/cache.go:95: Failed to list *api.Namespace: User "system:openshift-master" cannot list all namespaces in the cluster
E1013 06:56:33.685784       1 reflector.go:214] github.com/openshift/origin/pkg/controller/shared/shared_informer.go:92: Failed to list *api.Namespace: User "system:openshift-master" cannot list all namespaces in the cluster
E1013 06:56:33.746340       1 reflector.go:214] github.com/openshift/origin/pkg/controller/shared/shared_informer.go:92: Failed to list *api.SecurityContextConstraints: User "system:openshift-master" cannot list all securitycontextconstraints in the cluster
E1013 06:56:33.746889       1 reflector.go:214] github.com/openshift/origin/pkg/controller/shared/shared_informer.go:92: Failed to list *api.LimitRange: User "system:openshift-master" cannot list all limitranges in the cluster
E1013 06:56:34.357370       1 reflector.go:203] github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/limitranger/admission.go:154: Failed to list *api.LimitRange: User "system:openshift-master" cannot list all limitranges in the cluster
E1013 06:56:34.376587       1 reflector.go:214] github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/serviceaccount/admission.go:119: Failed to list *api.Secret: User "system:openshift-master" cannot list all secrets in the cluster
E1013 06:56:34.376781       1 reflector.go:203] github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/limitranger/admission.go:154: Failed to list *api.LimitRange: User "system:openshift-master" cannot list all limitranges in the cluster
E1013 06:56:34.377463       1 reflector.go:214] github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/serviceaccount/admission.go:103: Failed to list *api.ServiceAccount: User "system:openshift-master" cannot list all serviceaccounts in the cluster
E1013 06:56:34.377640       1 reflector.go:214] github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/resourcequota/resource_access.go:83: Failed to list *api.ResourceQuota: User "system:openshift-master" cannot list all resourcequotas in the cluster
E1013 06:56:34.377828       1 reflector.go:214] github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/storageclass/default/admission.go:62: Failed to list *storage.StorageClass: User "system:openshift-master" cannot list all storage.k8s.io.storageclasses in the cluster
E1013 06:56:34.396229       1 reflector.go:214] github.com/openshift/origin/pkg/controller/shared/shared_informer.go:104: Failed to list *api.ClusterResourceQuota: User "system:openshift-master" cannot list all clusterresourcequotas in the cluster
E1013 06:56:34.396409       1 reflector.go:203] github.com/openshift/origin/pkg/project/cache/cache.go:95: Failed to list *api.Namespace: User "system:openshift-master" cannot list all namespaces in the cluster
E1013 06:56:34.688656       1 reflector.go:214] github.com/openshift/origin/pkg/controller/shared/shared_informer.go:92: Failed to list *api.Namespace: User "system:openshift-master" cannot list all namespaces in the cluster
E1013 06:56:34.838164       1 reflector.go:214] github.com/openshift/origin/pkg/controller/shared/shared_informer.go:92: Failed to list *api.SecurityContextConstraints: User "system:openshift-master" cannot list all securitycontextconstraints in the cluster
E1013 06:56:34.855375       1 reflector.go:214] github.com/openshift/origin/pkg/controller/shared/shared_informer.go:92: Failed to list *api.LimitRange: User "system:openshift-master" cannot list all limitranges in the cluster
E1013 06:56:35.364750       1 reflector.go:203] github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/limitranger/admission.go:154: Failed to list *api.LimitRange: User "system:openshift-master" cannot list all limitranges in the cluster
E1013 06:56:35.388946       1 reflector.go:203] github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/limitranger/admission.go:154: Failed to list *api.LimitRange: User "system:openshift-master" cannot list all limitranges in the cluster
E1013 06:56:35.389266       1 reflector.go:214] github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/serviceaccount/admission.go:103: Failed to list *api.ServiceAccount: User "system:openshift-master" cannot list all serviceaccounts in the cluster
E1013 06:56:35.389515       1 reflector.go:214] github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/resourcequota/resource_access.go:83: Failed to list *api.ResourceQuota: User "system:openshift-master" cannot list all resourcequotas in the cluster
E1013 06:56:35.389794       1 reflector.go:214] github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/storageclass/default/admission.go:62: Failed to list *storage.StorageClass: User "system:openshift-master" cannot list all storage.k8s.io.storageclasses in the cluster
E1013 06:56:35.390014       1 reflector.go:214] github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/serviceaccount/admission.go:119: Failed to list *api.Secret: User "system:openshift-master" cannot list all secrets in the cluster
E1013 06:56:35.397930       1 reflector.go:214] github.com/openshift/origin/pkg/controller/shared/shared_informer.go:104: Failed to list *api.ClusterResourceQuota: User "system:openshift-master" cannot list all clusterresourcequotas in the cluster
E1013 06:56:35.398150       1 reflector.go:203] github.com/openshift/origin/pkg/project/cache/cache.go:95: Failed to list *api.Namespace: User "system:openshift-master" cannot list all namespaces in the cluster
E1013 06:56:35.859675       1 reflector.go:214] github.com/openshift/origin/pkg/controller/shared/shared_informer.go:92: Failed to list *api.SecurityContextConstraints: the server could not find the requested resource
I1013 06:56:35.863253       1 ensure.go:193] Ignoring default security context constraints when running on external Kubernetes.
E1013 06:56:36.414218       1 reflector.go:214] github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/storageclass/default/admission.go:62: Failed to list *storage.StorageClass: the server could not find the requested resource (get storageclasses.storage.k8s.io)
E1013 06:56:36.862688       1 reflector.go:214] github.com/openshift/origin/pkg/controller/shared/shared_informer.go:92: Failed to list *api.SecurityContextConstraints: the server could not find the requested resource
W1013 06:56:37.280011       1 run_components.go:205] Binding DNS on port 8053 instead of 53, which may not be resolvable from all clients
I1013 06:56:37.280335       1 logs.go:41] skydns: ready for queries on cluster.local. for tcp4://0.0.0.0:8053 [rcache 0]
I1013 06:56:37.280352       1 logs.go:41] skydns: ready for queries on cluster.local. for udp4://0.0.0.0:8053 [rcache 0]
I1013 06:56:37.381265       1 run_components.go:222] DNS listening at 0.0.0.0:8053
I1013 06:56:37.381545       1 start_master.go:576] Controllers starting (*)
I1013 06:56:37.381642       1 run_components.go:113] Skipped starting Service Account Token Manager, no private key specified
E1013 06:56:37.471166       1 reflector.go:214] github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/storageclass/default/admission.go:62: Failed to list *storage.StorageClass: the server could not find the requested resource (get storageclasses.storage.k8s.io)
I1013 06:56:37.497152       1 create_dockercfg_secrets.go:119] Dockercfg secret controller initialized, starting.
E1013 06:56:37.874948       1 reflector.go:214] github.com/openshift/origin/pkg/controller/shared/shared_informer.go:92: Failed to list *api.SecurityContextConstraints: the server could not find the requested resource
E1013 06:56:37.879822       1 util.go:45] Metric for replenishment_controller already registered
E1013 06:56:37.879851       1 util.go:45] Metric for replenishment_controller already registered
E1013 06:56:37.879867       1 util.go:45] Metric for replenishment_controller already registered
E1013 06:56:37.879934       1 util.go:45] Metric for replenishment_controller already registered
E1013 06:56:37.879956       1 util.go:45] Metric for replenishment_controller already registered
E1013 06:56:37.879973       1 util.go:45] Metric for replenishment_controller already registered
E1013 06:56:37.880303       1 util.go:45] Metric for replenishment_controller already registered
E1013 06:56:37.880326       1 util.go:45] Metric for replenishment_controller already registered
E1013 06:56:37.880358       1 util.go:45] Metric for replenishment_controller already registered
E1013 06:56:37.880380       1 util.go:45] Metric for replenishment_controller already registered
E1013 06:56:37.880396       1 util.go:45] Metric for replenishment_controller already registered
E1013 06:56:37.880412       1 util.go:45] Metric for replenishment_controller already registered
E1013 06:56:37.880480       1 util.go:45] Metric for replenishment_controller already registered
I1013 06:56:38.400636       1 start_master.go:738] Started Origin Controllers
E1013 06:56:38.471952       1 reflector.go:214] github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/storageclass/default/admission.go:62: Failed to list *storage.StorageClass: the server could not find the requested resource (get storageclasses.storage.k8s.io)
E1013 06:56:38.877357       1 reflector.go:214] github.com/openshift/origin/pkg/controller/shared/shared_informer.go:92: Failed to list *api.SecurityContextConstraints: the server could not find the requested resource
E1013 06:56:39.472665       1 reflector.go:214] github.com/openshift/origin/vendor/k8s.io/kubernetes/plugin/pkg/admission/storageclass/default/admission.go:62: Failed to list *storage.StorageClass: the server could not find the requested resource (get storageclasses.storage.k8s.io)
E1013 06:56:39.879872       1 reflector.go:214] github.com/openshift/origin/pkg/controller/shared/shared_informer.go:92: Failed to list *api.SecurityContextConstraints: the server could not find the requested resource

the last few messages are continuously logged forever Also quite a lot of like these

I1013 07:00:29.068113       1 logs.go:41] http: TLS handshake error from 10.2.53.0:56157: EOF
I1013 07:00:29.560691       1 logs.go:41] http: TLS handshake error from 10.2.48.1:53135: EOF

browsing to the ELB results in https://www.dropbox.com/s/zeblzta34gorvd9/Screenshot%202016-10-13%2010.16.46.png?dl=0

Suggestions?

paralin commented 8 years ago

Known issue, already an upstream issue posted here:

https://github.com/openshift/origin/issues/10367#issuecomment-252515516

To get this working we have to either wait for them to finish this work or use an older version of kube and openshift it seems.

markns commented 7 years ago

Hmm, I was seeing similar errors after following the guide here. Does this mean that openshift with external Kubernetes is impossible at the moment?

paralin commented 7 years ago

Yes, sadly, until something is fixed in openshift. I've completely dropped openshift now in favor of just using the kubernetes primitives and the CLI. Which is a shame, because I'd love to be able to use their awesome Jenkins integration.

markns commented 7 years ago

Oh, that's too bad. I was hoping to spin up origin on GKE, and have my own little PaaS up and running. There does seem to have been some activity related to the example in the Kubernetes repo relatively recently - I'll keep digging over there for a while.

paralin commented 7 years ago

If you can figure out a way to make it work, please please let me know! I'll integrate it back into this tool.

This repo is mostly here because I got frustrated and decided to automate the process in the OpenShift example with a few other little improvements and tweaks.