search

jenkins-x / jx

Jenkins X provides automated CI+CD for Kubernetes with Preview Environments on Pull Requests using Cloud Native pipelines from Tekton
https://jenkins-x.io/
Apache License 2.0
4.57k stars 786 forks source link

Jenkins install fails in on premise cluster #1317

Closed kevinsuperped closed 6 years ago

kevinsuperped commented 6 years ago

Summary

I have tried installing Jenkins-X on an on premise cluster twice, with the same results. After two hours, it has three running pods and these messages:

waiting for install to be ready, if this is the first time then it will take a while to download images
deployment jenkins failed to become ready in namespace jx
deployment jenkins-x-chartmuseum failed to become ready in namespace jx
deployment jenkins-x-docker-registry failed to become ready in namespace jx

Steps to reproduce the behavior

Install Jenkins-X using jx install --provider=kubernetes --on-premise

Jx version

The output of jx version is:

jx                 1.3.90
jenkins x platform 0.0.1827
kubernetes cluster v1.11.0
kubectl            v1.11.0
helm client        v2.9.1+g20adb27
helm server        v2.9.1+g20adb27
git                git version 2.7.4

A new jx version is available: 1.3.104
? Would you like to upgrade to the new jx version? [? for help] (Y/n)

Kubernetes cluster

What kind of Kubernetes cluster are you using & how did you create it?

On Premise running on Ubuntu serve 16.04

Used the kubeadm init method. I can deploy other k8s apps on it, Jenkins-X is the first issue I've had

Operating system / Environment

Ubuntu 16.04 Server running on Proxmox

Expected behavior

It to install

Actual behavior

Drags on install listing failed deployments about every half hour

rawlingsj commented 6 years ago

Can you run kubectl get pvc I wonder if you need a Kubernetes storage class to automatically create the Persistent Volumes for the Persistent Volume Claims that Jenkins X needs.

kevinsuperped commented 6 years ago

@rawlingsj Do you want to check for PVCs before an install or when it is trying to install? Or both?

rawlingsj commented 6 years ago

When it's installing, we'd expect the PVCs that are created as part of the Jenkins X installation bind to a PV. If they don't I can see this issue occurring. Another thing to check is if you have a Storage class for your cluster kubectl get sc

kevinsuperped commented 6 years ago

kubectl get pvc

NAME                         STATUS    VOLUME    CAPACITY   ACCESS MODES   STORAGECLASS   AGE
jenkins                      Pending                                                      2h
jenkins-x-chartmuseum        Pending                                                      2h
jenkins-x-docker-registry    Pending                                                      2h
jenkins-x-mongodb            Pending                                                      2h
jenkins-x-nexus              Pending                                                      2h
singing-buffalo-postgresql   Pending                                                      1h

kubectl get sc

no resources found

EDIT - Added local sc, going to retry install

@rawlingsj still no luck even with a local sc defined. Does my sc need to be able to create dynamic volumes?

kevinsuperped commented 6 years ago

Haven't heard anything, creates a default local sc, created a bunch of pvs before starting, the claims from jx don't grab them. Any ideas here?

kevinsuperped commented 6 years ago

SO I have tried this with Gluster now as well, and same issue with no pvc grabbing pvs. Are problems just a go **** yourself issue with this project?

rawlingsj commented 6 years ago

Sorry you sound frustrated but we do try and help as much as possible and it is an OSS community project. For more real-time support folks tend to ask on the slack channels https://jenkins-x.io/community/. There's also other Kubernetes channels there which might be able to help with the specific PV related issue but I'll try and offer more info here too.

So yes Jenkins X makes use of Kubernetes dynamic volume provisioning using a storageclass which most managed cloud providers some with OOTB. So my guess is the dynamic PV provisioning still isn't working as expected on your onprem setup. I wonder if the storage class is default? There's some more info here if you've not seen it https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/

I'd recommend going through and using this walkthrough + test project to ensure PVCs can bind to a PV on your cluster first https://kubernetes.io/docs/tasks/configure-pod-container/configure-persistent-volume-storage/ then once that's working I expect Jenkins X to be able to install correctly. Hope that helps.

kevinsuperped commented 6 years ago

@rawlingsj sorry for my crass remark, but it was late and I was very frustrated. I have just returned from a vacation, and I have set up glusterfs in the cluster, and set to default and dynamic sc, and the deploy progressed! It is now failing for an entirely different reason, which I will open a new ticket for.

I would like to recommend adding some info for requiring a dynamic storage sc before install, just so any other on premise installers save themselves some time.

Beyond that thanks for your help, the can be closed.

rawlingsj commented 6 years ago

Yeah totally understand :) Also great feedback and we've got the storage class and other prereqs added to a few places on the docs now, to avoid causing delays to other folks.

https://jenkins-x.io/getting-started/install-on-cluster/ https://jenkins-x.io/getting-started/install-on-cluster/#installing-jenkins-x-on-premise https://jenkins-x.io/faq/issues/

Thanks again!

vpoptani commented 4 years ago

@kevinsuperped , @rawlingsj i am facing the same issue. I have a single control plane cluster created using kubeadm. I created a Storage Class and made it default. But when i install jx i am getting the following outputs

Output of kubectl get pods -n jx

NAME READY STATUS RESTARTS AGE jenkins-54d84f9d87-jsk5h 0/1 Pending 0 4m5s jenkins-x-chartmuseum-774f8b95b-mmnj6 0/1 Pending 0 4m8s jenkins-x-controllerrole-79584f9477-fz9v8 1/1 Running 0 4m8s jenkins-x-docker-registry-dbf6889cd-kn4fs 0/1 Pending 0 4m8s jenkins-x-heapster-7fbb78bf79-hh982 2/2 Running 0 2m36s jenkins-x-nexus-555999cf9c-gcb8d 0/1 Pending 0 4m5s

Output of kubectl get pvc

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE jenkins Pending local-storage 2m23s jenkins-x-chartmuseum Pending local-storage 2m24s jenkins-x-docker-registry Pending local-storage 2m23s jenkins-x-nexus Pending local-storage 2m23s

Output of kubectl get sc

NAME PROVISIONER AGE local-storage (default) kubernetes.io/no-provisioner 112m slow kubernetes.io/gce-pd 128m

When i describe the PVC using kubectl get pvc jenkins -n jx i get below error

Events: Type Reason Age From Message

Normal WaitForFirstConsumer 2m32s (x42 over 12m) persistentvolume-controller waiting for first consumer to be created before binding

I know this issue is closed, but any help/inputs from you guys will be really helpful.