search

jenkins-x / jx

Jenkins X provides automated CI+CD for Kubernetes with Preview Environments on Pull Requests using Cloud Native pipelines from Tekton
https://jenkins-x.io/
Apache License 2.0
4.58k stars 788 forks source link

jx install on an existing cluster fails #2964

Closed agentgonzo closed 4 years ago

agentgonzo commented 5 years ago

Summary

Install a cluster, uninstall the cluster, then try to install again. install fails to create an environment with services "hook" not found

Steps to reproduce the behavior

Install a cluster do jx uninstall remove all remaining namespaces and CRDs to return the cluster to a 'virgin' state run jx install

Expected behavior

it should install the cluster

Actual behavior

It got as far as creating an envirnoment, then failed at 

Creating staging Environment in namespace jx
Created environment staging
Namespace jx-staging created
 error: creating the environments: failed to create staging environment in namespace jx: services "hook" not found

Jx version

The output of jx version is:

$ jx version
NAME               VERSION
jx                 ?[32m1.3.825?[0m
jenkins x platform ?[32m0.0.3321?[0m
Kubernetes cluster ?[32mv1.10.11-gke.1?[0m
kubectl            ?[32mv1.10.3?[0m
helm client        ?[32mv2.11.0+g2e55dbe?[0m
helm server        ?[32mv2.12.2+g7d2b0c7?[0m
git                ?[32mgit version 2.15.1.windows.2?[0m
Operating System   ?[32mWindows 10 Pro 1809 build 17763?[0m

Jenkins type

Kubernetes cluster

GKE

Operating system / Environment

Windows

Full output:

Namespace jx created
 Using helmBinary helm with feature flag: none
Context "shared-dev" modified.
? Cloud Provider gke
Storing the kubernetes provider gke in the TeamSettings
Git configured for user: Steve  and email <redacted>
Using helm2
Configuring tiller
Tiller Deployment is running in namespace kube-system
existing ingress controller found, no need to install a new one
Waiting for external loadbalancer to be created and update the nginx-ingress-controller service in kube-system namespace
Note: this loadbalancer will fail to be provisioned if you have insufficient quotas, this can happen easily on a GKE free account. To view quotas run: gcloud compute project-info describe
External loadbalancer created
Waiting to find the external host name of the ingress controller Service in namespace kube-system with name jxing-nginx-ingress-controller
You can now configure a wildcard DNS pointing to the new Load Balancer address <redacted>

If you do not have a custom domain setup yet, Ingress rules will be set for magic DNS nip.io.
Once you have a custom domain ready, you can update with the command jx upgrade ingress --cluster
If you don't have a wildcard DNS setup then setup a DNS (A) record and point it at: <redacted> then use the DNS domain in the next input...
? Domain <redacted>
nginx ingress controller installed and configured
Lets set up a Git user name and API token to be able to perform CI/CD

? local Git user for github.com server: jenkins-x-bot-test
Select the CI/CD pipelines Git server and user
? Do you wish to use github.com as the pipelines Git server: Yes
? pipelines Git user for github.com server: jenkins-x-bot-test
Setting the pipelines Git server https://github.com and user name jenkins-x-bot-test.
Saving the Git authentication configurationCurrent configuration dir: C:\Users\Steve\.jx
options.Flags.CloudEnvRepository: https://github.com/jenkins-x/cloud-environments
options.Flags.LocalCloudEnvironment: false
Cloning the Jenkins X cloud environments repo to C:\Users\Steve\.jx\cloud-environments
? A local Jenkins X cloud environments repository already exists, recreate with latest? Yes
Current configuration dir: C:\Users\Steve\.jx
options.Flags.CloudEnvRepository: https://github.com/jenkins-x/cloud-environments
options.Flags.LocalCloudEnvironment: false
Cloning the Jenkins X cloud environments repo to C:\Users\Steve\.jx\cloud-environments
Enumerating objects: 8, done.
Counting objects: 100% (8/8), done.
Compressing objects: 100% (6/6), done.
Total 1373 (delta 2), reused 6 (delta 2), pack-reused 1365
? Select Jenkins installation type: Serverless Jenkins
No default password set, generating a random one
Generated helm values C:\Users\Steve\.jx\extraValues.yaml
Creating Secret jx-install-config in namespace jx
Installing Jenkins X platform helm chart from: C:\Users\Steve\.jx\cloud-environments\env-gke

Installing knative into namespace jx
Updating Helm repository...
Helm repository update done.
Upgrading Chart 'upgrade --namespace jx --install --force --timeout 6000 --set build.auth.git.username=<redacted> --set build.auth.git.password=<redacted> --set  --set tillerNamespace= knative-build
 jenkins-x/knative-build'

retrying after error:failed to run 'helm upgrade --namespace jx --install --force --timeout 6000 --set build.auth.git.username=<redacted> --set build.auth.git.password=<redacted> *****  --set tiller
Namespace= knative-build jenkins-x/knative-build' command in directory '', output: 'Error: UPGRADE FAILED: a released named knative-build is in use, cannot re-use a name that is still in use'

Updating Helm repository...
Helm repository update done.
Upgrading Chart 'upgrade --namespace jx --install --force --timeout 6000 --set build.auth.git.username=<redacted> --set build.auth.git.password=<redacted> --set  --set tillerNamespace= knative-build
 jenkins-x/knative-build'
Waiting for tiller pod to be ready, service account name is tiller, namespace is jx, tiller namespace is kube-system
Waiting for cluster role binding to be defined, named tiller-role-binding in namespace jx
 tiller cluster role defined: cluster-admin in namespace jx
tiller pod running
? Pick workload build pack:  Kubernetes Workloads: Automated CI+CD with GitOps Promotion
Setting the team build pack to kubernetes-workloads repo: https://github.com/jenkins-x-buildpacks/jenkins-x-kubernetes.git ref: master
Installing jx into namespace jx
Installing jenkins-x-platform version: 0.0.3321
Adding values file C:\Users\Steve\.jx\cloud-environments\env-gke\myvalues.yaml
Adding values file C:\Users\Steve\.jx\adminSecrets.yaml
Adding values file C:\Users\Steve\.jx\extraValues.yaml
Adding values file C:\Users\Steve\.jx\cloud-environments\env-gke\secrets.yaml
Upgrading Chart 'upgrade --namespace jx --install --timeout 6000 --version 0.0.3321 --values C:\Users\Steve\.jx\cloud-environments\env-gke\myvalues.yaml --values C:\Users\Steve\.jx\adminSecrets.yaml --values C:\Users\Steve\.jx\extraValue
s.yaml --values C:\Users\Steve\.jx\cloud-environments\env-gke\secrets.yaml jenkins-x jenkins-x/jenkins-x-platform'
waiting for install to be ready, if this is the first time then it will take a while to download images
Jenkins X deployments ready in namespace jx
Configuring the TeamSettings for Prow

        ********************************************************

             NOTE: Your admin password is: <redacted>

        ********************************************************

Creating default staging and production environments
? Select the organization where you want to create the environment repository: cb-kubecd
Using Git provider ?[32mgithub.com at https://github.com?[0m

About to create repository ?[32menvironment-orioleglimmer-staging?[0m on server ?[32mhttps://github.com?[0m with user ?[32magentgonzo?[0m

Creating repository ?[32mcb-kubecd/environment-orioleglimmer-staging?[0m
Creating Git repository ?[32mcb-kubecd?[0m/?[32menvironment-orioleglimmer-staging?[0m
Pushed Git repository to ?[32mhttps://github.com/cb-kubecd/environment-orioleglimmer-staging?[0m

Creating staging Environment in namespace jx
Created environment staging
Namespace jx-staging created
 error: creating the environments: failed to create staging environment in namespace jx: services "hook" not found
agentgonzo commented 5 years ago 
NAME                               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
jenkins-x-chartmuseum              1         1         1            1           1h
jenkins-x-controllerbuild          1         1         1            1           1h
jenkins-x-controllercommitstatus   1         1         1            1           1h
jenkins-x-controllerrole           1         1         1            1           1h
jenkins-x-controllerteam           1         1         1            1           1h
jenkins-x-controllerworkflow       1         1         1            1           1h
jenkins-x-docker-registry          1         1         1            1           1h
jenkins-x-heapster                 1         1         1            1           1h
jenkins-x-mongodb                  1         1         1            1           1h
jenkins-x-monocular-api            1         1         1            1           1h
jenkins-x-monocular-prerender      1         1         1            1           1h
jenkins-x-monocular-ui             1         1         1            1           1h
jenkins-x-nexus                    1         1         1            1           1h
Steve@sarch ~/go/src/github.com/banzaicloud/bank-vaults/operator/pkg (master):[shared-dev]
$ helm list -a
NAME                                    REVISION        UPDATED                         STATUS          CHART                           APP VERSION     NAMESPACE
cb                                      13              Fri Jan 25 13:06:42 2019        DEPLOYED        core-0.0.501                                    jx
cert-manager                            1               Wed Dec  5 12:46:39 2018        DEPLOYED        cert-manager-v0.5.2             v0.5.2          cert-manager
jenkins-x                               2               Fri Feb  1 11:28:27 2019        DEPLOYED        jenkins-x-platform-0.0.3321                     jx
jx-build-templates                      15              Fri Jan 25 14:27:39 2019        DEPLOYED        jx-build-templates-0.0.379                      jx
jx-cb-kubecd-amuniz-golang-pr-1         1               Tue Dec 11 15:38:11 2018        DEPLOYED        preview-0.0.0-SNAPSHOT-PR-1-1                   jx-cb-kubecd-amuniz-golang-pr-1
jx-cb-kubecd-amuniz-golang-pr-10        1               Thu Dec 20 11:49:17 2018        DEPLOYED        preview-0.0.0-SNAPSHOT-PR-10-1                  jx-cb-kubecd-amuniz-golang-pr-10
jx-cb-kubecd-amuniz-golang-pr-11        1               Tue Jan 15 10:19:54 2019        DEPLOYED        preview-0.0.0-SNAPSHOT-PR-11-1                  jx-cb-kubecd-amuniz-golang-pr-11
jx-cb-kubecd-amuniz-golang-pr-12        1               Thu Jan 17 10:58:08 2019        DEPLOYED        preview-0.0.0-SNAPSHOT-PR-12-1                  jx-cb-kubecd-amuniz-golang-pr-12
jx-cb-kubecd-amuniz-golang-pr-3         4               Wed Dec 12 09:37:23 2018        DEPLOYED        preview-0.0.0-SNAPSHOT-PR-3-1                   jx-cb-kubecd-amuniz-golang-pr-3
jx-cb-kubecd-amuniz-golang-pr-5         1               Wed Dec 12 11:26:31 2018        DEPLOYED        preview-0.0.0-SNAPSHOT-PR-5-1                   jx-cb-kubecd-amuniz-golang-pr-5
jx-cb-kubecd-amuniz-golang-pr-6         1               Wed Dec 12 13:13:23 2018        DEPLOYED        preview-0.0.0-SNAPSHOT-PR-6-1                   jx-cb-kubecd-amuniz-golang-pr-6
jx-cb-kubecd-amuniz-golang-pr-8         2               Thu Dec 13 15:07:13 2018        DEPLOYED        preview-0.0.0-SNAPSHOT-PR-8-1                   jx-cb-kubecd-amuniz-golang-pr-8
jx-cb-kubecd-steve-golang-http-pr-1     1               Wed Dec  5 13:11:42 2018        DEPLOYED        preview-0.0.0-SNAPSHOT-PR-1-1                   jx-cb-kubecd-steve-golang-http-pr-1
jx-cb-kubecd-steve-golang-http-pr-2     1               Fri Dec  7 12:02:50 2018        DEPLOYED        preview-0.0.0-SNAPSHOT-PR-2-1                   jx-cb-kubecd-steve-golang-http-pr-2
jx-cb-kubecd-steve-golang-http-pr-3     1               Fri Dec  7 12:26:49 2018        DEPLOYED        preview-0.0.0-SNAPSHOT-PR-3-1                   jx-cb-kubecd-steve-golang-http-pr-3
jx-edit-steve-es                        1               Tue Jan 15 11:33:47 2019        DEPLOYED        exposecontroller-service-1.0.7                  jx-edit-steve
jx-sso-dex                              9               Fri Jan 25 13:06:47 2019        DEPLOYED        dex-2.13.4                                      sso
jx-sso-operator                         9               Fri Jan 25 13:06:51 2019        DEPLOYED        sso-operator-1.1.6                              sso
jxing                                   1               Wed Dec  5 12:37:40 2018        DEPLOYED        nginx-ingress-0.31.0            0.20.0          kube-system
knative-build                           3               Wed Dec 19 17:21:16 2018        DEPLOYED        knative-build-0.0.19                            jx
agentgonzo commented 5 years ago 
helm delete --purge knative-build
helm delete --purge jx-build-templates
helm delete --purge jx-prow
jx create addon prow

seemed to get it working again

chenfli commented 5 years ago

Happens to me as well on fresh versions 1.8XX and lately on 1.9

The same problem appears (I guess it is due to the empty --set on the helm upgrade for knative)

Summary

Install a cluster with serverless and expose controller with 'path-mode' Install succed, but didn't create all the prow components to enable jenkins-serverless

Steps to reproduce the behavior

Install a cluster with the following parmas

jx install --install-dependencies=true --provider=kubernetes --on-premise --log-level='debug' --namespace='jx' --exposecontroller-pathmode='path'

Expected behavior

it should install all jenkins x for serverless on the clusters including hook deck prow tide

Actual behavior

It got as far as creating an envirnoment finished with success, however, not svc/ingress/pod created for the above services.

Jx version

The output of jx version is:


NAME               VERSION
jx                 1.3.937
jenkins x platform 0.0.3402
Kubernetes cluster v1.13.3
kubectl            v1.13.3
helm client        v2.13.0+g79d0794
helm server        v2.13.0+g79d0794
git                git version 2.17.1
Operating System   Ubuntu 18.04.2 LTS

Upgrading Chart 

'upgrade --namespace jx --install --force --timeout 6000 --version 0.1.13 **--set**  --set tillerNamespace= --set build.auth.git.username=kubernetes-admin --set build.auth.git.username=kubernetes-admin --set build.auth.git.password=4e89d161980d0529c302a0ee42d77693ad85690f knative-build jenkins-x/knative-build'

jx install log

Here is the full instalation log, notice that it is an installation from scratch. And also **deck** and **prow** doesn't appear after installation

setting the dev namespace to: jx Using helmBinary helm with feature flag: none Context "kubernetes-admin@kubernetes" modified. set exposeController Config PathMode path Git configured for user: chenfli and email Using helm2 Configuring tiller Tiller Deployment is running in namespace kube-system ? No existing ingress controller found in the kube-system namespace, shall we install one? Yes Current configuration dir: /root/.jx versionRepository: https://github.com/jenkins-x/jenkins-x-versions using stable version 1.3.1 from charts of stable/nginx-ingress from /root/.jx/jenkins-x-versions Installing using helm binary: helm Installing Chart 'install --wait --name jxing --namespace kube-system stable/nginx-ingress --version 1.3.1 --set rbac.create=true' Waiting for external loadbalancer to be created and update the nginx-ingress-controller service in kube-system namespace Using external IP: 127.0.0.1 You can now configure a wildcard DNS pointing to the new Load Balancer address 127.0.0.1

If you do not have a custom domain setup yet, Ingress rules will be set for magic DNS nip.io. Once you have a custom domain ready, you can update with the command jx upgrade ingress --cluster If you don't have a wildcard DNS setup then setup a DNS (A) record and point it at: 127.0.0.1 then use the DNS domain in the next input... ? Domain nginx ingress controller installed and configured Lets set up a Git user name and API token to be able to perform CI/CD

Creating a local Git user for GitHub server ? GitHub user name: To be able to create a repository on GitHub we need an API Token Please click this URL https://github.com/settings/tokens/new?scopes=repo,read:user,read:org,user:email,write:repo_hook,delete_repo

Then COPY the token and enter in into the form below:

? API Token: **** Select the CI/CD pipelines Git server and user ? Do you wish to use GitHub as the pipelines Git server: Yes ? Do you wish to use chenfli as the pipelines Git user for GitHub server: Yes Setting the pipelines Git server https://github.com and user name . Saving the Git authentication configuration Current configuration dir: /root/.jx versionRepository: https://github.com/jenkins-x/jenkins-x-versions Current configuration dir: /root/.jx options.Flags.CloudEnvRepository: https://github.com/jenkins-x/cloud-environments

kubectl get svc

kubectl get po NAME READY STATUS RESTARTS AGE jenkins-x-chartmuseum-67f75c4884-2sms8 1/1 Running 0 110m jenkins-x-controllerbuild-7b59ccfcb6-bcjnl 1/1 Running 0 110m jenkins-x-controllercommitstatus-5cbc9f4776-lsl8s 1/1 Running 0 110m jenkins-x-controllerrole-767fd6f9b5-bgvp4 1/1 Running 0 110m jenkins-x-controllerteam-79989578c8-ngtr7 1/1 Running 0 110m jenkins-x-controllerworkflow-6476d44986-vfzfn 1/1 Running 0 110m jenkins-x-docker-registry-589d4d79c4-lzbvg 1/1 Running 0 110m jenkins-x-gcactivities-1551355200-vzw9z 0/1 Completed 0 67m jenkins-x-gcactivities-1551357000-zdpx8 0/1 Completed 0 37m jenkins-x-gcpods-1551355200-954r7 0/1 Completed 0 67m jenkins-x-gcpods-1551357000-7h9bx 0/1 Completed 0 37m jenkins-x-heapster-c476795-mkdbl 2/2 Running 0 110m jenkins-x-mongodb-5b9957c694-ckbwc 1/1 Running 1 110m jenkins-x-monocular-api-66b5f456b6-bt4kk 1/1 Running 0 110m jenkins-x-monocular-prerender-64fb689cfd-lnpbd 1/1 Running 0 110m jenkins-x-monocular-ui-fd67bbf5f-wsgxp 1/1 Running 0 110m jenkins-x-nexus-7d786f66d5-txnjh 0/1 ContainerCreating 0 106m

kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE heapster ClusterIP 10.3.0.12 8082/TCP 110m jenkins-x-chartmuseum ClusterIP 10.3.0.119 8080/TCP 110m jenkins-x-docker-registry ClusterIP 10.3.0.122 5000/TCP 110m jenkins-x-mongodb ClusterIP 10.3.0.182 27017/TCP 110m jenkins-x-monocular-api ClusterIP 10.3.0.44 80/TCP 110m jenkins-x-monocular-prerender ClusterIP 10.3.0.248 80/TCP 110m jenkins-x-monocular-ui ClusterIP 10.3.0.229 80/TCP 110m nexus ClusterIP 10.3.0.250 80/TCP 110m

kubectl get ing NAME HOSTS ADDRESS PORTS AGE chartmuseum hashed 80 107m docker-registry hashed 80 107m monocular hashed 80 107m nexus hashed 80 107m

jenkins-x-bot commented 5 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with /close. Provide feedback via https://jenkins-x.io/community. /lifecycle stale

jenkins-x-bot commented 5 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with /close. Provide feedback via https://jenkins-x.io/community. /lifecycle rotten

ccojocar commented 4 years ago

Closing this one since now jx boot is the recommended way to install Jenkins X and install command is now deprecated.