MuthuSundaravadivel commented 4 years ago

Describe the bug While installing CloudNative Developer toolkit in IKS Cluster. The following issues occurred.

Error: Error running command '.terraform/modules/dev_tools_jenkins_release/generic/tools/jenkins_release/scripts/deploy-jenkins.sh tools jenkins.gsi-dev-iks-cluster-7ec5d722a0ab3f463fdc90eeb94dbc70-0000.us-east.containers.appdomain.cloud 1.9.21 gsi-dev-iks-cluster-7ec5d722a0ab3f463fdc90eeb94dbc70-0000': exit status 1. Output: *** Fetching Jenkins helm chart from https://kubernetes-charts.storage.googleapis.com/ Error: unknown command "add" for "helm" Run 'helm --help' for usage.

Error: timeout while waiting for state to become 'success' (timeout: 30s)

on .terraform/modules/dev_serviceaccount_sonarqube/generic/cluster/serviceaccount/main.tf line 45, in resource "kubernetes_service_account" "create": 45: resource "kubernetes_service_account" "create" {

To Reproduce Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior A clear and concise description of what you expected to happen.

Screenshots

IBM Cloud Select the services and tools affected

[ ] IBM Kubernetes Service Managed Service
[ ] Red Hat OpenShift Managed Services
[ X] Jenkins
[ ] Tekton
[ X] SonarQube
[ ] Pact
[ ] Artifactory
[ ] IBM Image Registry

Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Additional context Add any other context about the problem here.

csantanapr commented 4 years ago

What version of iks your trying to deploy?

MuthuSundaravadivel commented 4 years ago

version is 1.15.11_1533

csantanapr commented 4 years ago

@MuthuSundaravadivel What settings did you use?

creating new cluster or existing cluster you ran the terraform twice on the same cluster without cleaning first, etc

The more information you put here for use to reproduce, that would be helpful.

csantanapr commented 4 years ago

Can you try with IKS 1.16 ?

MuthuSundaravadivel commented 4 years ago

Existing Cluster

MuthuSundaravadivel commented 4 years ago

@csantanapr This my env details

The type of cluster that will be created/used (kubernetes, openshift, ocp4, or crc) Use "openshift" for OpenShift 3.11

cluster_type="kubernetes"

Flag indicating if we are using an existing cluster or creating a new one

cluster_exists="true"

The prefix that should be applied to the cluster name and service names (if not provided

explicitly). If not provided then the resource_group_name will be used as the prefix.

name_prefix="gsi-cloudnative-dev-squad"

The cluster name can be provided (particularly if using an existing cluster). The value

for cluster name used by the scripts will be set in the following order of presidence:

- "${cluster_name}"

- "${name_prefix}-cluster"

- "${resource_group_name}-cluster"

cluster_name="gsi-dev-iks-cluster"

resource_group_name="gsi-cloudnative-dev-squad" vlan_region="us-south"

Flag indicating if we are using an existing postgres server or creating a new one

postgres_server_exists="false"

MuthuSundaravadivel commented 4 years ago

Even I have executed the terraform twice without cleaning. it's not working the same issue

craig2898 commented 4 years ago

I tried this a couple of times. Both creating a new cluster as well as using an existing. Same error either way.

The last time used these settings:

The type of cluster that will be created/used (kubernetes, openshift, ocp4, or crc) Use "openshift" for OpenShift 3.11

cluster_type="kubernetes"

Flag indicating if we are using an existing cluster or creating a new one

cluster_exists="true"

The prefix that should be applied to the cluster name and service names (if not provided

explicitly). If not provided then the resource_group_name will be used as the prefix.

name_prefix=""

The cluster name can be provided (particularly if using an existing cluster). The value

for cluster name used by the scripts will be set in the following order of presidence:

- "${cluster_name}"

- "${name_prefix}-cluster"

- "${resource_group_name}-cluster"

cluster_name="garagetools-cluster"

resource_group_name="garagetools" vlan_region="au-syd"

Flag indicating if we are using an existing postgres server or creating a new one

postgres_server_exists="true"

IKS Info

csantanapr commented 4 years ago

thanks for adding the settings used. And good to know @craig2898 you tried iks 1.16 and no prefix

MuthuSundaravadivel commented 4 years ago

@csantanapr Even I have tried "IKS 1.16" the same issue occurred Could you please help me to resolve

MuthuSundaravadivel commented 4 years ago

@csantanapr I have tried IKS 1.17 still I am facing the same issue. Hereby I have attached all the details please help us to resolve the issue. ENV details:

IKS Cluster Details:

Error Details:

csantanapr commented 4 years ago

@craig2898 yours is also failing with helm add error?

@seansund Any clues what it could be the helm add issue?

seansund commented 4 years ago

could you run it again with these env variables set first:

export TF_LOG=DEBUG
export TF_LOG_PATH={some path}

When its done you can zip up the log, put it in box, and attach the link?

csantanapr commented 4 years ago

Just to inform I was able to reproduce using latest master from iteration zero, setup credentials, vlan on wdc06, to create new cluster iks

Problem with helm add Problem with sonarqube Problem with cloudant

module.dev_tools_sonarqube_release.null_resource.sonarqube_release (local-exec): >>> Retry count exceeded. https://sonarqube.catalyst-demos-cluster-fa9ee67c9ab6a7791435450358e564cc-0000.us-east.containers.appdomain.cloud/about not available

Error: Error running command '.terraform/modules/dev_tools_jenkins_release/generic/tools/jenkins_release/scripts/deploy-jenkins.sh tools jenkins.catalyst-demos-cluster-fa9ee67c9ab6a7791435450358e564cc-0000.us-east.containers.appdomain.cloud 1.9.21 catalyst-demos-cluster-fa9ee67c9ab6a7791435450358e564cc-0000': exit status 1. Output: *** Fetching Jenkins helm chart from https://kubernetes-charts.storage.googleapis.com/
Error: unknown command "add" for "helm"
Run 'helm --help' for usage.

Error: Error running command '.terraform/modules/dev_tools_sonarqube_release/generic/tools/sonarqube_release/scripts/deploy-sonarqube.sh tools sonarqube.catalyst-demos-cluster-fa9ee67c9ab6a7791435450358e564cc-0000.us-east.containers.appdomain.cloud 4.4.0 sonarqube 2Gi "["https://binaries.sonarsource.com/Distribution/sonar-typescript-plugin/sonar-typescript-plugin-1.9.0.3766.jar","https://binaries.sonarsource.com/Distribution/sonar-java-plugin/sonar-java-plugin-5.14.0.18788.jar","https://github.com/checkstyle/sonar-checkstyle/releases/download/4.21/checkstyle-sonar-plugin-4.21.jar","https://binaries.sonarsource.com/Distribution/sonar-javascript-plugin/sonar-javascript-plugin-5.2.1.7778.jar","https://binaries.sonarsource.com/Distribution/sonar-python-plugin/sonar-python-plugin-1.14.1.3143.jar","https://binaries.sonarsource.com/Distribution/sonar-go-plugin/sonar-go-plugin-1.6.0.719.jar"]"': exit status 1. Output: *** Setting up kustomize directory
*** Updating patch-deployment.yaml with service account
*** Generating sonarqube yaml from helm template with plugins {https://binaries.sonarsource.com/Distribution/sonar-typescript-plugin/sonar-typescript-plugin-1.9.0.3766.jar,https://binaries.sonarsource.com/Distribution/sonar-java-plugin/sonar-java-plugin-5.14.0.18788.jar,https://github.com/checkstyle/sonar-checkstyle/releases/download/4.21/checkstyle-sonar-plugin-4.21.jar,https://binaries.sonarsource.com/Distribution/sonar-javascript-plugin/sonar-javascript-plugin-5.2.1.7778.jar,https://binaries.sonarsource.com/Distribution/sonar-python-plugin/sonar-python-plugin-1.14.1.3143.jar,https://binaries.sonarsource.com/Distribution/sonar-go-plugin/sonar-go-plugin-1.6.0.719.jar}
*** Building final kube yaml from kustomize into /home/devops/src/workspace/.tmp/sonarqube.yaml
*** Applying Sonarqube yaml to kube
configmap/sonarqube-sonarqube-config created
configmap/sonarqube-sonarqube-copy-plugins created
configmap/sonarqube-sonarqube-install-plugins created
secret/sonarqube-postgresql created
service/sonarqube-postgresql created
service/sonarqube-postgresql-headless created
service/sonarqube-sonarqube created
deployment.apps/sonarqube-sonarqube created
statefulset.apps/sonarqube-postgresql created
ingress.networking.k8s.io/sonarqube-sonarqube created
"toolkit-charts" has been added to your repositories
secret/sonarqube-access created
configmap/sonarqube-config created
*** Waiting for Sonarqube
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0    43    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
>>> waiting for https://sonarqube.catalyst-demos-cluster-fa9ee67c9ab6a7791435450358e564cc-0000.us-east.containers.appdomain.cloud/about to be available
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0    43    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
>>> waiting for https://sonarqube.catalyst-demos-cluster-fa9ee67c9ab6a7791435450358e564cc-0000.us-east.containers.appdomain.cloud/about to be available
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0    43    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
>>> waiting for https://sonarqube.catalyst-demos-cluster-fa9ee67c9ab6a7791435450358e564cc-0000.us-east.containers.appdomain.cloud/about to be available
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0    43    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
>>> waiting for https://sonarqube.catalyst-demos-cluster-fa9ee67c9ab6a7791435450358e564cc-0000.us-east.containers.appdomain.cloud/about to be available
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0    43    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
>>> waiting for https://sonarqube.catalyst-demos-cluster-fa9ee67c9ab6a7791435450358e564cc-0000.us-east.containers.appdomain.cloud/about to be available
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0    43    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
>>> waiting for https://sonarqube.catalyst-demos-cluster-fa9ee67c9ab6a7791435450358e564cc-0000.us-east.containers.appdomain.cloud/about to be available
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0    43    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
>>> waiting for https://sonarqube.catalyst-demos-cluster-fa9ee67c9ab6a7791435450358e564cc-0000.us-east.containers.appdomain.cloud/about to be available
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0    43    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
>>> waiting for https://sonarqube.catalyst-demos-cluster-fa9ee67c9ab6a7791435450358e564cc-0000.us-east.containers.appdomain.cloud/about to be available
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0    43    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
>>> waiting for https://sonarqube.catalyst-demos-cluster-fa9ee67c9ab6a7791435450358e564cc-0000.us-east.containers.appdomain.cloud/about to be available
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0    43    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
>>> waiting for https://sonarqube.catalyst-demos-cluster-fa9ee67c9ab6a7791435450358e564cc-0000.us-east.containers.appdomain.cloud/about to be available
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0    43    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
>>> Retry count exceeded. https://sonarqube.catalyst-demos-cluster-fa9ee67c9ab6a7791435450358e564cc-0000.us-east.containers.appdomain.cloud/about not available

Error: Error waiting for create resource instance (crn:v1:bluemix:public:cloudantnosqldb:us-east:a/98010327e775907f2d1bf637b10d2625:4a3d9bfd-8716-427b-b967-56f4a54e6576::) to be succeeded: unexpected state 'provisioning', wanted target 'active'. last error: %!s(<nil>)

  on .terraform/modules/dev_infrastructure_cloudant/cloud-managed/services/cloudant/main.tf line 14, in resource "ibm_resource_instance" "cloudant_instance":
  14: resource "ibm_resource_instance" "cloudant_instance" {

lsteck commented 4 years ago

I'm seeing what helm add error also. cluster_name="lds-test" resource_group_name="catalyst-springboot" vlan_region="us-south"

  0    43    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
>>> waiting for https://sonarqube.lds-test-793618-ddd93d3a0fef01f6b396b69d343df410-0000.us-south.containers.appdomain.cloud/about to be available
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0    43    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
>>> Retry count exceeded. https://sonarqube.lds-test-793618-ddd93d3a0fef01f6b396b69d343df410-0000.us-south.containers.appdomain.cloud/about not available

Error: Error running command '.terraform/modules/dev_tools_jenkins_release/generic/tools/jenkins_release/scripts/deploy-jenkins.sh tools jenkins.lds-test-793618-ddd93d3a0fef01f6b396b69d343df410-0000.us-south.containers.appdomain.cloud 1.9.21 lds-test-793618-ddd93d3a0fef01f6b396b69d343df410-0000': exit status 1. Output: *** Fetching Jenkins helm chart from https://kubernetes-charts.storage.googleapis.com/
Error: unknown command "add" for "helm"
Run 'helm --help' for usage.

Error: Error waiting for create resource instance (crn:v1:bluemix:public:cloudantnosqldb:us-south:a/c6b82bced8251e389648625d5b3ed9bf:fe4fffef-3aeb-42a7-a8eb-848222d1e436::) to be succeeded: unexpected state 'provisioning', wanted target 'active'. last error: %!s(<nil>)

  on .terraform/modules/dev_infrastructure_cloudant/cloud-managed/services/cloudant/main.tf line 14, in resource "ibm_resource_instance" "cloudant_instance":
  14: resource "ibm_resource_instance" "cloudant_instance" {

csantanapr commented 4 years ago

Here is a PR for jenkins helm add problem https://github.com/ibm-garage-cloud/garage-terraform-modules/pull/56

There is another problem with artifactory that @seansund is looking into it.

MuthuSundaravadivel commented 4 years ago

@csantanapr I am not sure I have done changes correctly or not. I have an updated version in my local machine "stage2-jenkins.tf" repo "ibm-garage-iteration-zero". Even though it's throwing the same error. To avoid the cloudant issue I have removed "stage3-cloudant.tf". For your reference, I have attached the screenshot.

I am requesting you to update the "ibm-garage-iteration-zero" repo. Please correct me if I am missed anything.

csantanapr commented 4 years ago

There are 2 more PRs in progress that need to be resolved first.

https://github.com/ibm-garage-cloud/garage-terraform-modules/pull/58

https://github.com/ibm-garage-cloud/garage-terraform-modules/pull/57

Then we will update iteration zero with the new references that include this 3 PRs

csantanapr commented 4 years ago

@lsteck @MuthuSundaravadivel @craig2898

We believed we have fix all issues with IKS, latest commits in master https://github.com/ibm-garage-cloud/ibm-garage-iteration-zero/commits/master

Please fetch all commits from master, or git clone the whole repo. Then try again creating a new cluster, or existing cluster that's doesn't have the toolkit partially installed

MuthuSundaravadivel commented 4 years ago

Cloud-Native toolkit installation successfully done in IKS 1.17. Thank you so much @csantanapr @seansund

cloud-native-toolkit / planning

Cloud Native toolkit installation issue in IKS Cluster #203

The type of cluster that will be created/used (kubernetes, openshift, ocp4, or crc) Use "openshift" for OpenShift 3.11

Flag indicating if we are using an existing cluster or creating a new one

The prefix that should be applied to the cluster name and service names (if not provided

explicitly). If not provided then the resource_group_name will be used as the prefix.

The cluster name can be provided (particularly if using an existing cluster). The value

for cluster name used by the scripts will be set in the following order of presidence:

- "${cluster_name}"

- "${name_prefix}-cluster"

- "${resource_group_name}-cluster"

Flag indicating if we are using an existing postgres server or creating a new one

The type of cluster that will be created/used (kubernetes, openshift, ocp4, or crc) Use "openshift" for OpenShift 3.11

Flag indicating if we are using an existing cluster or creating a new one

The prefix that should be applied to the cluster name and service names (if not provided

explicitly). If not provided then the resource_group_name will be used as the prefix.

name_prefix=""

The cluster name can be provided (particularly if using an existing cluster). The value

for cluster name used by the scripts will be set in the following order of presidence:

- "${cluster_name}"

- "${name_prefix}-cluster"

- "${resource_group_name}-cluster"

Flag indicating if we are using an existing postgres server or creating a new one