search

agilestacks / components

Stack components in Hub format
Apache License 2.0
8 stars 2 forks source link

Kubeflow deploy failed with Let's Encrypt #232

Closed kudla closed 4 years ago

kudla commented 5 years ago

Stack deployment logs

Enabled components: flannel, traefik, dex, cluster-autoscaler, minio, harbor, cert-manager, kubeflow

2019/05/21 12:07:51 deploy ***kubeflow*** (7/10)
2019/05/21 12:07:51 Parameters flattened (kubeflow):
2019/05/21 12:07:51         :component.ingress.protocol|kubeflow => `` (env:PROTOCOL)
2019/05/21 12:07:51         :component.acm.certificateArn|kubeflow => `` (env:ACM_CERTIFICATE_ARN)
2019/05/21 12:07:51         :kubernetes.api.endpoint|kubeflow => `` (env:K8S_API_SERVER)
2019/05/21 12:07:51         :terraform.bucket.name|kubeflow => `` (env:STATE_BUCKET)
2019/05/21 12:07:51         :terraform.bucket.region|kubeflow => `` (env:STATE_REGION)
2019/05/21 12:07:51         :dns.domain|kubeflow => `` (env:TF_VAR_domain_name)
2019/05/21 12:07:51         :dns.name|kubeflow => `` (env:TF_VAR_name)
2019/05/21 12:07:51         :component.kubeflow.name|kubeflow => `kubeflow` (env:TF_VAR_component)
2019/05/21 12:07:51         :component.kubeflow.namespace|kubeflow => `kubeflow` (env:TF_VAR_namespace)
2019/05/21 12:07:51         :component.kubeflow.version|kubeflow => `v0.5.1` (env:KUBEFLOW_VERSION)
2019/05/21 12:07:51         :component.kubeflow.urlPrefix|kubeflow => `kubeflow.svc` (env:TF_VAR_url_prefix)
2019/05/21 12:07:51         :component.kubeflow.apiSpec|kubeflow => `v1.13.5` (env:K8S_API_SPEC)
2019/05/21 12:07:51         :component.kubeflow.environment|kubeflow => `cloud` (env:KF_ENV)
2019/05/21 12:07:51         :component.kubeflow.argo.version|kubeflow => `v2.2.1` (env:ARGO_VERSION)
2019/05/21 12:07:51         :component.kubeflow.ambassador.image|kubeflow => `quay.io/datawire/ambassador:0.40.2` (env:AMBASSADOR_VERSION)
2019/05/21 12:07:51         :component.kubeflow.pipelines.tag|kubeflow => `0.1.19` (env:PIPELINE_VERSION)
2019/05/21 12:07:51 Empty parameter `component.acm.certificateArn|kubeflow` value allowed
2019/05/21 12:07:51 Adding implicit parameter hub.provides = `aws bucket cert-manager cluster-autoscaler helm ingress kubernetes minio tiller tls-ingress` (env: HUB_PROVIDES)
2019/05/21 12:07:51 Requirement `kubernetes` provided by `stack-k8s-aws`
2019/05/21 12:07:51 Requirement `ingress` provided by `traefik`
2019/05/21 12:07:51 Checking Kubeconfig context kube-01.c0foh01artem.dev.superhub.io
2019/05/21 12:07:51 kubectl output:
CURRENT   NAME                                   CLUSTER                                AUTHINFO                                     NAMESPACE
          kube-01.c0foh01artem.dev.superhub.io   kube-01.c0foh01artem.dev.superhub.io   admin@kube-01.c0foh01artem.dev.superhub.io   kube-system
2019/05/21 12:07:51 Assuming `ingress` requirement is setup
2019/05/21 12:07:51 State component `kubeflow` status: deploying
2019/05/21 12:07:51 State stack status: deploying
2019/05/21 12:07:51 State lifecycle phase `kubeflow` status: in-progress
2019/05/21 12:07:51 Component parameters:
2019/05/21 12:07:51     component.acm.certificateArn => `` (env:ACM_CERTIFICATE_ARN)
2019/05/21 12:07:51     component.ingress.protocol => `https` (env:PROTOCOL)
2019/05/21 12:07:51     component.kubeflow.ambassador.image => `quay.io/datawire/ambassador:0.40.2` (env:AMBASSADOR_VERSION)
2019/05/21 12:07:51     component.kubeflow.apiSpec => `v1.13.5` (env:K8S_API_SPEC)
2019/05/21 12:07:51     component.kubeflow.argo.version => `v2.2.1` (env:ARGO_VERSION)
2019/05/21 12:07:51     component.kubeflow.environment => `cloud` (env:KF_ENV)
2019/05/21 12:07:51     component.kubeflow.name => `kubeflow` (env:TF_VAR_component)
2019/05/21 12:07:51     component.kubeflow.namespace => `kubeflow` (env:TF_VAR_namespace)
2019/05/21 12:07:51     component.kubeflow.pipelines.tag => `0.1.19` (env:PIPELINE_VERSION)
2019/05/21 12:07:51     component.kubeflow.urlPrefix => `kubeflow.svc` (env:TF_VAR_url_prefix)
2019/05/21 12:07:51     component.kubeflow.version => `v0.5.1` (env:KUBEFLOW_VERSION)
2019/05/21 12:07:51     dns.domain => `kube-01.c0foh01artem.dev.superhub.io` (env:TF_VAR_domain_name)
2019/05/21 12:07:51     dns.name => `kube-01` (env:TF_VAR_name)
2019/05/21 12:07:51     hub.componentName => `kubeflow`
2019/05/21 12:07:51     hub.provides => `aws bucket cert-manager cluster-autoscaler helm ingress kubernetes minio tiller tls-ingress` (env:HUB_PROVIDES)
2019/05/21 12:07:51     kubernetes.api.endpoint => `api.kube-01.c0foh01artem.dev.superhub.io:6443` (env:K8S_API_SERVER)
2019/05/21 12:07:51     terraform.bucket.name => `agilestacks.c0foh01artem.dev.superhub.io` (env:STATE_BUCKET)
2019/05/21 12:07:51     terraform.bucket.region => `us-east-2` (env:STATE_REGION)
2019/05/21 12:07:51 Scanning for `kubeflow` templates `components/kubeflow/*.template`
2019/05/21 12:07:51 Component templates:
2019/05/21 12:07:51     components/kubeflow/default-storage-class.yaml.template (curly)
2019/05/21 12:07:51     components/kubeflow/oidc-crd.yaml.template (curly)
2019/05/21 12:07:51 Component environment:
2019/05/21 12:07:51     ACM_CERTIFICATE_ARN=
2019/05/21 12:07:51     AMBASSADOR_VERSION=quay.io/datawire/ambassador:0.40.2
2019/05/21 12:07:51     ARGO_VERSION=v2.2.1
2019/05/21 12:07:51     COMPONENT_NAME=kubeflow
2019/05/21 12:07:51     HUB_COMPONENT=kubeflow
2019/05/21 12:07:51     HUB_PROVIDES=aws bucket cert-manager cluster-autoscaler helm ingress kubernetes minio tiller tls-ingress
2019/05/21 12:07:51     K8S_API_SERVER=api.kube-01.c0foh01artem.dev.superhub.io:6443
2019/05/21 12:07:51     K8S_API_SPEC=v1.13.5
2019/05/21 12:07:51     KF_ENV=cloud
2019/05/21 12:07:51     KUBEFLOW_VERSION=v0.5.1
2019/05/21 12:07:51     PIPELINE_VERSION=0.1.19
2019/05/21 12:07:51     PROTOCOL=https
2019/05/21 12:07:51     STATE_BUCKET=agilestacks.c0foh01artem.dev.superhub.io
2019/05/21 12:07:51     STATE_REGION=us-east-2
2019/05/21 12:07:51     TF_VAR_component=kubeflow
2019/05/21 12:07:51     TF_VAR_domain_name=kube-01.c0foh01artem.dev.superhub.io
2019/05/21 12:07:51     TF_VAR_name=kube-01
2019/05/21 12:07:51     TF_VAR_namespace=kubeflow
2019/05/21 12:07:51     TF_VAR_url_prefix=kubeflow.svc
--- /usr/bin/make [deploy] (components/kubeflow)
make[1]: Entering directory '/workspace/repo/components/kubeflow'
rm -rf .terraform/modules .terraform/plugins ks-app
mkdir -p .terraform
kubectl --context="kube-01.c0foh01artem.dev.superhub.io" --namespace="kubeflow" create namespace kubeflow
2019/05/21 12:07:51 Wrote state `hub.yaml.state`
2019/05/21 12:07:51 Wrote state `s3://agilestacks.c0foh01artem.dev.superhub.io/kube-01.c0foh01artem.dev.superhub.io/hub/kube/hub.state`
ks init kubeflow --context=kube-01.c0foh01artem.dev.superhub.io --namespace=kubeflow --dir=ks-app
Error from server (AlreadyExists): namespaces "kubeflow" already exists
make[1]: [Makefile:34: namespace] Error 1 (ignored)
level=info msg="Using context \"kube-01.c0foh01artem.dev.superhub.io\" from kubeconfig file \"/root/.kube/config\""
level=info msg="Creating environment \"default\" with namespace \"kubeflow\", pointing to \"version:v1.8.0\" cluster at address \"https://api.kube-01.c0foh01artem.dev.superhub.io:6443\""
level=info msg="Generating ksonnet-lib data at path '/workspace/repo/components/kubeflow/ks-app/lib/ksonnet-lib/v1.8.0'"
terraform-v0.11 init  \
    -get=true -reconfigure -force-copy  \
    -backend=true -input=false \
    -backend-config="bucket=agilestacks.c0foh01artem.dev.superhub.io" \
    -backend-config="region=us-east-2" \
    -backend-config="key=kube-01.c0foh01artem.dev.superhub.io/kubeflow/kubeflow/terraform.tfstate" \
    -backend-config="profile=" \
    ./aws

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "null" (2.1.2)...
- Downloading plugin for provider "aws" (1.41.0)...
- Downloading plugin for provider "kubernetes" (1.0.1)...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.null: version = "~> 2.1"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
cd ks-app && ks env rm default
level=info msg="Deleting environment \"default\" with metadata at path \"/workspace/repo/components/kubeflow/ks-app/environments/default\""
level=info msg="Successfully removed environment 'default'"
cd ks-app && ks registry add kubeflow "/workspace/repo/components/kubeflow/kubeflow-v0.5.1/kubeflow"
cd ks-app && ks pkg install kubeflow/argo
level=info msg="Retrieved 6 files"
cd ks-app && ks pkg install kubeflow/common
level=info msg="Retrieved 22 files"
cd ks-app && ks pkg install kubeflow/jupyter
level=info msg="Retrieved 48 files"
cd ks-app && ks pkg install kubeflow/metacontroller
level=info msg="Retrieved 4 files"
cd ks-app && ks pkg install kubeflow/examples
level=info msg="Retrieved 9 files"
cd ks-app && ks pkg install kubeflow/katib
level=info msg="Retrieved 6 files"
cd ks-app && ks pkg install kubeflow/mpi-job
level=info msg="Retrieved 9 files"
cd ks-app && ks pkg install kubeflow/pytorch-job
level=info msg="Retrieved 6 files"
cd ks-app && ks pkg install kubeflow/pipeline
level=info msg="Retrieved 14 files"
cd ks-app && ks pkg install kubeflow/seldon
level=info msg="Retrieved 17 files"
cd ks-app && ks pkg install kubeflow/tf-training
level=info msg="Retrieved 4 files"
cd ks-app && ks generate argo argo
level=info msg="Writing component at '/workspace/repo/components/kubeflow/ks-app/components/argo.jsonnet'"
cd ks-app && ks generate ambassador ambassador
level=info msg="Writing component at '/workspace/repo/components/kubeflow/ks-app/components/ambassador.jsonnet'"
cd ks-app && ks generate metacontroller metacontroller
level=info msg="Writing component at '/workspace/repo/components/kubeflow/ks-app/components/metacontroller.jsonnet'"
cd ks-app && ks generate jupyter jupyter
level=info msg="Writing component at '/workspace/repo/components/kubeflow/ks-app/components/jupyter.jsonnet'"
cd ks-app && ks generate jupyter-web-app jupyter-web-app
level=info msg="Writing component at '/workspace/repo/components/kubeflow/ks-app/components/jupyter-web-app.jsonnet'"
cd ks-app && ks generate notebook-controller notebook-controller
level=info msg="Writing component at '/workspace/repo/components/kubeflow/ks-app/components/notebook-controller.jsonnet'"
cd ks-app && ks generate centraldashboard centraldashboard
level=info msg="Writing component at '/workspace/repo/components/kubeflow/ks-app/components/centraldashboard.jsonnet'"
cd ks-app && ks generate pytorch-operator pytorch-operator
level=info msg="Writing component at '/workspace/repo/components/kubeflow/ks-app/components/pytorch-operator.jsonnet'"
cd ks-app && ks generate katib katib
level=info msg="Writing component at '/workspace/repo/components/kubeflow/ks-app/components/katib.jsonnet'"
## ERROR Invalid value: \"Cluster\": ExternalTrafficPolicy can only be set on NodePort and LoadBalancer service"
cd ks-app && ks generate seldon seldon \
    --withApife true \
    --apifeServiceType ClusterIP \
    --externalTrafficPolicy="" \
    --withAmbassador false
level=info msg="Writing component at '/workspace/repo/components/kubeflow/ks-app/components/seldon.jsonnet'"
cd ks-app && ks generate tf-job-operator tf-job-operator
level=info msg="Writing component at '/workspace/repo/components/kubeflow/ks-app/components/tf-job-operator.jsonnet'"
cd ks-app && ks generate pipeline pipeline
level=info msg="Writing component at '/workspace/repo/components/kubeflow/ks-app/components/pipeline.jsonnet'"
cd ks-app && ks env add cloud --context=kube-01.c0foh01artem.dev.superhub.io
level=info msg="Using context \"kube-01.c0foh01artem.dev.superhub.io\" from kubeconfig file \"/root/.kube/config\""
level=info msg="Creating environment \"cloud\" with namespace \"kube-system\", pointing to \"version:v1.13.6\" cluster at address \"https://api.kube-01.c0foh01artem.dev.superhub.io:6443\""
level=info msg="Generating ksonnet-lib data at path '/workspace/repo/components/kubeflow/ks-app/lib/ksonnet-lib/v1.13.6'"
cd ks-app && ks env set cloud --namespace=kubeflow
cd ks-app && ks param set spartakus reportUsage false --env=cloud
level=info msg="setting environment component" component-name=spartakus
cd ks-app && ks param set argo imageTag v2.2.1 --env=cloud
level=info msg="setting environment component" component-name=argo
cd ks-app && ks param set ambassador ambassadorImage quay.io/datawire/ambassador:0.40.2 --env=cloud
level=info msg="setting environment component" component-name=ambassador
cd ks-app && ks param set ambassador ambassadorServiceType LoadBalancer --env=cloud
level=info msg="setting environment component" component-name=ambassador
cd ks-app && ks param set ambassador ambassadorProtocol https --env=cloud
level=info msg="setting environment component" component-name=ambassador
cd ks-app && ks param set ambassador ambassadorAcmCertificateArn  --env=cloud
level=info msg="setting environment globals"
cd ks-app && ks param set ambassador platform aws --env=cloud
level=info msg="setting environment component" component-name=ambassador
cd ks-app && ks param set jupyter-web-app image gcr.io/kubeflow-images-public/jupyter-web-app:v-latest-base-fcb983-1126234591183507456
cd ks-app && ks param set jupyter-web-app injectIstio false --env=cloud
level=info msg="setting environment component" component-name=jupyter-web-app
cd ks-app && ks param set --env=cloud pipeline apiImage gcr.io/ml-pipeline/api-server:0.1.19
level=info msg="setting environment component" component-name=pipeline
cd ks-app && ks param set --env=cloud pipeline scheduledWorkflowImage gcr.io/ml-pipeline/scheduledworkflow:0.1.19
level=info msg="setting environment component" component-name=pipeline
cd ks-app && ks param set --env=cloud pipeline persistenceAgentImage gcr.io/ml-pipeline/persistenceagent:0.1.19
level=info msg="setting environment component" component-name=pipeline
cd ks-app && ks param set --env=cloud pipeline viewerCrdControllerImage gcr.io/ml-pipeline/viewer-crd-controller:0.1.19
level=info msg="setting environment component" component-name=pipeline
cd ks-app && ks param set --env=cloud pipeline uiImage gcr.io/ml-pipeline/frontend:0.1.19
level=info msg="setting environment component" component-name=pipeline
kubectl --context="kube-01.c0foh01artem.dev.superhub.io" --namespace="kubeflow" apply -f default-storage-class.yaml
storageclass.storage.k8s.io/kubeflow-ebs-gp2 unchanged
cd ks-app && ks apply cloud -c ambassador --context=kube-01.c0foh01artem.dev.superhub.io
level=info msg="Applying services kubeflow.ambassador"
level=info msg="Applying services kubeflow.ambassador-admin"
level=info msg="Applying clusterroles ambassador"
level=info msg="Applying serviceaccounts kubeflow.ambassador"
level=info msg="Applying clusterrolebindings ambassador"
level=info msg="Applying deployments kubeflow.ambassador"
cd ks-app && ks apply cloud -c metacontroller --context=kube-01.c0foh01artem.dev.superhub.io
level=info msg="Applying customresourcedefinitions compositecontrollers.metacontroller.k8s.io"
level=info msg="Applying customresourcedefinitions controllerrevisions.metacontroller.k8s.io"
level=info msg="Applying customresourcedefinitions decoratorcontrollers.metacontroller.k8s.io"
level=info msg="Applying serviceaccounts kubeflow.meta-controller-service"
level=info msg="Applying clusterrolebindings meta-controller-cluster-role-binding"
level=info msg="Applying statefulsets kubeflow.metacontroller"
cd ks-app && ks apply cloud -c jupyter --context=kube-01.c0foh01artem.dev.superhub.io
level=info msg="Applying configmaps kubeflow.jupyter-config"
level=info msg="Applying services kubeflow.jupyter-0"
level=info msg="Applying rolebindings kubeflow.jupyter-role"
level=info msg="Applying roles kubeflow.jupyter-role"
level=info msg="Applying roles kubeflow.jupyter-notebook-role"
level=info msg="Applying services kubeflow.jupyter-lb"
level=info msg="Applying serviceaccounts kubeflow.jupyter"
level=info msg="Applying serviceaccounts kubeflow.jupyter-notebook"
level=info msg="Applying rolebindings kubeflow.jupyter-notebook-role"
level=info msg="Applying statefulsets kubeflow.jupyter"
cd ks-app && ks apply cloud -c jupyter-web-app --context=kube-01.c0foh01artem.dev.superhub.io
level=info msg="Applying services kubeflow.jupyter-web-app"
level=info msg="Applying deployments kubeflow.jupyter-web-app"
level=info msg="Applying configmaps kubeflow.jupyter-web-app-config"
level=info msg="Applying serviceaccounts kubeflow.jupyter-web-app"
level=info msg="Applying clusterrolebindings jupyter-web-app-binding"
level=info msg="Applying clusterroles jupyter-web-app-cluster-role"
level=info msg="Applying serviceaccounts kubeflow.default-editor"
level=info msg="Applying rolebindings kubeflow.default-editor-role-binding"
cd ks-app && ks apply cloud -c notebook-controller --context=kube-01.c0foh01artem.dev.superhub.io
level=info msg="Applying customresourcedefinitions notebooks.kubeflow.org"
level=info msg="Applying services kubeflow.notebooks-controller"
level=info msg="Applying serviceaccounts kubeflow.notebook-controller"
level=info msg="Applying deployments kubeflow.notebooks-controller"
level=info msg="Applying clusterroles notebooks-controller"
level=info msg="Applying clusterrolebindings notebooks-controller"
cd ks-app && ks apply cloud -c centraldashboard --context=kube-01.c0foh01artem.dev.superhub.io
level=info msg="Applying clusterrolebindings centraldashboard"
level=info msg="Applying services kubeflow.centraldashboard"
level=info msg="Applying serviceaccounts kubeflow.centraldashboard"
level=info msg="Applying roles kubeflow.centraldashboard"
level=info msg="Applying rolebindings kubeflow.centraldashboard"
level=info msg="Applying clusterroles centraldashboard"
level=info msg="Applying deployments kubeflow.centraldashboard"
cd ks-app && ks apply cloud -c pytorch-operator --context=kube-01.c0foh01artem.dev.superhub.io
level=info msg="Applying configmaps kubeflow.pytorch-operator-config"
level=info msg="Applying serviceaccounts kubeflow.pytorch-operator"
level=info msg="Applying clusterroles pytorch-operator"
level=info msg="Applying clusterrolebindings pytorch-operator"
level=info msg="Applying customresourcedefinitions pytorchjobs.kubeflow.org"
level=info msg="Applying deployments kubeflow.pytorch-operator"
cd ks-app && ks apply cloud -c argo --context=kube-01.c0foh01artem.dev.superhub.io
level=info msg="Applying customresourcedefinitions workflows.argoproj.io"
level=info msg="Applying clusterrolebindings argo"
level=info msg="Applying serviceaccounts kubeflow.argo-ui"
level=info msg="Applying services kubeflow.argo-ui"
level=info msg="Applying configmaps kubeflow.workflow-controller-configmap"
level=info msg="Applying serviceaccounts kubeflow.argo"
level=info msg="Applying clusterroles argo"
level=info msg="Applying clusterroles argo-ui"
level=info msg="Applying clusterrolebindings argo-ui"
level=info msg="Applying deployments kubeflow.workflow-controller"
level=info msg="Applying deployments kubeflow.argo-ui"
cd ks-app && ks apply cloud -c katib --context=kube-01.c0foh01artem.dev.superhub.io
level=info msg="Applying clusterroles metrics-collector"
level=info msg="Applying services kubeflow.studyjob-controller"
level=info msg="Applying services kubeflow.vizier-db"
level=info msg="Applying persistentvolumeclaims kubeflow.katib-mysql"
level=info msg="Applying clusterrolebindings studyjob-controller"
level=info msg="Applying secrets kubeflow.vizier-db-secrets"
level=info msg="Applying clusterroles vizier-core"
level=info msg="Applying clusterrolebindings vizier-core"
level=info msg="Applying serviceaccounts kubeflow.vizier-core"
level=info msg="Applying services kubeflow.vizier-core-rest"
level=info msg="Applying serviceaccounts kubeflow.studyjob-controller"
level=info msg="Applying services kubeflow.katib-ui"
level=info msg="Applying clusterroles studyjob-controller"
level=info msg="Applying clusterroles katib-ui"
level=info msg="Applying clusterrolebindings katib-ui"
level=info msg="Applying serviceaccounts kubeflow.katib-ui"
level=info msg="Applying services kubeflow.vizier-suggestion-random"
level=info msg="Applying services kubeflow.vizier-core"
level=info msg="Applying services kubeflow.vizier-suggestion-grid"
level=info msg="Applying configmaps kubeflow.metricscollector-template"
level=info msg="Applying services kubeflow.vizier-suggestion-hyperband"
level=info msg="Applying clusterrolebindings metrics-collector"
level=info msg="Applying services kubeflow.vizier-suggestion-bayesianoptimization"
level=info msg="Applying serviceaccounts kubeflow.metrics-collector"
level=info msg="Applying customresourcedefinitions studyjobs.kubeflow.org"
level=info msg="Applying configmaps kubeflow.worker-template"
level=info msg="Applying deployments kubeflow.vizier-suggestion-bayesianoptimization"
level=info msg="Applying deployments kubeflow.vizier-suggestion-hyperband"
level=info msg="Applying deployments kubeflow.vizier-suggestion-grid"
level=info msg="Applying deployments kubeflow.katib-ui"
level=info msg="Applying deployments kubeflow.vizier-core-rest"
level=info msg="Applying deployments kubeflow.vizier-db"
level=info msg="Applying deployments kubeflow.studyjob-controller"
level=info msg="Applying deployments kubeflow.vizier-core"
level=info msg="Applying deployments kubeflow.vizier-suggestion-random"
cd ks-app && ks apply cloud -c seldon --context=kube-01.c0foh01artem.dev.superhub.io
level=info msg="Applying serviceaccounts kubeflow.seldon"
level=info msg="Applying roles kubeflow.seldon-local"
level=info msg="Applying rolebindings kubeflow.seldon"
level=info msg="Applying services kubeflow.seldon-redis"
level=info msg="Applying customresourcedefinitions seldondeployments.machinelearning.seldon.io"
level=info msg="Applying deployments kubeflow.seldon-seldon-apiserver"
level=info msg="Applying services kubeflow.seldon-seldon-apiserver"
level=info msg="Applying deployments kubeflow.seldon-seldon-cluster-manager"
level=info msg="Applying deployments kubeflow.seldon-redis"
cd ks-app && ks apply cloud -c pipeline --context=kube-01.c0foh01artem.dev.superhub.io
level=info msg="Applying services kubeflow.minio-service"
level=info msg="Applying deployments kubeflow.minio"
level=info msg="Applying secrets kubeflow.mlpipeline-minio-artifact"
level=info msg="Applying services kubeflow.mysql"
level=info msg="Applying deployments kubeflow.mysql"
level=info msg="Applying serviceaccounts kubeflow.ml-pipeline"
level=info msg="Applying rolebindings kubeflow.ml-pipeline"
level=info msg="Applying roles kubeflow.ml-pipeline"
level=info msg="Applying services kubeflow.ml-pipeline"
level=info msg="Applying deployments kubeflow.ml-pipeline"
level=info msg="Applying serviceaccounts kubeflow.pipeline-runner"
level=info msg="Applying clusterroles pipeline-runner"
level=info msg="Applying clusterrolebindings pipeline-runner"
level=info msg="Applying serviceaccounts kubeflow.ml-pipeline-scheduledworkflow"
level=info msg="Applying clusterrolebindings ml-pipeline-scheduledworkflow"
level=info msg="Applying roles kubeflow.ml-pipeline-scheduledworkflow"
level=info msg="Applying deployments kubeflow.ml-pipeline-scheduledworkflow"
level=info msg="Applying customresourcedefinitions scheduledworkflows.kubeflow.org"
level=info msg="Applying serviceaccounts kubeflow.ml-pipeline-persistenceagent"
level=info msg="Applying clusterrolebindings ml-pipeline-persistenceagent"
level=info msg="Applying clusterroles ml-pipeline-persistenceagent"
level=info msg="Applying deployments kubeflow.ml-pipeline-persistenceagent"
level=info msg="Applying serviceaccounts kubeflow.ml-pipeline-viewer-crd-service-account"
level=info msg="Applying clusterrolebindings ml-pipeline-viewer-crd-role-binding"
level=info msg="Applying clusterroles ml-pipeline-viewer-controller-role"
level=info msg="Applying deployments kubeflow.ml-pipeline-viewer-controller-deployment"
level=info msg="Applying customresourcedefinitions viewers.kubeflow.org"
level=info msg="Applying serviceaccounts kubeflow.ml-pipeline-ui"
level=info msg="Applying services kubeflow.ml-pipeline-ui"
level=info msg="Applying services kubeflow.ml-pipeline-tensorboard-ui"
level=info msg="Applying rolebindings kubeflow.ml-pipeline-ui"
level=info msg="Applying roles kubeflow.ml-pipeline-ui"
level=info msg="Applying deployments kubeflow.ml-pipeline-ui"
level=info msg="Applying persistentvolumeclaims kubeflow.mysql-pv-claim"
level=info msg="Applying persistentvolumeclaims kubeflow.minio-pvc"
cd ks-app && ks apply cloud -c tf-job-operator --context=kube-01.c0foh01artem.dev.superhub.io
level=info msg="Applying customresourcedefinitions tfjobs.kubeflow.org"
level=info msg="Applying serviceaccounts kubeflow.tf-job-dashboard"
level=info msg="Applying configmaps kubeflow.tf-job-operator-config"
level=info msg="Applying serviceaccounts kubeflow.tf-job-operator"
level=info msg="Applying clusterroles tf-job-operator"
level=info msg="Applying clusterrolebindings tf-job-operator"
level=info msg="Applying services kubeflow.tf-job-dashboard"
level=info msg="Applying clusterroles tf-job-dashboard"
level=info msg="Applying clusterrolebindings tf-job-dashboard"
level=info msg="Applying deployments kubeflow.tf-job-operator"
level=info msg="Applying deployments kubeflow.tf-job-dashboard"
kubectl --context="kube-01.c0foh01artem.dev.superhub.io" --namespace="kubeflow" create -f oidc-crd.yaml
Error from server (AlreadyExists): error when creating "oidc-crd.yaml": oidcs.auth.agilestacks.com "oidc-ambassador" already exists
kubectl --context="kube-01.c0foh01artem.dev.superhub.io" --namespace="kubeflow" create secret generic ambassador-auth-jwt-key --from-literal=jwt-key=qafBTgd27hg0cX11tawDLu0SLsXXiiEW4kl6SC5p4eBwRCr0ejiSBDSdW8mQQwFh2nS0wOXtdWq28DvxEnIFaQ==
make[1]: [Makefile:115: install] Error 1 (ignored)
Error from server (AlreadyExists): secrets "ambassador-auth-jwt-key" already exists
kubectl --context="kube-01.c0foh01artem.dev.superhub.io" --namespace="kubeflow" create secret generic ambassador-auth-redis-password --from-literal=redis-password=MH2POuA75acYSvm5RvpOJsAJINI=
make[1]: [Makefile:116: install] Error 1 (ignored)
Error from server (AlreadyExists): secrets "ambassador-auth-redis-password" already exists
kubectl --context="kube-01.c0foh01artem.dev.superhub.io" --namespace="kubeflow" create secret generic ambassador-auth-oidc-provider --from-literal=oidc-provider=https://auth.kube-01.c0foh01artem.dev.superhub.io
make[1]: [Makefile:117: install] Error 1 (ignored)
Error from server (AlreadyExists): secrets "ambassador-auth-oidc-provider" already exists
kubectl --context="kube-01.c0foh01artem.dev.superhub.io" --namespace="kubeflow" create secret generic ambassador-auth-self-url --from-literal=self-url=https://kubeflow.svc.kube-01.c0foh01artem.dev.superhub.io/
make[1]: [Makefile:118: install] Error 1 (ignored)
Error from server (AlreadyExists): secrets "ambassador-auth-self-url" already exists
kubectl --context="kube-01.c0foh01artem.dev.superhub.io" --namespace="kubeflow" create secret generic ambassador-auth-client-id --from-literal=client-id=kubeflow-client
make[1]: [Makefile:119: install] Error 1 (ignored)
Error from server (AlreadyExists): secrets "ambassador-auth-client-id" already exists
make[1]: [Makefile:120: install] Error 1 (ignored)
kubectl --context="kube-01.c0foh01artem.dev.superhub.io" --namespace="kubeflow" create secret generic ambassador-auth-client-secret --from-literal=client-secret=345t6dfgdfg4-57yDSDfjeuDVQEW
Error from server (AlreadyExists): secrets "ambassador-auth-client-secret" already exists
kubectl --context="kube-01.c0foh01artem.dev.superhub.io" --namespace="kubeflow" create -f oidc-deployment.yaml
make[1]: [Makefile:121: install] Error 1 (ignored)
Error from server (AlreadyExists): error when creating "oidc-deployment.yaml": services "oidc-auth" already exists
Error from server (AlreadyExists): error when creating "oidc-deployment.yaml": deployments.apps "oidc-auth" already exists
kubectl --context="kube-01.c0foh01artem.dev.superhub.io" --namespace="kubeflow" create -f oidc-ambassador-service.yaml
make[1]: [Makefile:122: install] Error 1 (ignored)
Error from server (AlreadyExists): error when creating "oidc-ambassador-service.yaml": services "ambassador-oidc-auth" already exists
make[1]: [Makefile:123: install] Error 1 (ignored)
terraform-v0.11 get  ./aws
terraform-v0.11 plan  -refresh=true -module-depth=-1 -out=.terraform/terraform.tfplan ./aws
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

data.kubernetes_service.ambassador: Refreshing state...
data.aws_route53_zone.ext_zone: Refreshing state...
data.aws_region.current: Refreshing state...

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  + aws_route53_record.dns_url
      id:                 <computed>
      allow_overwrite:    "true"
      fqdn:               <computed>
      name:               "kubeflow.svc"
      records.#:          "1"
      records.4108050209: "0"
      ttl:                "300"
      type:               "CNAME"
      zone_id:            "Z2QJB7DL80CD9J"

  + aws_route53_record.dns_url_wildcard
      id:                 <computed>
      allow_overwrite:    "true"
      fqdn:               <computed>
      name:               "*.kubeflow.svc"
      records.#:          "1"
      records.4108050209: "0"
      ttl:                "300"
      type:               "CNAME"
      zone_id:            "Z2QJB7DL80CD9J"

  + null_resource.drop_elb
      id:                 <computed>


Plan: 3 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

This plan was saved to: .terraform/terraform.tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply ".terraform/terraform.tfplan"

Waiting for ambassador service elb
........................................................................................................................................................................................................................................................................................ ERROR timeout 600sec
make[1]: *** [Makefile:184: wait_elb] Error 1
make[1]: Leaving directory '/workspace/repo/components/kubeflow'
2019/05/21 12:20:42 State lifecycle operation `deploy` log appended: /usr/bin/make [deploy] (components/kubeflow): exit status 2\n--- stdout:\nmake[1]: Entering director...
2019/05/21 12:20:42 WARN: Component `kubeflow` failed to deploy: /usr/bin/make [deploy] (components/kubeflow): exit status 2
2019/05/21 12:20:42 State component `kubeflow` status: error
2019/05/21 12:20:42 State lifecycle phase `kubeflow` status: error
akranga commented 5 years ago

@kudla it looks like a timeout for DNS to be reachable. Do you have some steps to reproduce?

arkadijs commented 5 years ago

I'm experiencing same issue. If (platform) stack has no ACM (but has Let's Encrypt) then Ambassador load balancer cannot be created, pending forever with error due to service.beta.kubernetes.io/aws-load-balancer-ssl-cert: null.

mameshini commented 5 years ago

Let's wait for 0.6 upgrade since it will use Istio instead of Ambassador.

mameshini commented 4 years ago

Duplicate of applications#137