search

kubeflow / manifests

A repository for Kustomize manifests
Apache License 2.0
826 stars 887 forks source link

Numerous kubeflow manifests can't deploy on Kubernetes 1.16 #375

Closed jbrette closed 4 years ago

jbrette commented 5 years ago

Old declaration of key objects such as DaemonSet, StatefulSet and Deployment can't be deployed on Kubernetes 1.16 anymore if they are using the old apiVersion such as extensions/v1beta or the apps/v1beta

K8s 1.16 rejects Deployment for which the selector has not been set => Needs updates.

jbrette commented 5 years ago

ran

kfctl init master
kfctl generate

For each directory in the kustomize directory, run kubectl apply -k followed by kubectl delete -k

########################################################
# INIT                                                 #
########################################################
namespace/kubeflow created
customresourcedefinition.apiextensions.k8s.io/applications.app.k8s.io created
customresourcedefinition.apiextensions.k8s.io/pytorchjobs.kubeflow.org created
customresourcedefinition.apiextensions.k8s.io/adapters.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/apikeys.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/attributemanifests.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/authorizations.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/bypasses.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/certificates.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/challenges.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/checknothings.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/circonuses.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/cloudwatches.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/clusterrbacconfigs.rbac.istio.io created
customresourcedefinition.apiextensions.k8s.io/deniers.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/destinationrules.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/dogstatsds.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/edges.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/envoyfilters.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/fluentds.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/gateways.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/handlers.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/httpapispecbindings.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/httpapispecs.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/instances.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/issuers.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/kubernetesenvs.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/kuberneteses.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/listcheckers.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/listentries.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/logentries.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/memquotas.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/meshpolicies.authentication.istio.io created
customresourcedefinition.apiextensions.k8s.io/metrics.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/noops.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/opas.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/orders.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/policies.authentication.istio.io created
customresourcedefinition.apiextensions.k8s.io/prometheuses.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/quotas.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/quotaspecbindings.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/quotaspecs.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/rbacconfigs.rbac.istio.io created
customresourcedefinition.apiextensions.k8s.io/rbacs.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/redisquotas.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/reportnothings.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/rules.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/serviceentries.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/servicerolebindings.rbac.istio.io created
customresourcedefinition.apiextensions.k8s.io/serviceroles.rbac.istio.io created
customresourcedefinition.apiextensions.k8s.io/sidecars.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/signalfxs.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/solarwindses.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/stackdrivers.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/statsds.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/stdios.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/templates.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/tracespans.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/virtualservices.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/zipkins.config.istio.io created
########################################################
# ------------------
#  api-service
# ------------------
serviceaccount/ml-pipeline created
role.rbac.authorization.k8s.io/ml-pipeline created
rolebinding.rbac.authorization.k8s.io/ml-pipeline created
service/ml-pipeline created
error: unable to recognize "kustomize/api-service/": no matches for kind "Deployment" in version "apps/v1beta2"
# -------
serviceaccount "ml-pipeline" deleted
role.rbac.authorization.k8s.io "ml-pipeline" deleted
rolebinding.rbac.authorization.k8s.io "ml-pipeline" deleted
service "ml-pipeline" deleted
error: unable to recognize "kustomize/api-service/": no matches for kind "Deployment" in version "apps/v1beta2"
# ------------------
#  application
# ------------------
serviceaccount/application-controller-service-account created
clusterrole.rbac.authorization.k8s.io/application-controller-cluster-role created
clusterrolebinding.rbac.authorization.k8s.io/application-controller-cluster-role-binding created
configmap/application-controller-parameters created
service/application-controller-service created
statefulset.apps/application-controller-stateful-set created
application.app.k8s.io/kubeflow created
# -------
serviceaccount "application-controller-service-account" deleted
clusterrole.rbac.authorization.k8s.io "application-controller-cluster-role" deleted
clusterrolebinding.rbac.authorization.k8s.io "application-controller-cluster-role-binding" deleted
configmap "application-controller-parameters" deleted
service "application-controller-service" deleted
statefulset.apps "application-controller-stateful-set" deleted
application.app.k8s.io "kubeflow" deleted
# ------------------
#  argo
# ------------------
customresourcedefinition.apiextensions.k8s.io/workflows.argoproj.io created
serviceaccount/argo-ui created
serviceaccount/argo created
clusterrole.rbac.authorization.k8s.io/argo-ui created
clusterrole.rbac.authorization.k8s.io/argo created
clusterrolebinding.rbac.authorization.k8s.io/argo-ui created
clusterrolebinding.rbac.authorization.k8s.io/argo created
configmap/workflow-controller-configmap created
configmap/workflow-controller-parameters created
service/argo-ui created
virtualservice.networking.istio.io/argo-ui created
unable to recognize "kustomize/argo/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/argo/": no matches for kind "Deployment" in version "extensions/v1beta1"
# -------
customresourcedefinition.apiextensions.k8s.io "workflows.argoproj.io" deleted
serviceaccount "argo-ui" deleted
serviceaccount "argo" deleted
clusterrole.rbac.authorization.k8s.io "argo-ui" deleted
clusterrole.rbac.authorization.k8s.io "argo" deleted
clusterrolebinding.rbac.authorization.k8s.io "argo-ui" deleted
clusterrolebinding.rbac.authorization.k8s.io "argo" deleted
configmap "workflow-controller-configmap" deleted
configmap "workflow-controller-parameters" deleted
service "argo-ui" deleted
virtualservice.networking.istio.io "argo-ui" deleted
unable to recognize "kustomize/argo/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/argo/": no matches for kind "Deployment" in version "extensions/v1beta1"
# ------------------
#  bootstrap
# ------------------
serviceaccount/admission-webhook-bootstrap-service-account created
clusterrole.rbac.authorization.k8s.io/admission-webhook-bootstrap-cluster-role created
clusterrolebinding.rbac.authorization.k8s.io/admission-webhook-bootstrap-cluster-role-binding created
configmap/admission-webhook-bootstrap-config-map created
statefulset.apps/admission-webhook-bootstrap-stateful-set created
# -------
serviceaccount "admission-webhook-bootstrap-service-account" deleted
clusterrole.rbac.authorization.k8s.io "admission-webhook-bootstrap-cluster-role" deleted
clusterrolebinding.rbac.authorization.k8s.io "admission-webhook-bootstrap-cluster-role-binding" deleted
configmap "admission-webhook-bootstrap-config-map" deleted
statefulset.apps "admission-webhook-bootstrap-stateful-set" deleted
# ------------------
#  centraldashboard
# ------------------
serviceaccount/centraldashboard created
role.rbac.authorization.k8s.io/centraldashboard created
clusterrole.rbac.authorization.k8s.io/centraldashboard created
rolebinding.rbac.authorization.k8s.io/centraldashboard created
clusterrolebinding.rbac.authorization.k8s.io/centraldashboard created
configmap/parameters created
service/centraldashboard created
application.app.k8s.io/centraldashboard created
virtualservice.networking.istio.io/centraldashboard created
error: unable to recognize "kustomize/centraldashboard/": no matches for kind "Deployment" in version "extensions/v1beta1"
# -------
serviceaccount "centraldashboard" deleted
role.rbac.authorization.k8s.io "centraldashboard" deleted
clusterrole.rbac.authorization.k8s.io "centraldashboard" deleted
rolebinding.rbac.authorization.k8s.io "centraldashboard" deleted
clusterrolebinding.rbac.authorization.k8s.io "centraldashboard" deleted
configmap "parameters" deleted
service "centraldashboard" deleted
application.app.k8s.io "centraldashboard" deleted
virtualservice.networking.istio.io "centraldashboard" deleted
error: unable to recognize "kustomize/centraldashboard/": no matches for kind "Deployment" in version "extensions/v1beta1"
# ------------------
#  istio
# ------------------
configmap/istio-parameters-9m7m4fhdb8 created
gateway.networking.istio.io/kubeflow-gateway created
serviceentry.networking.istio.io/google-api-entry created
serviceentry.networking.istio.io/google-storage-api-entry created
virtualservice.networking.istio.io/google-api-vs created
virtualservice.networking.istio.io/google-storage-api-vs created
virtualservice.networking.istio.io/grafana-vs created
clusterrbacconfig.rbac.istio.io/default created
# -------
configmap "istio-parameters-9m7m4fhdb8" deleted
gateway.networking.istio.io "kubeflow-gateway" deleted
serviceentry.networking.istio.io "google-api-entry" deleted
serviceentry.networking.istio.io "google-storage-api-entry" deleted
virtualservice.networking.istio.io "google-api-vs" deleted
virtualservice.networking.istio.io "google-storage-api-vs" deleted
virtualservice.networking.istio.io "grafana-vs" deleted
clusterrbacconfig.rbac.istio.io "default" deleted
# ------------------
#  istio-install
# ------------------
namespace/istio-system created
mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector created
serviceaccount/istio-citadel-service-account created
serviceaccount/istio-cleanup-secrets-service-account created
serviceaccount/istio-egressgateway-service-account created
serviceaccount/istio-galley-service-account created
serviceaccount/istio-grafana-post-install-account created
serviceaccount/istio-ingressgateway-service-account created
serviceaccount/istio-mixer-service-account created
serviceaccount/istio-multi created
serviceaccount/istio-pilot-service-account created
serviceaccount/istio-security-post-install-account created
serviceaccount/istio-sidecar-injector-service-account created
serviceaccount/kiali-service-account created
serviceaccount/prometheus created
role.rbac.authorization.k8s.io/istio-ingressgateway-sds created
clusterrole.rbac.authorization.k8s.io/istio-citadel-istio-system created
clusterrole.rbac.authorization.k8s.io/istio-cleanup-secrets-istio-system created
clusterrole.rbac.authorization.k8s.io/istio-egressgateway-istio-system created
clusterrole.rbac.authorization.k8s.io/istio-galley-istio-system created
clusterrole.rbac.authorization.k8s.io/istio-grafana-post-install-istio-system created
clusterrole.rbac.authorization.k8s.io/istio-ingressgateway-istio-system created
clusterrole.rbac.authorization.k8s.io/istio-mixer-istio-system created
clusterrole.rbac.authorization.k8s.io/istio-pilot-istio-system created
clusterrole.rbac.authorization.k8s.io/istio-reader created
clusterrole.rbac.authorization.k8s.io/istio-sidecar-injector-istio-system created
clusterrole.rbac.authorization.k8s.io/kiali-viewer created
clusterrole.rbac.authorization.k8s.io/kiali created
clusterrole.rbac.authorization.k8s.io/prometheus-istio-system created
clusterrole.rbac.authorization.k8s.io/istio-security-post-install-istio-system created
rolebinding.rbac.authorization.k8s.io/istio-ingressgateway-sds created
clusterrolebinding.rbac.authorization.k8s.io/istio-citadel-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-cleanup-secrets-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-egressgateway-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-galley-admin-role-binding-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-grafana-post-install-role-binding-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-ingressgateway-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-kiali-admin-role-binding-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-mixer-admin-role-binding-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-multi created
clusterrolebinding.rbac.authorization.k8s.io/istio-pilot-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-sidecar-injector-admin-role-binding-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-security-post-install-role-binding-istio-system created
configmap/istio-galley-configuration created
configmap/istio-grafana-configuration-dashboards-galley-dashboard created
configmap/istio-grafana-configuration-dashboards-istio-mesh-dashboard created
configmap/istio-grafana-configuration-dashboards-istio-performance-dashboard created
configmap/istio-grafana-configuration-dashboards-istio-service-dashboard created
configmap/istio-grafana-configuration-dashboards-istio-workload-dashboard created
configmap/istio-grafana-configuration-dashboards-mixer-dashboard created
configmap/istio-grafana-configuration-dashboards-pilot-dashboard created
configmap/istio-grafana-custom-resources created
configmap/istio-grafana created
configmap/istio-security-custom-resources created
configmap/istio-sidecar-injector created
configmap/istio created
configmap/kiali created
configmap/prometheus created
secret/kiali created
service/grafana created
service/istio-citadel created
service/istio-egressgateway created
service/istio-galley created
service/istio-ingressgateway created
service/istio-pilot created
service/istio-policy created
service/istio-sidecar-injector created
service/istio-telemetry created
service/jaeger-agent created
service/jaeger-collector created
service/jaeger-query created
service/kiali created
service/prometheus created
service/tracing created
service/zipkin created
poddisruptionbudget.policy/istio-egressgateway created
poddisruptionbudget.policy/istio-galley created
poddisruptionbudget.policy/istio-ingressgateway created
poddisruptionbudget.policy/istio-pilot created
poddisruptionbudget.policy/istio-policy created
poddisruptionbudget.policy/istio-telemetry created
horizontalpodautoscaler.autoscaling/istio-egressgateway created
horizontalpodautoscaler.autoscaling/istio-ingressgateway created
horizontalpodautoscaler.autoscaling/istio-pilot created
horizontalpodautoscaler.autoscaling/istio-policy created
horizontalpodautoscaler.autoscaling/istio-telemetry created
job.batch/istio-cleanup-secrets-1.1.6 created
job.batch/istio-grafana-post-install-1.1.6 created
job.batch/istio-security-post-install-1.1.6 created
attributemanifest.config.istio.io/istioproxy created
attributemanifest.config.istio.io/kubernetes created
handler.config.istio.io/kubernetesenv created
handler.config.istio.io/prometheus created
handler.config.istio.io/stdio created
kubernetes.config.istio.io/attributes created
logentry.config.istio.io/accesslog created
logentry.config.istio.io/tcpaccesslog created
metric.config.istio.io/requestcount created
metric.config.istio.io/requestduration created
metric.config.istio.io/requestsize created
metric.config.istio.io/responsesize created
metric.config.istio.io/tcpbytereceived created
metric.config.istio.io/tcpbytesent created
metric.config.istio.io/tcpconnectionsclosed created
metric.config.istio.io/tcpconnectionsopened created
rule.config.istio.io/kubeattrgenrulerule created
rule.config.istio.io/promhttp created
rule.config.istio.io/promtcpconnectionclosed created
rule.config.istio.io/promtcpconnectionopen created
rule.config.istio.io/promtcp created
rule.config.istio.io/stdiotcp created
rule.config.istio.io/stdio created
rule.config.istio.io/tcpkubeattrgenrulerule created
destinationrule.networking.istio.io/istio-policy created
destinationrule.networking.istio.io/istio-telemetry created
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
# -------
namespace "istio-system" deleted
mutatingwebhookconfiguration.admissionregistration.k8s.io "istio-sidecar-injector" deleted
serviceaccount "istio-citadel-service-account" deleted
serviceaccount "istio-cleanup-secrets-service-account" deleted
serviceaccount "istio-egressgateway-service-account" deleted
serviceaccount "istio-galley-service-account" deleted
serviceaccount "istio-grafana-post-install-account" deleted
serviceaccount "istio-ingressgateway-service-account" deleted
serviceaccount "istio-mixer-service-account" deleted
serviceaccount "istio-multi" deleted
serviceaccount "istio-pilot-service-account" deleted
serviceaccount "istio-security-post-install-account" deleted
serviceaccount "istio-sidecar-injector-service-account" deleted
serviceaccount "kiali-service-account" deleted
serviceaccount "prometheus" deleted
role.rbac.authorization.k8s.io "istio-ingressgateway-sds" deleted
clusterrole.rbac.authorization.k8s.io "istio-citadel-istio-system" deleted
clusterrole.rbac.authorization.k8s.io "istio-cleanup-secrets-istio-system" deleted
clusterrole.rbac.authorization.k8s.io "istio-egressgateway-istio-system" deleted
clusterrole.rbac.authorization.k8s.io "istio-galley-istio-system" deleted
clusterrole.rbac.authorization.k8s.io "istio-grafana-post-install-istio-system" deleted
clusterrole.rbac.authorization.k8s.io "istio-ingressgateway-istio-system" deleted
clusterrole.rbac.authorization.k8s.io "istio-mixer-istio-system" deleted
clusterrole.rbac.authorization.k8s.io "istio-pilot-istio-system" deleted
clusterrole.rbac.authorization.k8s.io "istio-reader" deleted
clusterrole.rbac.authorization.k8s.io "istio-sidecar-injector-istio-system" deleted
clusterrole.rbac.authorization.k8s.io "kiali-viewer" deleted
clusterrole.rbac.authorization.k8s.io "kiali" deleted
clusterrole.rbac.authorization.k8s.io "prometheus-istio-system" deleted
clusterrole.rbac.authorization.k8s.io "istio-security-post-install-istio-system" deleted
rolebinding.rbac.authorization.k8s.io "istio-ingressgateway-sds" deleted
clusterrolebinding.rbac.authorization.k8s.io "istio-citadel-istio-system" deleted
clusterrolebinding.rbac.authorization.k8s.io "istio-cleanup-secrets-istio-system" deleted
clusterrolebinding.rbac.authorization.k8s.io "istio-egressgateway-istio-system" deleted
clusterrolebinding.rbac.authorization.k8s.io "istio-galley-admin-role-binding-istio-system" deleted
clusterrolebinding.rbac.authorization.k8s.io "istio-grafana-post-install-role-binding-istio-system" deleted
clusterrolebinding.rbac.authorization.k8s.io "istio-ingressgateway-istio-system" deleted
clusterrolebinding.rbac.authorization.k8s.io "istio-kiali-admin-role-binding-istio-system" deleted
clusterrolebinding.rbac.authorization.k8s.io "istio-mixer-admin-role-binding-istio-system" deleted
clusterrolebinding.rbac.authorization.k8s.io "istio-multi" deleted
clusterrolebinding.rbac.authorization.k8s.io "istio-pilot-istio-system" deleted
clusterrolebinding.rbac.authorization.k8s.io "istio-sidecar-injector-admin-role-binding-istio-system" deleted
clusterrolebinding.rbac.authorization.k8s.io "prometheus-istio-system" deleted
clusterrolebinding.rbac.authorization.k8s.io "istio-security-post-install-role-binding-istio-system" deleted
configmap "istio-galley-configuration" deleted
configmap "istio-grafana-configuration-dashboards-galley-dashboard" deleted
configmap "istio-grafana-configuration-dashboards-istio-mesh-dashboard" deleted
configmap "istio-grafana-configuration-dashboards-istio-performance-dashboard" deleted
configmap "istio-grafana-configuration-dashboards-istio-service-dashboard" deleted
configmap "istio-grafana-configuration-dashboards-istio-workload-dashboard" deleted
configmap "istio-grafana-configuration-dashboards-mixer-dashboard" deleted
configmap "istio-grafana-configuration-dashboards-pilot-dashboard" deleted
configmap "istio-grafana-custom-resources" deleted
configmap "istio-grafana" deleted
configmap "istio-security-custom-resources" deleted
configmap "istio-sidecar-injector" deleted
configmap "istio" deleted
configmap "kiali" deleted
configmap "prometheus" deleted
secret "kiali" deleted
service "grafana" deleted
service "istio-citadel" deleted
service "istio-egressgateway" deleted
service "istio-galley" deleted
service "istio-ingressgateway" deleted
service "istio-pilot" deleted
service "istio-policy" deleted
service "istio-sidecar-injector" deleted
service "istio-telemetry" deleted
service "jaeger-agent" deleted
service "jaeger-collector" deleted
service "jaeger-query" deleted
service "kiali" deleted
service "prometheus" deleted
service "tracing" deleted
service "zipkin" deleted
poddisruptionbudget.policy "istio-egressgateway" deleted
poddisruptionbudget.policy "istio-galley" deleted
poddisruptionbudget.policy "istio-ingressgateway" deleted
poddisruptionbudget.policy "istio-pilot" deleted
poddisruptionbudget.policy "istio-policy" deleted
poddisruptionbudget.policy "istio-telemetry" deleted
horizontalpodautoscaler.autoscaling "istio-egressgateway" deleted
horizontalpodautoscaler.autoscaling "istio-ingressgateway" deleted
horizontalpodautoscaler.autoscaling "istio-pilot" deleted
horizontalpodautoscaler.autoscaling "istio-policy" deleted
horizontalpodautoscaler.autoscaling "istio-telemetry" deleted
job.batch "istio-cleanup-secrets-1.1.6" deleted
job.batch "istio-grafana-post-install-1.1.6" deleted
job.batch "istio-security-post-install-1.1.6" deleted
attributemanifest.config.istio.io "istioproxy" deleted
attributemanifest.config.istio.io "kubernetes" deleted
handler.config.istio.io "kubernetesenv" deleted
handler.config.istio.io "prometheus" deleted
handler.config.istio.io "stdio" deleted
kubernetes.config.istio.io "attributes" deleted
logentry.config.istio.io "accesslog" deleted
logentry.config.istio.io "tcpaccesslog" deleted
metric.config.istio.io "requestcount" deleted
metric.config.istio.io "requestduration" deleted
metric.config.istio.io "requestsize" deleted
metric.config.istio.io "responsesize" deleted
metric.config.istio.io "tcpbytereceived" deleted
metric.config.istio.io "tcpbytesent" deleted
metric.config.istio.io "tcpconnectionsclosed" deleted
metric.config.istio.io "tcpconnectionsopened" deleted
rule.config.istio.io "kubeattrgenrulerule" deleted
rule.config.istio.io "promhttp" deleted
rule.config.istio.io "promtcpconnectionclosed" deleted
rule.config.istio.io "promtcpconnectionopen" deleted
rule.config.istio.io "promtcp" deleted
rule.config.istio.io "stdiotcp" deleted
rule.config.istio.io "stdio" deleted
rule.config.istio.io "tcpkubeattrgenrulerule" deleted
destinationrule.networking.istio.io "istio-policy" deleted
destinationrule.networking.istio.io "istio-telemetry" deleted
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
# ------------------
#  jupyter-web-app
# ------------------
serviceaccount/jupyter-web-app-service-account created
role.rbac.authorization.k8s.io/jupyter-web-app-jupyter-notebook-role created
clusterrole.rbac.authorization.k8s.io/jupyter-web-app-cluster-role created
rolebinding.rbac.authorization.k8s.io/jupyter-web-app-jupyter-notebook-role-binding created
clusterrolebinding.rbac.authorization.k8s.io/jupyter-web-app-cluster-role-binding created
configmap/jupyter-web-app-config created
configmap/jupyter-web-app-parameters created
service/jupyter-web-app-service created
deployment.apps/jupyter-web-app-deployment created
application.app.k8s.io/jupyter-web-app created
virtualservice.networking.istio.io/jupyter-web-app created
# -------
serviceaccount "jupyter-web-app-service-account" deleted
role.rbac.authorization.k8s.io "jupyter-web-app-jupyter-notebook-role" deleted
clusterrole.rbac.authorization.k8s.io "jupyter-web-app-cluster-role" deleted
rolebinding.rbac.authorization.k8s.io "jupyter-web-app-jupyter-notebook-role-binding" deleted
clusterrolebinding.rbac.authorization.k8s.io "jupyter-web-app-cluster-role-binding" deleted
configmap "jupyter-web-app-config" deleted
configmap "jupyter-web-app-parameters" deleted
service "jupyter-web-app-service" deleted
deployment.apps "jupyter-web-app-deployment" deleted
application.app.k8s.io "jupyter-web-app" deleted
virtualservice.networking.istio.io "jupyter-web-app" deleted
# ------------------
#  katib-controller
# ------------------
customresourcedefinition.apiextensions.k8s.io/experiments.kubeflow.org created
customresourcedefinition.apiextensions.k8s.io/trials.kubeflow.org created
serviceaccount/katib-controller created
clusterrole.rbac.authorization.k8s.io/katib-controller created
clusterrolebinding.rbac.authorization.k8s.io/katib-controller created
configmap/trial-template created
secret/katib-controller created
service/katib-controller created
deployment.apps/katib-controller created
# -------
customresourcedefinition.apiextensions.k8s.io "experiments.kubeflow.org" deleted
customresourcedefinition.apiextensions.k8s.io "trials.kubeflow.org" deleted
serviceaccount "katib-controller" deleted
clusterrole.rbac.authorization.k8s.io "katib-controller" deleted
clusterrolebinding.rbac.authorization.k8s.io "katib-controller" deleted
configmap "trial-template" deleted
secret "katib-controller" deleted
service "katib-controller" deleted
deployment.apps "katib-controller" deleted
# ------------------
#  katib-db
# ------------------
secret/katib-db-secrets created
service/katib-db created
persistentvolumeclaim/katib-mysql created
error: unable to recognize "kustomize/katib-db/": no matches for kind "Deployment" in version "extensions/v1beta1"
# -------
secret "katib-db-secrets" deleted
service "katib-db" deleted
persistentvolumeclaim "katib-mysql" deleted
error: unable to recognize "kustomize/katib-db/": no matches for kind "Deployment" in version "extensions/v1beta1"
# ------------------
#  katib-manager
# ------------------
service/katib-manager-rest created
service/katib-manager created
unable to recognize "kustomize/katib-manager/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/katib-manager/": no matches for kind "Deployment" in version "extensions/v1beta1"
# -------
service "katib-manager-rest" deleted
service "katib-manager" deleted
unable to recognize "kustomize/katib-manager/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/katib-manager/": no matches for kind "Deployment" in version "extensions/v1beta1"
# ------------------
#  katib-ui
# ------------------
serviceaccount/katib-ui created
clusterrole.rbac.authorization.k8s.io/katib-ui created
clusterrolebinding.rbac.authorization.k8s.io/katib-ui created
configmap/katib-parameters created
service/katib-ui created
virtualservice.networking.istio.io/katib-ui created
error: unable to recognize "kustomize/katib-ui/": no matches for kind "Deployment" in version "extensions/v1beta1"
# -------
serviceaccount "katib-ui" deleted
clusterrole.rbac.authorization.k8s.io "katib-ui" deleted
clusterrolebinding.rbac.authorization.k8s.io "katib-ui" deleted
configmap "katib-parameters" deleted
service "katib-ui" deleted
virtualservice.networking.istio.io "katib-ui" deleted
error: unable to recognize "kustomize/katib-ui/": no matches for kind "Deployment" in version "extensions/v1beta1"
# ------------------
#  metacontroller
# ------------------
customresourcedefinition.apiextensions.k8s.io/compositecontrollers.metacontroller.k8s.io created
customresourcedefinition.apiextensions.k8s.io/controllerrevisions.metacontroller.k8s.io created
customresourcedefinition.apiextensions.k8s.io/decoratorcontrollers.metacontroller.k8s.io created
serviceaccount/meta-controller-service created
clusterrolebinding.rbac.authorization.k8s.io/meta-controller-cluster-role-binding created
error: unable to recognize "kustomize/metacontroller/": no matches for kind "StatefulSet" in version "apps/v1beta2"
# -------
customresourcedefinition.apiextensions.k8s.io "compositecontrollers.metacontroller.k8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "controllerrevisions.metacontroller.k8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "decoratorcontrollers.metacontroller.k8s.io" deleted
serviceaccount "meta-controller-service" deleted
clusterrolebinding.rbac.authorization.k8s.io "meta-controller-cluster-role-binding" deleted
error: unable to recognize "kustomize/metacontroller/": no matches for kind "StatefulSet" in version "apps/v1beta2"
# ------------------
#  metadata
# ------------------
serviceaccount/metadata-ui created
role.rbac.authorization.k8s.io/metadata-ui created
rolebinding.rbac.authorization.k8s.io/metadata-ui created
configmap/metadata-ui-parameters-b6c8ghff7c created
secret/metadata-db-secrets created
service/metadata-db created
service/metadata-service created
service/metadata-ui created
virtualservice.networking.istio.io/metadata-ui created
persistentvolumeclaim/metadata-mysql created
unable to recognize "kustomize/metadata/": no matches for kind "Deployment" in version "apps/v1beta2"
unable to recognize "kustomize/metadata/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/metadata/": no matches for kind "Deployment" in version "extensions/v1beta1"
# -------
serviceaccount "metadata-ui" deleted
role.rbac.authorization.k8s.io "metadata-ui" deleted
rolebinding.rbac.authorization.k8s.io "metadata-ui" deleted
configmap "metadata-ui-parameters-b6c8ghff7c" deleted
secret "metadata-db-secrets" deleted
service "metadata-db" deleted
service "metadata-service" deleted
service "metadata-ui" deleted
virtualservice.networking.istio.io "metadata-ui" deleted
persistentvolumeclaim "metadata-mysql" deleted
unable to recognize "kustomize/metadata/": no matches for kind "Deployment" in version "apps/v1beta2"
unable to recognize "kustomize/metadata/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/metadata/": no matches for kind "Deployment" in version "extensions/v1beta1"
# ------------------
#  metrics-collector
# ------------------
serviceaccount/metrics-collector created
clusterrole.rbac.authorization.k8s.io/metrics-collector created
clusterrolebinding.rbac.authorization.k8s.io/metrics-collector created
configmap/metrics-collector-template created
# -------
serviceaccount "metrics-collector" deleted
clusterrole.rbac.authorization.k8s.io "metrics-collector" deleted
clusterrolebinding.rbac.authorization.k8s.io "metrics-collector" deleted
configmap "metrics-collector-template" deleted
# ------------------
#  minio
# ------------------
configmap/pipeline-minio-parameters created
secret/mlpipeline-minio-artifact created
service/minio-service created
persistentvolume/minio-pv created
persistentvolumeclaim/minio-pv-claim created
error: unable to recognize "kustomize/minio/": no matches for kind "Deployment" in version "apps/v1beta1"
# -------
configmap "pipeline-minio-parameters" deleted
secret "mlpipeline-minio-artifact" deleted
service "minio-service" deleted
persistentvolume "minio-pv" deleted
persistentvolumeclaim "minio-pv-claim" deleted
error: unable to recognize "kustomize/minio/": no matches for kind "Deployment" in version "apps/v1beta1"
# ------------------
#  mysql
# ------------------
configmap/pipeline-mysql-parameters created
service/mysql created
persistentvolume/mysql-pv created
persistentvolumeclaim/mysql-pv-claim created
error: unable to recognize "kustomize/mysql/": no matches for kind "Deployment" in version "apps/v1beta2"
# -------
configmap "pipeline-mysql-parameters" deleted
service "mysql" deleted
persistentvolume "mysql-pv" deleted
persistentvolumeclaim "mysql-pv-claim" deleted
error: unable to recognize "kustomize/mysql/": no matches for kind "Deployment" in version "apps/v1beta2"
# ------------------
#  notebook-controller
# ------------------
customresourcedefinition.apiextensions.k8s.io/notebooks.kubeflow.org created
serviceaccount/notebook-controller-service-account created
clusterrole.rbac.authorization.k8s.io/notebook-controller-role created
clusterrolebinding.rbac.authorization.k8s.io/notebook-controller-role-binding created
configmap/notebook-controller-parameters created
service/notebook-controller-service created
application.app.k8s.io/notebook-controller created
error: unable to recognize "kustomize/notebook-controller/": no matches for kind "Deployment" in version "apps/v1beta1"
# -------
customresourcedefinition.apiextensions.k8s.io "notebooks.kubeflow.org" deleted
serviceaccount "notebook-controller-service-account" deleted
clusterrole.rbac.authorization.k8s.io "notebook-controller-role" deleted
clusterrolebinding.rbac.authorization.k8s.io "notebook-controller-role-binding" deleted
configmap "notebook-controller-parameters" deleted
service "notebook-controller-service" deleted
application.app.k8s.io "notebook-controller" deleted
error: unable to recognize "kustomize/notebook-controller/": no matches for kind "Deployment" in version "apps/v1beta1"
# ------------------
#  persistent-agent
# ------------------
serviceaccount/ml-pipeline-persistenceagent created
clusterrole.rbac.authorization.k8s.io/ml-pipeline-persistenceagent created
clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-persistenceagent created
error: unable to recognize "kustomize/persistent-agent/": no matches for kind "Deployment" in version "apps/v1beta2"
# -------
serviceaccount "ml-pipeline-persistenceagent" deleted
clusterrole.rbac.authorization.k8s.io "ml-pipeline-persistenceagent" deleted
clusterrolebinding.rbac.authorization.k8s.io "ml-pipeline-persistenceagent" deleted
error: unable to recognize "kustomize/persistent-agent/": no matches for kind "Deployment" in version "apps/v1beta2"
# ------------------
#  pipelines-runner
# ------------------
serviceaccount/pipeline-runner created
clusterrole.rbac.authorization.k8s.io/pipeline-runner created
clusterrolebinding.rbac.authorization.k8s.io/pipeline-runner created
# -------
serviceaccount "pipeline-runner" deleted
clusterrole.rbac.authorization.k8s.io "pipeline-runner" deleted
clusterrolebinding.rbac.authorization.k8s.io "pipeline-runner" deleted
# ------------------
#  pipelines-ui
# ------------------
serviceaccount/ml-pipeline-ui created
role.rbac.authorization.k8s.io/ml-pipeline-ui created
rolebinding.rbac.authorization.k8s.io/ml-pipeline-ui created
configmap/ui-parameters-hb792fcf5d created
service/ml-pipeline-tensorboard-ui created
service/ml-pipeline-ui created
virtualservice.networking.istio.io/ml-pipeline-tensorboard-ui created
virtualservice.networking.istio.io/ml-pipeline-ui created
error: unable to recognize "kustomize/pipelines-ui/": no matches for kind "Deployment" in version "apps/v1beta2"
# -------
serviceaccount "ml-pipeline-ui" deleted
role.rbac.authorization.k8s.io "ml-pipeline-ui" deleted
rolebinding.rbac.authorization.k8s.io "ml-pipeline-ui" deleted
configmap "ui-parameters-hb792fcf5d" deleted
service "ml-pipeline-tensorboard-ui" deleted
service "ml-pipeline-ui" deleted
virtualservice.networking.istio.io "ml-pipeline-tensorboard-ui" deleted
virtualservice.networking.istio.io "ml-pipeline-ui" deleted
error: unable to recognize "kustomize/pipelines-ui/": no matches for kind "Deployment" in version "apps/v1beta2"
# ------------------
#  pipelines-viewer
# ------------------
customresourcedefinition.apiextensions.k8s.io/viewers.kubeflow.org created
serviceaccount/ml-pipeline-viewer-crd-service-account created
clusterrole.rbac.authorization.k8s.io/ml-pipeline-viewer-controller-role created
clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-viewer-crd-role-binding created
error: unable to recognize "kustomize/pipelines-viewer/": no matches for kind "Deployment" in version "apps/v1beta2"
# -------
customresourcedefinition.apiextensions.k8s.io "viewers.kubeflow.org" deleted
serviceaccount "ml-pipeline-viewer-crd-service-account" deleted
clusterrole.rbac.authorization.k8s.io "ml-pipeline-viewer-controller-role" deleted
clusterrolebinding.rbac.authorization.k8s.io "ml-pipeline-viewer-crd-role-binding" deleted
error: unable to recognize "kustomize/pipelines-viewer/": no matches for kind "Deployment" in version "apps/v1beta2"
# ------------------
#  pytorch-operator
# ------------------
serviceaccount/pytorch-operator created
clusterrole.rbac.authorization.k8s.io/pytorch-operator created
clusterrolebinding.rbac.authorization.k8s.io/pytorch-operator created
configmap/pytorch-operator-config created
service/pytorch-operator created
application.app.k8s.io/pytorch-operator created
error: unable to recognize "kustomize/pytorch-operator/": no matches for kind "Deployment" in version "extensions/v1beta1"
# -------
serviceaccount "pytorch-operator" deleted
clusterrole.rbac.authorization.k8s.io "pytorch-operator" deleted
clusterrolebinding.rbac.authorization.k8s.io "pytorch-operator" deleted
configmap "pytorch-operator-config" deleted
service "pytorch-operator" deleted
application.app.k8s.io "pytorch-operator" deleted
error: unable to recognize "kustomize/pytorch-operator/": no matches for kind "Deployment" in version "extensions/v1beta1"
# ------------------
#  scheduledworkflow
# ------------------
customresourcedefinition.apiextensions.k8s.io/scheduledworkflows.kubeflow.org created
serviceaccount/ml-pipeline-scheduledworkflow created
role.rbac.authorization.k8s.io/ml-pipeline-scheduledworkflow created
clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-scheduledworkflow created
error: unable to recognize "kustomize/scheduledworkflow/": no matches for kind "Deployment" in version "apps/v1beta2"
# -------
customresourcedefinition.apiextensions.k8s.io "scheduledworkflows.kubeflow.org" deleted
serviceaccount "ml-pipeline-scheduledworkflow" deleted
role.rbac.authorization.k8s.io "ml-pipeline-scheduledworkflow" deleted
clusterrolebinding.rbac.authorization.k8s.io "ml-pipeline-scheduledworkflow" deleted
error: unable to recognize "kustomize/scheduledworkflow/": no matches for kind "Deployment" in version "apps/v1beta2"
# ------------------
#  spartakus
# ------------------
serviceaccount/spartakus created
clusterrole.rbac.authorization.k8s.io/spartakus created
clusterrolebinding.rbac.authorization.k8s.io/spartakus created
configmap/spartakus-parameters created
error: unable to recognize "kustomize/spartakus/": no matches for kind "Deployment" in version "extensions/v1beta1"
# -------
serviceaccount "spartakus" deleted
clusterrole.rbac.authorization.k8s.io "spartakus" deleted
clusterrolebinding.rbac.authorization.k8s.io "spartakus" deleted
configmap "spartakus-parameters" deleted
error: unable to recognize "kustomize/spartakus/": no matches for kind "Deployment" in version "extensions/v1beta1"
# ------------------
#  suggestion
# ------------------
service/katib-suggestion-bayesianoptimization created
service/katib-suggestion-grid created
service/katib-suggestion-hyperband created
service/katib-suggestion-nasrl created
service/katib-suggestion-random created
unable to recognize "kustomize/suggestion/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/suggestion/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/suggestion/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/suggestion/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/suggestion/": no matches for kind "Deployment" in version "extensions/v1beta1"
# -------
service "katib-suggestion-bayesianoptimization" deleted
service "katib-suggestion-grid" deleted
service "katib-suggestion-hyperband" deleted
service "katib-suggestion-nasrl" deleted
service "katib-suggestion-random" deleted
unable to recognize "kustomize/suggestion/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/suggestion/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/suggestion/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/suggestion/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/suggestion/": no matches for kind "Deployment" in version "extensions/v1beta1"
# ------------------
#  tensorboard
# ------------------
configmap/parameters-dgd4h256h5 created
service/tensorboard created
virtualservice.networking.istio.io/tensorboard created
error: unable to recognize "kustomize/tensorboard/": no matches for kind "Deployment" in version "apps/v1beta1"
# -------
configmap "parameters-dgd4h256h5" deleted
service "tensorboard" deleted
virtualservice.networking.istio.io "tensorboard" deleted
error: unable to recognize "kustomize/tensorboard/": no matches for kind "Deployment" in version "apps/v1beta1"
# ------------------
#  tf-job-operator
# ------------------
customresourcedefinition.apiextensions.k8s.io/tfjobs.kubeflow.org created
serviceaccount/tf-job-dashboard created
serviceaccount/tf-job-operator created
clusterrole.rbac.authorization.k8s.io/tf-job-dashboard created
clusterrole.rbac.authorization.k8s.io/tf-job-operator created
clusterrolebinding.rbac.authorization.k8s.io/tf-job-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/tf-job-operator created
configmap/parameters-dgd4h256h5 created
configmap/tf-job-operator-config created
service/tf-job-dashboard created
service/tf-job-operator created
virtualservice.networking.istio.io/tf-job-dashboard created
unable to recognize "kustomize/tf-job-operator/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/tf-job-operator/": no matches for kind "Deployment" in version "extensions/v1beta1"
# -------
customresourcedefinition.apiextensions.k8s.io "tfjobs.kubeflow.org" deleted
serviceaccount "tf-job-dashboard" deleted
serviceaccount "tf-job-operator" deleted
clusterrole.rbac.authorization.k8s.io "tf-job-dashboard" deleted
clusterrole.rbac.authorization.k8s.io "tf-job-operator" deleted
clusterrolebinding.rbac.authorization.k8s.io "tf-job-dashboard" deleted
clusterrolebinding.rbac.authorization.k8s.io "tf-job-operator" deleted
configmap "parameters-dgd4h256h5" deleted
configmap "tf-job-operator-config" deleted
service "tf-job-dashboard" deleted
service "tf-job-operator" deleted
virtualservice.networking.istio.io "tf-job-dashboard" deleted
unable to recognize "kustomize/tf-job-operator/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/tf-job-operator/": no matches for kind "Deployment" in version "extensions/v1beta1"
# ------------------
#  webhook
# ------------------
customresourcedefinition.apiextensions.k8s.io/poddefaults.kubeflow.org created
mutatingwebhookconfiguration.admissionregistration.k8s.io/admission-webhook-mutating-webhook-configuration created
serviceaccount/admission-webhook-service-account created
clusterrole.rbac.authorization.k8s.io/admission-webhook-cluster-role created
clusterrolebinding.rbac.authorization.k8s.io/admission-webhook-cluster-role-binding created
configmap/admission-webhook-admission-webhook-parameters created
service/admission-webhook-service created
error: unable to recognize "kustomize/webhook/": no matches for kind "Deployment" in version "extensions/v1beta1"
# -------
customresourcedefinition.apiextensions.k8s.io "poddefaults.kubeflow.org" deleted
mutatingwebhookconfiguration.admissionregistration.k8s.io "admission-webhook-mutating-webhook-configuration" deleted
serviceaccount "admission-webhook-service-account" deleted
clusterrole.rbac.authorization.k8s.io "admission-webhook-cluster-role" deleted
clusterrolebinding.rbac.authorization.k8s.io "admission-webhook-cluster-role-binding" deleted
configmap "admission-webhook-admission-webhook-parameters" deleted
service "admission-webhook-service" deleted
error: unable to recognize "kustomize/webhook/": no matches for kind "Deployment" in version "extensions/v1beta1"
########################################################
# DELETE                                               #
########################################################
customresourcedefinition.apiextensions.k8s.io "adapters.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "apikeys.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "attributemanifests.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "authorizations.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "bypasses.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "certificates.certmanager.k8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "challenges.certmanager.k8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "checknothings.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "circonuses.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "cloudwatches.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "clusterissuers.certmanager.k8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "clusterrbacconfigs.rbac.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "deniers.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "destinationrules.networking.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "dogstatsds.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "edges.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "envoyfilters.networking.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "fluentds.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "gateways.networking.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "handlers.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "httpapispecbindings.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "httpapispecs.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "instances.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "issuers.certmanager.k8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "kubernetesenvs.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "kuberneteses.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "listcheckers.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "listentries.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "logentries.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "memquotas.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "meshpolicies.authentication.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "metrics.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "noops.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "opas.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "orders.certmanager.k8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "policies.authentication.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "prometheuses.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "quotas.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "quotaspecbindings.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "quotaspecs.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "rbacconfigs.rbac.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "rbacs.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "redisquotas.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "reportnothings.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "rules.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "serviceentries.networking.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "servicerolebindings.rbac.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "serviceroles.rbac.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "sidecars.networking.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "signalfxs.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "solarwindses.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "stackdrivers.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "statsds.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "stdios.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "templates.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "tracespans.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "virtualservices.networking.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "zipkins.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "applications.app.k8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "pytorchjobs.kubeflow.org" deleted
namespace "kubeflow" deleted
########################################################
jbrette commented 5 years ago

Changes to apiVersion statefulset, daemonset and deployment have been made. Ingress have been left to extensions but roles have been adjusted.

jlewi commented 5 years ago

@jbrette it looks like we are still getting reports that Kubeflow doesn't work on 1.16 Looks like the deployments might be an issue

See: kubeflow/kubeflow#4386 for example.

Can we create an E2E test to verify that Kubeflow is working on 1.16.

It doesn't look like GKE supports 1.16 yet so we either need to

  1. Wait for GKE to support 1.16
  2. Start to build an E2E test that uses something like kops to deploy Kubernetes
jlewi commented 5 years ago

I think we should treat 1.16 support as P0 for Kubeflow 1.0. 1.16 was released in September (https://github.com/kubernetes/sig-release/tree/master/releases/release-1.16).

And Kubernetes 1.17 is slated for December https://github.com/kubernetes/sig-release/tree/master/releases/release-1.17

So by January 2020 (ETA for Kubeflow 1.0) it will be approximately 4 months old and 1.17 will likely be here.

kubeflow/kubeflow#3496 is tracking setting up an E2E test that would hopefully allow us to test on Kubernetes 1.16.

AndriiNeverov commented 4 years ago

Getting a number of errors with Seldon CRD on Kubernetes 1.16.3. Noticed some differences to https://github.com/SeldonIO/seldon-core/blob/master/helm-charts/seldon-core-operator/templates/customresourcedefinition_seldondeployments.machinelearning.seldon.io.yaml:

                      endpoint:
                        properties:
                          service_host:
                            type: string
                          service_port:
                            type: integer
                          type:
                            enum:
                            - REST
                            - GRPC
                            type: string

vs

endpoint:
                        properties:
                          service_host:
                            type: string
                          service_port:
                            format: int32
                            type: integer
                          type:
                            type: string
                        type: object

WARN[0010] Encountered error during apply: (kubeflow.error): Code 500 with message: Apply.Run Error error when creating "/tmp/kout829193174": CustomResourceDefinition.apiextensions.k8s.io "seldondeployments.machinelearning.seldon.io" is invalid:

[spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[explainer].properties[endpoint].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[graph].properties[children].items.properties[children].items.properties[children].items.type: Required value: must not be empty for specified array items,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[graph].properties[children].items.properties[children].items.properties[endpoint].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[graph].properties[children].items.properties[children].items.type: Required value: must not be empty for specified array items,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[graph].properties[children].items.properties[endpoint].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[graph].properties[children].items.type: Required value: must not be empty for specified array items,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[graph].properties[endpoint].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[graph].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[svcOrchSpec].properties[env].items.properties[valueFrom].properties[configMapKeyRef].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[svcOrchSpec].properties[env].items.properties[valueFrom].properties[fieldRef].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[svcOrchSpec].properties[env].items.properties[valueFrom].properties[resourceFieldRef].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[svcOrchSpec].properties[env].items.properties[valueFrom].properties[secretKeyRef].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[svcOrchSpec].properties[env].items.properties[valueFrom].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[svcOrchSpec].properties[env].items.type: Required value: must not be empty for specified array items,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[svcOrchSpec].properties[resources].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.type: Required value: must not be empty for specified array items,

spec.validation.openAPIV3Schema.properties[spec].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.type: Required value: must not be empty at the root] filename="kustomize/kustomize.go:183"

ukclivecox commented 4 years ago

Getting a number of errors with Seldon CRD on Kubernetes 1.16.3. Noticed some differences to https://github.com/SeldonIO/seldon-core/blob/master/helm-charts/seldon-core-operator/templates/customresourcedefinition_seldondeployments.machinelearning.seldon.io.yaml:

                      endpoint:
                        properties:
                          service_host:
                            type: string
                          service_port:
                            type: integer
                          type:
                            enum:
                            - REST
                            - GRPC
                            type: string

vs

endpoint:
                        properties:
                          service_host:
                            type: string
                          service_port:
                            format: int32
                            type: integer
                          type:
                            type: string
                        type: object

WARN[0010] Encountered error during apply: (kubeflow.error): Code 500 with message: Apply.Run Error error when creating "/tmp/kout829193174": CustomResourceDefinition.apiextensions.k8s.io "seldondeployments.machinelearning.seldon.io" is invalid:

[spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[explainer].properties[endpoint].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[graph].properties[children].items.properties[children].items.properties[children].items.type: Required value: must not be empty for specified array items,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[graph].properties[children].items.properties[children].items.properties[endpoint].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[graph].properties[children].items.properties[children].items.type: Required value: must not be empty for specified array items,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[graph].properties[children].items.properties[endpoint].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[graph].properties[children].items.type: Required value: must not be empty for specified array items,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[graph].properties[endpoint].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[graph].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[svcOrchSpec].properties[env].items.properties[valueFrom].properties[configMapKeyRef].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[svcOrchSpec].properties[env].items.properties[valueFrom].properties[fieldRef].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[svcOrchSpec].properties[env].items.properties[valueFrom].properties[resourceFieldRef].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[svcOrchSpec].properties[env].items.properties[valueFrom].properties[secretKeyRef].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[svcOrchSpec].properties[env].items.properties[valueFrom].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[svcOrchSpec].properties[env].items.type: Required value: must not be empty for specified array items,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[svcOrchSpec].properties[resources].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.type: Required value: must not be empty for specified array items,

spec.validation.openAPIV3Schema.properties[spec].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.type: Required value: must not be empty at the root] filename="kustomize/kustomize.go:183"

@AndriiNeverov Can you try installing the latest Seldon as discussed at https://docs.seldon.io/projects/seldon-core/en/latest/workflow/install.html

This should work on 1.16.2.

We will update the kubeflow Manifests for Seldon to match the latest.

AndriiNeverov commented 4 years ago

@cliveseldon Followed the Kustomize section against v0.5.1, installed, adjusted ISTIO_ vars to match Kubeflow, created a deployment and posted a request with no issues. Please update the Manifests. Thanks!

jlewi commented 4 years ago

GKE 1.16.0 is now available via the rapid channel. https://cloud.google.com/kubernetes-engine/docs/release-notes-rapid

So it should now be possible to begin testing against 1.16 using GKE.

Anyone interested in picking this up?

kubeflow/manifests#676 should unblock being able to test against 1.16 with GKE.

jlewi commented 4 years ago

@jbrette or anyone else interested in trying to test this out with 1.16?

marziman commented 4 years ago

@jlewi I think we will run into that by the way. Maybe me and @svilenvul we can take a look into it. As we need to make it run in any case with 1.16.

vpavlin commented 4 years ago

We just hit this with our custom KF 0.7 on OpenShift 4.3 as that got rebased on Kubernetes 1.16 - any plans on backporting the changes to 0.7, or is this only intended to be forward compatible?

MnrGreg commented 4 years ago

Still experiencing an issue with the Seldon CRD on K8S 1.16.2. All other manifests seem fine.

failed to apply:  (kubeflow.error): Code 500 with message: kfApp Apply failed for kustomize:  (kubeflow.error): Code 500 with message: Apply.Run  Error error when creating "/tmp/kout776753220":
CustomResourceDefinition.apiextensions.k8s.io "seldondeployments.machinelearning.seldon.io" is invalid: [
spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[explainer].properties[endpoint].type: Required value: must not be empty for specified object fields,
spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[graph].properties[children].items.properties[children].items.properties[children].items.type: Required value: must not be empty for specified array items,
spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[graph].properties[children].items.properties[children].items.properties[endpoint].type: Required value: must not be empty for specified object fields
...

The spartakus deployment is up-to-date and running fine:

$ kubectl get deploy -n kubeflow  spartakus-volunteer -oyaml | more
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  creationTimestamp: "2020-02-12T00:40:56Z"
  generation: 1
  labels:
    app: spartakus
    app.kubernetes.io/component: spartakus
    app.kubernetes.io/instance: spartakus-v0.7.0
    app.kubernetes.io/managed-by: kfctl
    app.kubernetes.io/name: spartakus
    app.kubernetes.io/part-of: kubeflow
    app.kubernetes.io/version: v0.7.0
    kustomize.component: spartakus
  name: spartakus-volunteer
  namespace: kubeflow

Just seems the Seldon CRD needs updating.

$ kfctl version
kfctl v1.0-rc.3-1-g24b60e8

kubeflow manifest: https://github.com/kubeflow/manifests/raw/master/kfdef/kfctl_k8s_istio.yaml

ukclivecox commented 4 years ago

We could create a 0.7 PR to just update the apiVersions that need updating for Seldon. Have you checked Kubeflow 1.0 RC to ensure its 1.16 compatible?

MnrGreg commented 4 years ago

@cliveseldon yes, the documentation still reflects Kubeflow 1.0 being incompatible with K8S 1.16. minimum-system-requirements

When deploying the master Kubeflow manifests with kfctl v1.0-rc.3-1-g24b60e8 yesterday, the Seldon CRD was the only component which failed to deploy.

ukclivecox commented 4 years ago

Which kfdef are you using?

MnrGreg commented 4 years ago

I was using https://github.com/kubeflow/manifests/raw/master/kfdef/kfctl_k8s_istio.yaml

and pointing repos to:

  repos:
  - name: manifests
    uri: https://github.com/kubeflow/manifests/archive/v1.0-rc.0.tar.gz

since using https://github.com/kubeflow/manifests/raw/master/kfdef/kfctl_k8s_istio.v1.0.0.yaml with

  repos:
  - name: manifests
    uri: https://github.com/kubeflow/manifests/archive/v1.0-branch.tar.gz

I haven't seen the Seldon CRD issue. kfctl apply goes though fine.

ukclivecox commented 4 years ago

@MnrGreg The rc.0.tar had the old version of Seldon from 0.7 release as the updates were not yet cherrypicked to the release branch

yanniszark commented 4 years ago

@MnrGreg did you manage to deploy any of the Kubeflow configurations successfully on K8s 1.16?

MnrGreg commented 4 years ago

@yanniszark yes, with kfctl_k8s_istio.v1.0.0.yaml we were able to create all CRDs and deployments on k8s 1.16.2

Kubernetes 1.16.2 containerd/cri PhotonOS 3R2 nfs-client-provisioner v3.1.0-k8s1.11

saschagrunert commented 4 years ago

@yanniszark yes, with kfctl_k8s_istio.v1.0.0.yaml we were able to create all CRDs and deployments on k8s 1.16.2

Kubernetes 1.16.2 containerd/cri PhotonOS 3R2 nfs-client-provisioner v3.1.0-k8s1.11

Can confirm that, works on the latest k8s beta as well (v1.18.0-beta.1)

Kyrremann commented 4 years ago

This may be the wrong thread, but I'm trying to run Kubeflow on on-prem v1.16, and we're having trouble with the cert-manager. The webhook returns 403 when the api-server tries to communicate with it.

failing or missing response from https://10.254.80.38:443/apis/webhook.cert-manager.io/v1beta1: bad status from https://10.254.80.38:443/apis/webhook.cert-manager.io/v1beta1: 403

I've made an issue over at https://github.com/jetstack/cert-manager/issues/2678.

jlewi commented 4 years ago

@Kyrremann yeah that looks like a different issue then this issue. If there is something specific to kubeflow regarding jetstack/cert-manager#2678 then I would suggest opening a new issue.

animeshsingh commented 4 years ago

@jlewi looks like based on thread, except for Seldon most of the components are running on 1.16 - is there some formalization for testing et all you are looking at?

ukclivecox commented 4 years ago

@animeshsingh Seldon should run on 1.16

vpavlin commented 4 years ago

Yeah, Seldon in master works fine on OpenShift 4.3 (K8S 1.16), it just takes longer to successfully sttart due to slow ceert-manager.

jlewi commented 4 years ago

@animeshsingh yes. We need CI. We should be able to look at one or more of our test grids and have confidence that everything is working. https://k8s-testgrid.appspot.com/sig-big-data

This probably means we need to create a GKE deployment running on 1.16 and run our tests on that.

Alternatively, if we didn't want to use GKE someone would have to invest in CI for deploying some otherway; e.g. using kops.

yanniszark commented 4 years ago

Can we close this issue now that users have confirmed Kubeflow to be working on 1.16?

jlewi commented 4 years ago

Ideally we'd have a test to verify this. Is anyone willing to work on such a test?

jlewi commented 4 years ago

Can we add unittests that verify that we don't have any manifests that violate the restrictions in this issue? So all the issues @jbrette mentioned in his original comment?

I added some validation tests here: https://github.com/kubeflow/manifests/blob/master/tests/validate_resources_test.go

It should be easy to copy those tests to perform additional validations; e.g. for version.

jlewi commented 4 years ago

Downgrading to P2 and removing from 1.1 since no one is working on the unittests to verify we don't have any incompatible resources.