Operator reconciling and redeploying the kfdef without known reason

Hmr-ramzi commented 4 years ago

We deployed kubeflow 1.0.2 using https://github.com/kubeflow/manifests/blob/v1.0.2/kfdef/kfctl_istio_dex.v1.0.2.yaml kfdef using the kubeflow operator and after around 45 days and without prior noticing, the kubeflow-operator suddenly redeployed the kfdef which resulted in the following:

the redeployment of the oicdservice and the cert-manager applications. which made the deployment lose the after installation configurations (for example the static usernames and passwords for dex).
the deletion of all profiles (including all of its data. etc. Namespaces were deleted, pvc and pv of jupyter notebooks were also deleted)

the redeployment of the kubeflow application happened again all of a sudden after 9 hours again for no reason.

issue-label-bot[bot] commented 4 years ago

Issue-Label Bot is automatically applying the labels:

Label	Probability
area/kfctl	0.92
kind/bug	0.82

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback! Links: app homepage, dashboard and code for this bot.

adrian555 commented 4 years ago

@Hmr-ramzi could you please provide the operator log by running kubectl logs deployment/kubeflow-operator -n <operator-namespace>? Operator reconcile routine watches over any change to the kfdef CR and the deletion of one of the resources. The log may be able to tell us if any of these events happened. Thanks.

animeshsingh commented 4 years ago

@adrian555 even if one of the events did happen - how do we ensure that after the fact artifacts -

the static usernames and passwords for dex

the deletion of all profiles including all of its data. etc. Namespaces were deleted, pvc and pv of jupyter notebooks were also deleted)

aren't wiped out? As far as I know, the operator doesn't delete any namespace it has not created?

adrian555 commented 4 years ago

the static usernames and passwords for dex

The current reconcile routine will re-apply the vanilla config map as it is part of the kustomize resource.

the deletion of all profiles including all of its data. etc. Namespaces were deleted, pvc and pv of jupyter notebooks were also deleted)

Any user profile (ie. namespace) has the ownerReferences set to its parent Profile, for example

# kubectl get ns admin -oyaml
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    owner: admin@kubeflow.org
  creationTimestamp: "2020-08-07T22:05:35Z"
  labels:
    app.kubernetes.io/part-of: kubeflow-profile
    istio-injection: enabled
    katib-metricscollector-injection: enabled
    serving.kubeflow.org/inferenceservice: enabled
  name: admin
  ownerReferences:
  - apiVersion: kubeflow.org/v1
    blockOwnerDeletion: true
    controller: true
    kind: Profile
    name: admin
    uid: f0f58baf-de56-4e10-8eb6-a4842d014381
  resourceVersion: "4089"
  selfLink: /api/v1/namespaces/admin
  uid: 1eb5d37d-3c4c-4cff-a5f0-e246b5cc3d8e

So if the Profile is deleted, the namespace and objects in the namespace will be deleted. It is not directly wiped out by the operator.

As far as I know, the operator doesn't delete any namespace it has not created?

When the operator installs the Kubeflow, the kfdef CR is inserted as the root owner of all kustomize resources. When a namespace is not part of the manifests, it should not be deleted when the kfdef CR is deleted. For example, cert-manager will be deleted as it is part of the cert-manager application's manifest https://github.com/kubeflow/manifests/blob/master/cert-manager/cert-manager/base/namespace.yaml. But kubeflow namespace should not be deleted because is not part of the manifests.

Hmr-ramzi commented 4 years ago

W0810 20:02:41.721249 1 reflector.go:302] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: watch of *unstructured.Unstructured ended with: too old resource version: 173064217 (173166386) time="2020-08-11T04:51:14Z" level=info msg="Reconciling KfDef. Request.Namespace: kubeflow, Request.Name: kubeflow." time="2020-08-11T04:51:14Z" level=info msg="Creating a new KubeFlow Deployment. KubeFlow.Namespace: kubeflow." time="2020-08-11T04:51:14Z" level=info msg="\n****************************************************************\nNotice anonymous usage reporting enabled using spartakus\nTo disable it\nIf you have already deployed it run the following commands:\n cd $(pwd)\n kubectl -n ${K8S_NAMESPACE} delete deploy -l app=spartakus\n\nFor more info: https://www.kubeflow.org/docs/other-guides/usage-reporting/\n****************************************************************\n\n" time="2020-08-11T04:51:14Z" level=info msg="Creating directory /tmp/kubeflow/kubeflow/.cache" time="2020-08-11T04:51:14Z" level=info msg="Fetching https://github.com/kubeflow/manifests/archive/v1.0.2.tar.gz to /tmp/kubeflow/kubeflow/.cache/manifests" time="2020-08-11T04:51:17Z" level=info msg="Updating localPath to /tmp/kubeflow/kubeflow/.cache/manifests/manifests-1.0.2" time="2020-08-11T04:51:17Z" level=info msg="Fetch succeeded; LocalPath /tmp/kubeflow/kubeflow/.cache/manifests/manifests-1.0.2" time="2020-08-11T04:51:17Z" level=info msg="Processing application: dex" time="2020-08-11T04:51:17Z" level=info msg="Processing application: kfserving-gateway" time="2020-08-11T04:51:17Z" level=info msg="Processing application: istio" time="2020-08-11T04:51:17Z" level=info msg="Processing application: add-anonymous-user-filter" time="2020-08-11T04:51:17Z" level=info msg="Processing application: application-crds" time="2020-08-11T04:51:17Z" level=info msg="Processing application: application" time="2020-08-11T04:51:17Z" level=info msg="Processing application: cert-manager-crds" time="2020-08-11T04:51:17Z" level=info msg="Processing application: cert-manager-kube-system-resources" time="2020-08-11T04:51:17Z" level=info msg="Processing application: cert-manager" time="2020-08-11T04:51:17Z" level=info msg="Processing application: metacontroller" time="2020-08-11T04:51:17Z" level=info msg="Processing application: argo" time="2020-08-11T04:51:17Z" level=info msg="Processing application: kubeflow-roles" time="2020-08-11T04:51:17Z" level=info msg="Processing application: centraldashboard" time="2020-08-11T04:51:17Z" level=info msg="Processing application: bootstrap" time="2020-08-11T04:51:17Z" level=info msg="Processing application: webhook" time="2020-08-11T04:51:17Z" level=info msg="Processing application: jupyter-web-app" time="2020-08-11T04:51:17Z" level=info msg="Processing application: spark-operator" time="2020-08-11T04:51:17Z" level=info msg="Processing application: metadata" time="2020-08-11T04:51:17Z" level=info msg="Processing application: notebook-controller" time="2020-08-11T04:51:17Z" level=info msg="Processing application: pytorch-job-crds" time="2020-08-11T04:51:17Z" level=info msg="Processing application: pytorch-operator" time="2020-08-11T04:51:17Z" level=info msg="Processing application: kfserving-crds" time="2020-08-11T04:51:17Z" level=info msg="Processing application: kfserving-install" time="2020-08-11T04:51:17Z" level=info msg="Processing application: spartakus" time="2020-08-11T04:51:17Z" level=info msg="Processing application: tensorboard" time="2020-08-11T04:51:17Z" level=info msg="Processing application: tf-job-crds" time="2020-08-11T04:51:17Z" level=info msg="Processing application: tf-job-operator" time="2020-08-11T04:51:17Z" level=info msg="Processing application: katib-crds" time="2020-08-11T04:51:17Z" level=info msg="Processing application: katib-controller" time="2020-08-11T04:51:17Z" level=info msg="Processing application: api-service" time="2020-08-11T04:51:17Z" level=info msg="Processing application: minio" time="2020-08-11T04:51:17Z" level=info msg="Processing application: mysql" time="2020-08-11T04:51:17Z" level=info msg="Processing application: persistent-agent" time="2020-08-11T04:51:17Z" level=info msg="Processing application: pipelines-runner" time="2020-08-11T04:51:17Z" level=info msg="Processing application: pipelines-ui" time="2020-08-11T04:51:17Z" level=info msg="Processing application: pipelines-viewer" time="2020-08-11T04:51:17Z" level=info msg="Processing application: scheduledworkflow" time="2020-08-11T04:51:17Z" level=info msg="Processing application: pipeline-visualization-service" time="2020-08-11T04:51:17Z" level=info msg="Processing application: profiles" time="2020-08-11T04:51:17Z" level=info msg="Processing application: seldon-core-operator" time="2020-08-11T04:51:17Z" level=info msg="Processing application: oidc-authservice" time="2020-08-11T04:51:17Z" level=info msg="/tmp/kubeflow/kubeflow/.cache/manifests exists; not resyncing " time="2020-08-11T04:51:17Z" level=info msg="namespace: kubeflow" time="2020-08-11T04:51:17Z" level=warning msg="Could not load config Error: open /.kube/config: no such file or directory" time="2020-08-11T04:51:17Z" level=warning msg="Unable to load .kubeconfig." time="2020-08-11T04:51:17Z" level=info msg="Deploying application dex" 2020/08/11 04:51:17 well-defined vars that were never replaced: dex_domain time="2020-08-11T04:51:17Z" level=info msg="ownerReferences added for resource auth." time="2020-08-11T04:51:17Z" level=info msg="ownerReferences added for resource authcodes.dex.coreos.com." time="2020-08-11T04:51:17Z" level=info msg="ownerReferences added for resource dex.auth" time="2020-08-11T04:51:17Z" level=info msg="ownerReferences added for resource dex." time="2020-08-11T04:51:17Z" level=info msg="ownerReferences added for resource dex." time="2020-08-11T04:51:17Z" level=info msg="ownerReferences added for resource dex.auth" time="2020-08-11T04:51:17Z" level=info msg="ownerReferences added for resource dex-parameters.auth" time="2020-08-11T04:51:17Z" level=info msg="ownerReferences added for resource dex.auth" time="2020-08-11T04:51:17Z" level=info msg="ownerReferences added for resource dex.auth" time="2020-08-11T04:51:17Z" level=info msg="ownerReferences added for resource dex.auth" namespace/auth unchanged customresourcedefinition.apiextensions.k8s.io/authcodes.dex.coreos.com unchanged serviceaccount/dex unchanged clusterrole.rbac.authorization.k8s.io/dex unchanged clusterrolebinding.rbac.authorization.k8s.io/dex unchanged configmap/dex unchanged configmap/dex-parameters configured service/dex unchanged deployment.apps/dex unchanged time="2020-08-11T04:51:17Z" level=info msg="Successfully applied application dex" time="2020-08-11T04:51:17Z" level=info msg="Deploying application kfserving-gateway" virtualservice.networking.istio.io/dex unchanged time="2020-08-11T04:51:17Z" level=info msg="ownerReferences added for resource kfserving-ingressgateway.istio-system" time="2020-08-11T04:51:17Z" level=info msg="ownerReferences added for resource kfserving-ingressgateway.istio-system" time="2020-08-11T04:51:17Z" level=info msg="ownerReferences added for resource kfserving-ingressgateway.istio-system" service/kfserving-ingressgateway unchanged deployment.apps/kfserving-ingressgateway unchanged time="2020-08-11T04:51:17Z" level=info msg="Successfully applied application kfserving-gateway" time="2020-08-11T04:51:17Z" level=info msg="Deploying application istio" horizontalpodautoscaler.autoscaling/kfserving-ingressgateway unchanged time="2020-08-11T04:51:17Z" level=info msg="ownerReferences added for resource kubeflow-istio-admin." time="2020-08-11T04:51:17Z" level=info msg="ownerReferences added for resource kubeflow-istio-edit." time="2020-08-11T04:51:17Z" level=info msg="ownerReferences added for resource kubeflow-istio-view." time="2020-08-11T04:51:17Z" level=info msg="ownerReferences added for resource istio-parameters-t6hhgfg9k2.kubeflow" time="2020-08-11T04:51:17Z" level=info msg="ownerReferences added for resource kubeflow-gateway.kubeflow" time="2020-08-11T04:51:17Z" level=info msg="ownerReferences added for resource google-api-entry.kubeflow" time="2020-08-11T04:51:17Z" level=info msg="ownerReferences added for resource google-storage-api-entry.kubeflow" time="2020-08-11T04:51:17Z" level=info msg="ownerReferences added for resource google-api-vs.kubeflow" time="2020-08-11T04:51:17Z" level=info msg="ownerReferences added for resource google-storage-api-vs.kubeflow" time="2020-08-11T04:51:17Z" level=info msg="ownerReferences added for resource grafana-vs.kubeflow" time="2020-08-11T04:51:17Z" level=info msg="ownerReferences added for resource default.kubeflow" clusterrole.rbac.authorization.k8s.io/kubeflow-istio-admin configured clusterrole.rbac.authorization.k8s.io/kubeflow-istio-edit unchanged clusterrole.rbac.authorization.k8s.io/kubeflow-istio-view unchanged configmap/istio-parameters-t6hhgfg9k2 unchanged gateway.networking.istio.io/kubeflow-gateway unchanged serviceentry.networking.istio.io/google-api-entry unchanged serviceentry.networking.istio.io/google-storage-api-entry unchanged virtualservice.networking.istio.io/google-api-vs unchanged virtualservice.networking.istio.io/google-storage-api-vs unchanged virtualservice.networking.istio.io/grafana-vs unchanged clusterrbacconfig.rbac.istio.io/default unchanged time="2020-08-11T04:51:18Z" level=info msg="Successfully applied application istio" time="2020-08-11T04:51:18Z" level=info msg="Deploying application add-anonymous-user-filter" time="2020-08-11T04:51:18Z" level=info msg="ownerReferences added for resource add-user-filter.istio-system" time="2020-08-11T04:51:18Z" level=info msg="Successfully applied application add-anonymous-user-filter" time="2020-08-11T04:51:18Z" level=info msg="Deploying application application-crds" envoyfilter.networking.istio.io/add-user-filter unchanged time="2020-08-11T04:51:18Z" level=info msg="ownerReferences added for resource applications.app.k8s.io." customresourcedefinition.apiextensions.k8s.io/applications.app.k8s.io configured time="2020-08-11T04:51:18Z" level=info msg="Successfully applied application application-crds" time="2020-08-11T04:51:18Z" level=info msg="Deploying application application" time="2020-08-11T04:51:18Z" level=info msg="ownerReferences added for resource application-controller-service-account.kubeflow" time="2020-08-11T04:51:18Z" level=info msg="ownerReferences added for resource application-controller-cluster-role." time="2020-08-11T04:51:18Z" level=info msg="ownerReferences added for resource application-controller-cluster-role-binding." time="2020-08-11T04:51:18Z" level=info msg="ownerReferences added for resource application-controller-parameters.kubeflow" time="2020-08-11T04:51:18Z" level=info msg="ownerReferences added for resource application-controller-service.kubeflow" time="2020-08-11T04:51:18Z" level=info msg="ownerReferences added for resource application-controller-stateful-set.kubeflow" time="2020-08-11T04:51:18Z" level=info msg="ownerReferences added for resource kubeflow.kubeflow" serviceaccount/application-controller-service-account unchanged clusterrole.rbac.authorization.k8s.io/application-controller-cluster-role unchanged clusterrolebinding.rbac.authorization.k8s.io/application-controller-cluster-role-binding unchanged configmap/application-controller-parameters unchanged service/application-controller-service unchanged statefulset.apps/application-controller-stateful-set configured application.app.k8s.io/kubeflow created time="2020-08-11T04:51:18Z" level=info msg="Successfully applied application application" time="2020-08-11T04:51:18Z" level=info msg="Deploying application cert-manager-crds" time="2020-08-11T04:51:18Z" level=info msg="ownerReferences added for resource certificaterequests.cert-manager.io." time="2020-08-11T04:51:18Z" level=info msg="ownerReferences added for resource certificates.cert-manager.io." time="2020-08-11T04:51:18Z" level=info msg="ownerReferences added for resource challenges.acme.cert-manager.io." time="2020-08-11T04:51:18Z" level=info msg="ownerReferences added for resource clusterissuers.cert-manager.io." time="2020-08-11T04:51:18Z" level=info msg="ownerReferences added for resource issuers.cert-manager.io." time="2020-08-11T04:51:18Z" level=info msg="ownerReferences added for resource orders.acme.cert-manager.io." customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io configured customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io configured customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io configured customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io configured customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io configured customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io configured time="2020-08-11T04:51:19Z" level=info msg="Successfully applied application cert-manager-crds" time="2020-08-11T04:51:19Z" level=info msg="Deploying application cert-manager-kube-system-resources" time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager-cainjector:leaderelection.kube-system" time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager:leaderelection.kube-system" time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager-cainjector:leaderelection.kube-system" time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager-webhook:webhook-authentication-reader.kube-system" time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager:leaderelection.kube-system" time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager-kube-params-parameters.kube-system" role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection unchanged role.rbac.authorization.k8s.io/cert-manager:leaderelection unchanged rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection configured rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:webhook-authentication-reader configured rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection configured configmap/cert-manager-kube-params-parameters unchanged time="2020-08-11T04:51:19Z" level=info msg="Successfully applied application cert-manager-kube-system-resources" time="2020-08-11T04:51:19Z" level=info msg="Deploying application cert-manager" time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager." time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager-webhook." time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager.cert-manager" time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager-cainjector.cert-manager" time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager-webhook.cert-manager" time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager-edit." time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager-view." time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager-webhook:webhook-requester." time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager-cainjector." time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager-controller-certificates." time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager-controller-challenges." time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager-controller-clusterissuers." time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager-controller-ingress-shim." time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager-controller-issuers." time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager-controller-orders." time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager-cainjector." time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager-controller-certificates." time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager-controller-challenges." time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager-controller-clusterissuers." time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager-controller-ingress-shim." time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager-controller-issuers." time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager-controller-orders." time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager-webhook:auth-delegator." time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager-parameters.cert-manager" time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager.cert-manager" time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager-webhook.cert-manager" time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager.cert-manager" time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager-cainjector.cert-manager" time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager-webhook.cert-manager" time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource v1beta1.webhook.cert-manager.io." time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager.cert-manager" time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource kubeflow-self-signing-issuer.cert-manager" time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource cert-manager-webhook." namespace/cert-manager unchanged mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook configured serviceaccount/cert-manager unchanged serviceaccount/cert-manager-cainjector unchanged serviceaccount/cert-manager-webhook unchanged clusterrole.rbac.authorization.k8s.io/cert-manager-edit unchanged clusterrole.rbac.authorization.k8s.io/cert-manager-view unchanged clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:webhook-requester unchanged clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector unchanged clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates unchanged clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges unchanged clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers unchanged clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim unchanged clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers unchanged clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders unchanged clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector unchanged clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates unchanged clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges unchanged clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers unchanged clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim unchanged clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers unchanged clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders unchanged clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:auth-delegator configured configmap/cert-manager-parameters unchanged service/cert-manager unchanged service/cert-manager-webhook unchanged deployment.apps/cert-manager unchanged deployment.apps/cert-manager-cainjector configured deployment.apps/cert-manager-webhook configured apiservice.apiregistration.k8s.io/v1beta1.webhook.cert-manager.io unchanged application.app.k8s.io/cert-manager configured clusterissuer.cert-manager.io/kubeflow-self-signing-issuer unchanged validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook configured time="2020-08-11T04:51:19Z" level=info msg="Successfully applied application cert-manager" time="2020-08-11T04:51:19Z" level=info msg="Deploying application metacontroller" time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource compositecontrollers.metacontroller.k8s.io." time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource controllerrevisions.metacontroller.k8s.io." time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource decoratorcontrollers.metacontroller.k8s.io." time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource meta-controller-service.kubeflow" time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource meta-controller-cluster-role-binding." time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource metacontroller.kubeflow" customresourcedefinition.apiextensions.k8s.io/compositecontrollers.metacontroller.k8s.io unchanged customresourcedefinition.apiextensions.k8s.io/controllerrevisions.metacontroller.k8s.io unchanged customresourcedefinition.apiextensions.k8s.io/decoratorcontrollers.metacontroller.k8s.io unchanged serviceaccount/meta-controller-service unchanged clusterrolebinding.rbac.authorization.k8s.io/meta-controller-cluster-role-binding unchanged statefulset.apps/metacontroller configured time="2020-08-11T04:51:19Z" level=info msg="Successfully applied application metacontroller" time="2020-08-11T04:51:19Z" level=info msg="Deploying application argo" time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource workflows.argoproj.io." time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource argo.kubeflow" time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource argo-ui.kubeflow" time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource argo." time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource argo-ui." time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource argo." time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource argo-ui." time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource workflow-controller-configmap.kubeflow" time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource workflow-controller-parameters.kubeflow" time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource argo-ui.kubeflow" time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource argo-ui.kubeflow" time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource workflow-controller.kubeflow" time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource argo.kubeflow" time="2020-08-11T04:51:19Z" level=info msg="ownerReferences added for resource argo-ui.kubeflow" customresourcedefinition.apiextensions.k8s.io/workflows.argoproj.io unchanged serviceaccount/argo unchanged serviceaccount/argo-ui unchanged clusterrole.rbac.authorization.k8s.io/argo unchanged clusterrole.rbac.authorization.k8s.io/argo-ui unchanged clusterrolebinding.rbac.authorization.k8s.io/argo unchanged clusterrolebinding.rbac.authorization.k8s.io/argo-ui unchanged configmap/workflow-controller-configmap unchanged configmap/workflow-controller-parameters unchanged service/argo-ui unchanged deployment.apps/argo-ui configured deployment.apps/workflow-controller configured application.app.k8s.io/argo configured time="2020-08-11T04:51:20Z" level=info msg="Successfully applied application argo" time="2020-08-11T04:51:20Z" level=info msg="Deploying application kubeflow-roles" virtualservice.networking.istio.io/argo-ui unchanged time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource kubeflow-admin." time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource kubeflow-edit." time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource kubeflow-kubernetes-admin." time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource kubeflow-kubernetes-edit." time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource kubeflow-kubernetes-view." time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource kubeflow-view." clusterrole.rbac.authorization.k8s.io/kubeflow-admin configured clusterrole.rbac.authorization.k8s.io/kubeflow-edit configured clusterrole.rbac.authorization.k8s.io/kubeflow-kubernetes-admin unchanged clusterrole.rbac.authorization.k8s.io/kubeflow-kubernetes-edit unchanged clusterrole.rbac.authorization.k8s.io/kubeflow-kubernetes-view unchanged clusterrole.rbac.authorization.k8s.io/kubeflow-view configured time="2020-08-11T04:51:20Z" level=info msg="Successfully applied application kubeflow-roles" time="2020-08-11T04:51:20Z" level=info msg="Deploying application centraldashboard" time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource centraldashboard.kubeflow" time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource centraldashboard.kubeflow" time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource centraldashboard." time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource centraldashboard.kubeflow" time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource centraldashboard." time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource parameters.kubeflow" time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource centraldashboard.kubeflow" time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource centraldashboard.kubeflow" time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource centraldashboard.kubeflow" time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource centraldashboard.kubeflow" serviceaccount/centraldashboard unchanged role.rbac.authorization.k8s.io/centraldashboard configured clusterrole.rbac.authorization.k8s.io/centraldashboard unchanged rolebinding.rbac.authorization.k8s.io/centraldashboard configured clusterrolebinding.rbac.authorization.k8s.io/centraldashboard unchanged configmap/parameters unchanged service/centraldashboard unchanged deployment.apps/centraldashboard configured application.app.k8s.io/centraldashboard unchanged virtualservice.networking.istio.io/centraldashboard unchanged time="2020-08-11T04:51:20Z" level=info msg="Successfully applied application centraldashboard" time="2020-08-11T04:51:20Z" level=info msg="Deploying application bootstrap" time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource admission-webhook-bootstrap-service-account.kubeflow" time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource admission-webhook-bootstrap-cluster-role." time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource admission-webhook-bootstrap-cluster-role-binding." time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource admission-webhook-bootstrap-config-map.kubeflow" time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource admission-webhook-bootstrap-stateful-set.kubeflow" time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource bootstrap.kubeflow" serviceaccount/admission-webhook-bootstrap-service-account unchanged clusterrole.rbac.authorization.k8s.io/admission-webhook-bootstrap-cluster-role unchanged clusterrolebinding.rbac.authorization.k8s.io/admission-webhook-bootstrap-cluster-role-binding unchanged configmap/admission-webhook-bootstrap-config-map unchanged statefulset.apps/admission-webhook-bootstrap-stateful-set configured application.app.k8s.io/bootstrap configured time="2020-08-11T04:51:20Z" level=info msg="Successfully applied application bootstrap" time="2020-08-11T04:51:20Z" level=info msg="Deploying application webhook" time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource poddefaults.kubeflow.org." time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource admission-webhook-mutating-webhook-configuration." time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource admission-webhook-service-account.kubeflow" time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource admission-webhook-cluster-role." time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource admission-webhook-kubeflow-poddefaults-admin." time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource admission-webhook-kubeflow-poddefaults-edit." time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource admission-webhook-kubeflow-poddefaults-view." time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource admission-webhook-cluster-role-binding." time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource admission-webhook-admission-webhook-parameters.kubeflow" time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource admission-webhook-service.kubeflow" time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource admission-webhook-deployment.kubeflow" time="2020-08-11T04:51:20Z" level=info msg="ownerReferences added for resource webhook.kubeflow" customresourcedefinition.apiextensions.k8s.io/poddefaults.kubeflow.org unchanged mutatingwebhookconfiguration.admissionregistration.k8s.io/admission-webhook-mutating-webhook-configuration configured serviceaccount/admission-webhook-service-account unchanged clusterrole.rbac.authorization.k8s.io/admission-webhook-cluster-role unchanged clusterrole.rbac.authorization.k8s.io/admission-webhook-kubeflow-poddefaults-admin configured clusterrole.rbac.authorization.k8s.io/admission-webhook-kubeflow-poddefaults-edit configured clusterrole.rbac.authorization.k8s.io/admission-webhook-kubeflow-poddefaults-view unchanged clusterrolebinding.rbac.authorization.k8s.io/admission-webhook-cluster-role-binding unchanged configmap/admission-webhook-admission-webhook-parameters unchanged service/admission-webhook-service unchanged deployment.apps/admission-webhook-deployment unchanged application.app.k8s.io/webhook configured time="2020-08-11T04:51:21Z" level=info msg="Successfully applied application webhook" time="2020-08-11T04:51:21Z" level=info msg="Deploying application jupyter-web-app" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource jupyter-web-app-service-account.kubeflow" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource jupyter-web-app-jupyter-notebook-role.kubeflow" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource jupyter-web-app-cluster-role." time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource jupyter-web-app-kubeflow-notebook-ui-admin." time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource jupyter-web-app-kubeflow-notebook-ui-edit." time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource jupyter-web-app-kubeflow-notebook-ui-view." time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource jupyter-web-app-jupyter-notebook-role-binding.kubeflow" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource jupyter-web-app-cluster-role-binding." time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource jupyter-web-app-config.kubeflow" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource jupyter-web-app-parameters.kubeflow" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource jupyter-web-app-service.kubeflow" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource jupyter-web-app-deployment.kubeflow" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource jupyter-web-app.kubeflow" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource jupyter-web-app.kubeflow" serviceaccount/jupyter-web-app-service-account unchanged role.rbac.authorization.k8s.io/jupyter-web-app-jupyter-notebook-role configured clusterrole.rbac.authorization.k8s.io/jupyter-web-app-cluster-role unchanged clusterrole.rbac.authorization.k8s.io/jupyter-web-app-kubeflow-notebook-ui-admin configured clusterrole.rbac.authorization.k8s.io/jupyter-web-app-kubeflow-notebook-ui-edit unchanged clusterrole.rbac.authorization.k8s.io/jupyter-web-app-kubeflow-notebook-ui-view unchanged rolebinding.rbac.authorization.k8s.io/jupyter-web-app-jupyter-notebook-role-binding configured clusterrolebinding.rbac.authorization.k8s.io/jupyter-web-app-cluster-role-binding unchanged configmap/jupyter-web-app-config unchanged configmap/jupyter-web-app-parameters unchanged service/jupyter-web-app-service unchanged deployment.apps/jupyter-web-app-deployment configured application.app.k8s.io/jupyter-web-app unchanged virtualservice.networking.istio.io/jupyter-web-app unchanged time="2020-08-11T04:51:21Z" level=info msg="Successfully applied application jupyter-web-app" time="2020-08-11T04:51:21Z" level=info msg="Deploying application spark-operator" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource scheduledsparkapplications.sparkoperator.k8s.io." time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource sparkapplications.sparkoperator.k8s.io." time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource spark-operatoroperator-sa.kubeflow" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource spark-operatorspark-sa.kubeflow" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource spark-operatoroperator-cr." time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource spark-operatorsparkoperator-crb." time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource spark-operatorsparkoperator.kubeflow" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource spark-operator.kubeflow" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource spark-operatorcrd-cleanup.kubeflow" customresourcedefinition.apiextensions.k8s.io/scheduledsparkapplications.sparkoperator.k8s.io configured customresourcedefinition.apiextensions.k8s.io/sparkapplications.sparkoperator.k8s.io configured serviceaccount/spark-operatoroperator-sa unchanged serviceaccount/spark-operatorspark-sa unchanged clusterrole.rbac.authorization.k8s.io/spark-operatoroperator-cr unchanged clusterrolebinding.rbac.authorization.k8s.io/spark-operatorsparkoperator-crb unchanged deployment.apps/spark-operatorsparkoperator configured application.app.k8s.io/spark-operator unchanged job.batch/spark-operatorcrd-cleanup unchanged time="2020-08-11T04:51:21Z" level=info msg="Successfully applied application spark-operator" time="2020-08-11T04:51:21Z" level=info msg="Deploying application metadata" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource metadata-ui.kubeflow" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource metadata-ui.kubeflow" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource metadata-ui.kubeflow" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource metadata-db-parameters.kubeflow" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource metadata-grpc-configmap.kubeflow" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource metadata-ui-parameters.kubeflow" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource metadata-db-secrets.kubeflow" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource metadata-db.kubeflow" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource metadata-envoy-service.kubeflow" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource metadata-grpc-service.kubeflow" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource metadata-service.kubeflow" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource metadata-ui.kubeflow" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource metadata-db.kubeflow" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource metadata-deployment.kubeflow" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource metadata-envoy-deployment.kubeflow" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource metadata-grpc-deployment.kubeflow" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource metadata-ui.kubeflow" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource metadata.kubeflow" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource metadata-grpc.kubeflow" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource metadata-ui.kubeflow" time="2020-08-11T04:51:21Z" level=info msg="ownerReferences added for resource metadata-mysql.kubeflow" serviceaccount/metadata-ui unchanged role.rbac.authorization.k8s.io/metadata-ui unchanged rolebinding.rbac.authorization.k8s.io/metadata-ui unchanged configmap/metadata-db-parameters unchanged configmap/metadata-grpc-configmap unchanged configmap/metadata-ui-parameters unchanged secret/metadata-db-secrets unchanged service/metadata-db unchanged service/metadata-envoy-service unchanged service/metadata-grpc-service unchanged service/metadata-service unchanged service/metadata-ui unchanged deployment.apps/metadata-db configured deployment.apps/metadata-deployment configured deployment.apps/metadata-envoy-deployment configured deployment.apps/metadata-grpc-deployment configured deployment.apps/metadata-ui configured application.app.k8s.io/metadata unchanged virtualservice.networking.istio.io/metadata-grpc unchanged virtualservice.networking.istio.io/metadata-ui unchanged time="2020-08-11T04:51:22Z" level=info msg="Successfully applied application metadata" time="2020-08-11T04:51:22Z" level=info msg="Deploying application notebook-controller" persistentvolumeclaim/metadata-mysql unchanged time="2020-08-11T04:51:22Z" level=info msg="ownerReferences added for resource notebooks.kubeflow.org." time="2020-08-11T04:51:22Z" level=info msg="ownerReferences added for resource notebook-controller-service-account.kubeflow" time="2020-08-11T04:51:22Z" level=info msg="ownerReferences added for resource notebook-controller-kubeflow-notebooks-admin." time="2020-08-11T04:51:22Z" level=info msg="ownerReferences added for resource notebook-controller-kubeflow-notebooks-edit." time="2020-08-11T04:51:22Z" level=info msg="ownerReferences added for resource notebook-controller-kubeflow-notebooks-view." time="2020-08-11T04:51:22Z" level=info msg="ownerReferences added for resource notebook-controller-role." time="2020-08-11T04:51:22Z" level=info msg="ownerReferences added for resource notebook-controller-role-binding." time="2020-08-11T04:51:22Z" level=info msg="ownerReferences added for resource notebook-controller-parameters.kubeflow" time="2020-08-11T04:51:22Z" level=info msg="ownerReferences added for resource notebook-controller-service.kubeflow" time="2020-08-11T04:51:22Z" level=info msg="ownerReferences added for resource notebook-controller-deployment.kubeflow" time="2020-08-11T04:51:22Z" level=info msg="ownerReferences added for resource notebook-controller.kubeflow" customresourcedefinition.apiextensions.k8s.io/notebooks.kubeflow.org configured serviceaccount/notebook-controller-service-account unchanged clusterrole.rbac.authorization.k8s.io/notebook-controller-kubeflow-notebooks-admin configured clusterrole.rbac.authorization.k8s.io/notebook-controller-kubeflow-notebooks-edit unchanged clusterrole.rbac.authorization.k8s.io/notebook-controller-kubeflow-notebooks-view unchanged clusterrole.rbac.authorization.k8s.io/notebook-controller-role unchanged clusterrolebinding.rbac.authorization.k8s.io/notebook-controller-role-binding unchanged configmap/notebook-controller-parameters unchanged service/notebook-controller-service unchanged deployment.apps/notebook-controller-deployment configured time="2020-08-11T04:51:22Z" level=info msg="Successfully applied application notebook-controller" application.app.k8s.io/notebook-controller unchanged time="2020-08-11T04:51:22Z" level=info msg="Deploying application pytorch-job-crds" time="2020-08-11T04:51:22Z" level=info msg="ownerReferences added for resource pytorchjobs.kubeflow.org." time="2020-08-11T04:51:22Z" level=info msg="ownerReferences added for resource pytorch-job-crds.kubeflow" customresourcedefinition.apiextensions.k8s.io/pytorchjobs.kubeflow.org unchanged application.app.k8s.io/pytorch-job-crds unchanged time="2020-08-11T04:51:22Z" level=info msg="Successfully applied application pytorch-job-crds" time="2020-08-11T04:51:22Z" level=info msg="Deploying application pytorch-operator" time="2020-08-11T04:51:22Z" level=info msg="ownerReferences added for resource pytorch-operator.kubeflow" time="2020-08-11T04:51:22Z" level=info msg="ownerReferences added for resource kubeflow-pytorchjobs-admin." time="2020-08-11T04:51:22Z" level=info msg="ownerReferences added for resource kubeflow-pytorchjobs-edit." time="2020-08-11T04:51:22Z" level=info msg="ownerReferences added for resource kubeflow-pytorchjobs-view." time="2020-08-11T04:51:22Z" level=info msg="ownerReferences added for resource pytorch-operator." time="2020-08-11T04:51:22Z" level=info msg="ownerReferences added for resource pytorch-operator." time="2020-08-11T04:51:22Z" level=info msg="ownerReferences added for resource pytorch-operator.kubeflow" time="2020-08-11T04:51:22Z" level=info msg="ownerReferences added for resource pytorch-operator.kubeflow" time="2020-08-11T04:51:22Z" level=info msg="ownerReferences added for resource pytorch-operator.kubeflow" serviceaccount/pytorch-operator unchanged clusterrole.rbac.authorization.k8s.io/kubeflow-pytorchjobs-admin configured clusterrole.rbac.authorization.k8s.io/kubeflow-pytorchjobs-edit unchanged clusterrole.rbac.authorization.k8s.io/kubeflow-pytorchjobs-view unchanged clusterrole.rbac.authorization.k8s.io/pytorch-operator unchanged clusterrolebinding.rbac.authorization.k8s.io/pytorch-operator unchanged service/pytorch-operator unchanged deployment.apps/pytorch-operator configured application.app.k8s.io/pytorch-operator unchanged time="2020-08-11T04:51:22Z" level=info msg="Successfully applied application pytorch-operator" time="2020-08-11T04:51:22Z" level=info msg="Deploying application kfserving-crds" time="2020-08-11T04:51:22Z" level=info msg="ownerReferences added for resource inferenceservices.serving.kubeflow.org." time="2020-08-11T04:51:22Z" level=info msg="ownerReferences added for resource kfserving-crds.kubeflow" customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kubeflow.org configured application.app.k8s.io/kfserving-crds configured time="2020-08-11T04:51:23Z" level=info msg="Successfully applied application kfserving-crds" time="2020-08-11T04:51:23Z" level=info msg="Deploying application kfserving-install" time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource kfserving-proxy-role." time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource kubeflow-kfserving-admin." time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource kubeflow-kfserving-edit." time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource kubeflow-kfserving-view." time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource manager-role." time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource kfserving-proxy-rolebinding." time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource manager-rolebinding." time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource inferenceservice-config.kubeflow" time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource kfserving-parameters-dbdb8cm9t2.kubeflow" time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource kfserving-webhook-server-secret.kubeflow" time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource kfserving-controller-manager-metrics-service.kubeflow" time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource kfserving-controller-manager-service.kubeflow" time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource kfserving-controller-manager.kubeflow" time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource kfserving.kubeflow" clusterrole.rbac.authorization.k8s.io/kfserving-proxy-role unchanged clusterrole.rbac.authorization.k8s.io/kubeflow-kfserving-admin configured clusterrole.rbac.authorization.k8s.io/kubeflow-kfserving-edit unchanged clusterrole.rbac.authorization.k8s.io/kubeflow-kfserving-view unchanged clusterrole.rbac.authorization.k8s.io/manager-role configured clusterrolebinding.rbac.authorization.k8s.io/kfserving-proxy-rolebinding unchanged clusterrolebinding.rbac.authorization.k8s.io/manager-rolebinding configured configmap/inferenceservice-config unchanged configmap/kfserving-parameters-dbdb8cm9t2 unchanged secret/kfserving-webhook-server-secret unchanged service/kfserving-controller-manager-metrics-service unchanged service/kfserving-controller-manager-service unchanged statefulset.apps/kfserving-controller-manager configured time="2020-08-11T04:51:23Z" level=info msg="Successfully applied application kfserving-install" time="2020-08-11T04:51:23Z" level=info msg="Deploying application spartakus" application.app.k8s.io/kfserving configured time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource spartakus.kubeflow" time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource spartakus." time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource spartakus." time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource spartakus-parameters.kubeflow" time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource spartakus-volunteer.kubeflow" time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource spartakus.kubeflow" serviceaccount/spartakus unchanged clusterrole.rbac.authorization.k8s.io/spartakus unchanged clusterrolebinding.rbac.authorization.k8s.io/spartakus unchanged configmap/spartakus-parameters unchanged deployment.apps/spartakus-volunteer configured application.app.k8s.io/spartakus configured time="2020-08-11T04:51:23Z" level=info msg="Successfully applied application spartakus" time="2020-08-11T04:51:23Z" level=info msg="Deploying application tensorboard" time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource parameters-dgd4h256h5.kubeflow" time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource tensorboard.kubeflow" time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource tensorboard.kubeflow" time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource tensorboard.kubeflow" configmap/parameters-dgd4h256h5 unchanged service/tensorboard unchanged deployment.apps/tensorboard unchanged time="2020-08-11T04:51:23Z" level=info msg="Successfully applied application tensorboard" time="2020-08-11T04:51:23Z" level=info msg="Deploying application tf-job-crds" virtualservice.networking.istio.io/tensorboard unchanged time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource tfjobs.kubeflow.org." time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource tf-job-crds.kubeflow" customresourcedefinition.apiextensions.k8s.io/tfjobs.kubeflow.org unchanged application.app.k8s.io/tf-job-crds unchanged time="2020-08-11T04:51:23Z" level=info msg="Successfully applied application tf-job-crds" time="2020-08-11T04:51:23Z" level=info msg="Deploying application tf-job-operator" time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource tf-job-dashboard.kubeflow" time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource tf-job-operator.kubeflow" time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource kubeflow-tfjobs-admin." time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource kubeflow-tfjobs-edit." time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource kubeflow-tfjobs-view." time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource tf-job-operator." time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource tf-job-operator." time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource tf-job-operator.kubeflow" time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource tf-job-operator.kubeflow" time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource tf-job-operator.kubeflow" serviceaccount/tf-job-dashboard unchanged serviceaccount/tf-job-operator unchanged clusterrole.rbac.authorization.k8s.io/kubeflow-tfjobs-admin configured clusterrole.rbac.authorization.k8s.io/kubeflow-tfjobs-edit unchanged clusterrole.rbac.authorization.k8s.io/kubeflow-tfjobs-view unchanged clusterrole.rbac.authorization.k8s.io/tf-job-operator unchanged clusterrolebinding.rbac.authorization.k8s.io/tf-job-operator unchanged service/tf-job-operator unchanged deployment.apps/tf-job-operator configured application.app.k8s.io/tf-job-operator unchanged time="2020-08-11T04:51:23Z" level=info msg="Successfully applied application tf-job-operator" time="2020-08-11T04:51:23Z" level=info msg="Deploying application katib-crds" time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource experiments.kubeflow.org." time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource suggestions.kubeflow.org." time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource trials.kubeflow.org." time="2020-08-11T04:51:23Z" level=info msg="ownerReferences added for resource katib-crds.kubeflow" customresourcedefinition.apiextensions.k8s.io/experiments.kubeflow.org unchanged customresourcedefinition.apiextensions.k8s.io/suggestions.kubeflow.org unchanged customresourcedefinition.apiextensions.k8s.io/trials.kubeflow.org unchanged time="2020-08-11T04:51:23Z" level=info msg="Successfully applied application katib-crds" time="2020-08-11T04:51:23Z" level=info msg="Deploying application katib-controller" application.app.k8s.io/katib-crds unchanged time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource katib-controller.kubeflow" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource katib-ui.kubeflow" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource katib-controller." time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource katib-ui." time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource kubeflow-katib-admin." time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource kubeflow-katib-edit." time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource kubeflow-katib-view." time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource katib-controller." time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource katib-ui." time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource katib-config.kubeflow" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource katib-parameters.kubeflow" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource trial-template.kubeflow" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource katib-controller.kubeflow" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource katib-mysql-secrets.kubeflow" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource katib-controller.kubeflow" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource katib-db-manager.kubeflow" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource katib-mysql.kubeflow" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource katib-ui.kubeflow" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource katib-controller.kubeflow" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource katib-db-manager.kubeflow" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource katib-mysql.kubeflow" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource katib-ui.kubeflow" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource katib-controller.kubeflow" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource katib-ui.kubeflow" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource katib-mysql.kubeflow" serviceaccount/katib-controller unchanged serviceaccount/katib-ui unchanged clusterrole.rbac.authorization.k8s.io/katib-controller unchanged clusterrole.rbac.authorization.k8s.io/katib-ui unchanged clusterrole.rbac.authorization.k8s.io/kubeflow-katib-admin configured clusterrole.rbac.authorization.k8s.io/kubeflow-katib-edit unchanged clusterrole.rbac.authorization.k8s.io/kubeflow-katib-view unchanged clusterrolebinding.rbac.authorization.k8s.io/katib-controller unchanged clusterrolebinding.rbac.authorization.k8s.io/katib-ui unchanged configmap/katib-config unchanged configmap/katib-parameters unchanged configmap/trial-template unchanged secret/katib-controller unchanged secret/katib-mysql-secrets unchanged service/katib-controller unchanged service/katib-db-manager unchanged service/katib-mysql unchanged service/katib-ui unchanged deployment.apps/katib-controller configured deployment.apps/katib-db-manager configured deployment.apps/katib-mysql configured deployment.apps/katib-ui configured application.app.k8s.io/katib-controller unchanged virtualservice.networking.istio.io/katib-ui unchanged persistentvolumeclaim/katib-mysql unchanged time="2020-08-11T04:51:24Z" level=info msg="Successfully applied application katib-controller" time="2020-08-11T04:51:24Z" level=info msg="Deploying application api-service" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource ml-pipeline.kubeflow" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource ml-pipeline.kubeflow" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource ml-pipeline.kubeflow" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource ml-pipeline-config.kubeflow" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource ml-pipeline.kubeflow" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource ml-pipeline.kubeflow" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource api-service.kubeflow" serviceaccount/ml-pipeline unchanged role.rbac.authorization.k8s.io/ml-pipeline unchanged rolebinding.rbac.authorization.k8s.io/ml-pipeline unchanged configmap/ml-pipeline-config unchanged service/ml-pipeline unchanged deployment.apps/ml-pipeline configured time="2020-08-11T04:51:24Z" level=info msg="Successfully applied application api-service" time="2020-08-11T04:51:24Z" level=info msg="Deploying application minio" application.app.k8s.io/api-service configured time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource pipeline-minio-parameters.kubeflow" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource mlpipeline-minio-artifact.kubeflow" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource minio-service.kubeflow" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource minio.kubeflow" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource minio.kubeflow" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource minio-pv-claim.kubeflow" configmap/pipeline-minio-parameters unchanged secret/mlpipeline-minio-artifact unchanged service/minio-service unchanged deployment.apps/minio configured application.app.k8s.io/minio configured persistentvolumeclaim/minio-pv-claim unchanged time="2020-08-11T04:51:24Z" level=info msg="Successfully applied application minio" time="2020-08-11T04:51:24Z" level=info msg="Deploying application mysql" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource pipeline-mysql-parameters.kubeflow" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource mysql.kubeflow" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource mysql.kubeflow" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource mysql.kubeflow" time="2020-08-11T04:51:24Z" level=info msg="ownerReferences added for resource mysql-pv-claim.kubeflow" configmap/pipeline-mysql-parameters unchanged service/mysql unchanged deployment.apps/mysql configured application.app.k8s.io/mysql configured persistentvolumeclaim/mysql-pv-claim unchanged time="2020-08-11T04:51:25Z" level=info msg="Successfully applied application mysql" time="2020-08-11T04:51:25Z" level=info msg="Deploying application persistent-agent" time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource ml-pipeline-persistenceagent.kubeflow" time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource ml-pipeline-persistenceagent." time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource ml-pipeline-persistenceagent." time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource ml-pipeline-persistenceagent.kubeflow" time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource persistent-agent.kubeflow" serviceaccount/ml-pipeline-persistenceagent unchanged clusterrole.rbac.authorization.k8s.io/ml-pipeline-persistenceagent unchanged clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-persistenceagent unchanged deployment.apps/ml-pipeline-persistenceagent configured time="2020-08-11T04:51:25Z" level=info msg="Successfully applied application persistent-agent" time="2020-08-11T04:51:25Z" level=info msg="Deploying application pipelines-runner" application.app.k8s.io/persistent-agent configured time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource pipeline-runner.kubeflow" time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource pipeline-runner." time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource pipeline-runner." time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource pipelines-runner.kubeflow" serviceaccount/pipeline-runner unchanged clusterrole.rbac.authorization.k8s.io/pipeline-runner unchanged clusterrolebinding.rbac.authorization.k8s.io/pipeline-runner unchanged application.app.k8s.io/pipelines-runner configured time="2020-08-11T04:51:25Z" level=info msg="Successfully applied application pipelines-runner" time="2020-08-11T04:51:25Z" level=info msg="Deploying application pipelines-ui" time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource ml-pipeline-ui.kubeflow" time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource ml-pipeline-ui.kubeflow" time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource ml-pipeline-ui.kubeflow" time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource ui-parameters-hb792fcf5d.kubeflow" time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource ml-pipeline-tensorboard-ui.kubeflow" time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource ml-pipeline-ui.kubeflow" time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource ml-pipeline-ui.kubeflow" time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource pipelines-ui.kubeflow" time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource ml-pipeline-tensorboard-ui.kubeflow" time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource ml-pipeline-ui.kubeflow" serviceaccount/ml-pipeline-ui unchanged role.rbac.authorization.k8s.io/ml-pipeline-ui unchanged rolebinding.rbac.authorization.k8s.io/ml-pipeline-ui unchanged configmap/ui-parameters-hb792fcf5d unchanged service/ml-pipeline-tensorboard-ui unchanged service/ml-pipeline-ui unchanged deployment.apps/ml-pipeline-ui configured application.app.k8s.io/pipelines-ui configured virtualservice.networking.istio.io/ml-pipeline-tensorboard-ui unchanged virtualservice.networking.istio.io/ml-pipeline-ui unchanged time="2020-08-11T04:51:25Z" level=info msg="Successfully applied application pipelines-ui" time="2020-08-11T04:51:25Z" level=info msg="Deploying application pipelines-viewer" time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource viewers.kubeflow.org." time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource ml-pipeline-viewer-crd-service-account.kubeflow" time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource ml-pipeline-viewer-kubeflow-pipeline-viewers-admin." time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource ml-pipeline-viewer-kubeflow-pipeline-viewers-edit." time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource ml-pipeline-viewer-kubeflow-pipeline-viewers-view." time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource ml-pipeline-viewer-controller-role." time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource ml-pipeline-viewer-crd-role-binding." time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource ml-pipeline-viewer-controller-deployment.kubeflow" time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource pipelines-viewer.kubeflow" customresourcedefinition.apiextensions.k8s.io/viewers.kubeflow.org unchanged serviceaccount/ml-pipeline-viewer-crd-service-account unchanged clusterrole.rbac.authorization.k8s.io/ml-pipeline-viewer-kubeflow-pipeline-viewers-admin configured clusterrole.rbac.authorization.k8s.io/ml-pipeline-viewer-kubeflow-pipeline-viewers-edit unchanged clusterrole.rbac.authorization.k8s.io/ml-pipeline-viewer-kubeflow-pipeline-viewers-view unchanged clusterrole.rbac.authorization.k8s.io/ml-pipeline-viewer-controller-role unchanged clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-viewer-crd-role-binding unchanged deployment.apps/ml-pipeline-viewer-controller-deployment configured application.app.k8s.io/pipelines-viewer configured time="2020-08-11T04:51:25Z" level=info msg="Successfully applied application pipelines-viewer" time="2020-08-11T04:51:25Z" level=info msg="Deploying application scheduledworkflow" time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource scheduledworkflows.kubeflow.org." time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource ml-pipeline-scheduledworkflow.kubeflow" time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource ml-pipeline-scheduledworkflow.kubeflow" time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource kubeflow-scheduledworkflows-admin." time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource kubeflow-scheduledworkflows-edit." time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource kubeflow-scheduledworkflows-view." time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource ml-pipeline-scheduledworkflow." time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource ml-pipeline-scheduledworkflow.kubeflow" time="2020-08-11T04:51:25Z" level=info msg="ownerReferences added for resource scheduledworkflow.kubeflow" customresourcedefinition.apiextensions.k8s.io/scheduledworkflows.kubeflow.org unchanged serviceaccount/ml-pipeline-scheduledworkflow unchanged role.rbac.authorization.k8s.io/ml-pipeline-scheduledworkflow unchanged clusterrole.rbac.authorization.k8s.io/kubeflow-scheduledworkflows-admin configured clusterrole.rbac.authorization.k8s.io/kubeflow-scheduledworkflows-edit unchanged clusterrole.rbac.authorization.k8s.io/kubeflow-scheduledworkflows-view unchanged clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-scheduledworkflow unchanged deployment.apps/ml-pipeline-scheduledworkflow configured application.app.k8s.io/scheduledworkflow configured time="2020-08-11T04:51:26Z" level=info msg="Successfully applied application scheduledworkflow" time="2020-08-11T04:51:26Z" level=info msg="Deploying application pipeline-visualization-service" time="2020-08-11T04:51:26Z" level=info msg="ownerReferences added for resource ml-pipeline-ml-pipeline-visualizationserver.kubeflow" time="2020-08-11T04:51:26Z" level=info msg="ownerReferences added for resource ml-pipeline-ml-pipeline-visualizationserver.kubeflow" time="2020-08-11T04:51:26Z" level=info msg="ownerReferences added for resource pipeline-visualization-service.kubeflow" service/ml-pipeline-ml-pipeline-visualizationserver unchanged deployment.apps/ml-pipeline-ml-pipeline-visualizationserver configured application.app.k8s.io/pipeline-visualization-service configured time="2020-08-11T04:51:26Z" level=info msg="Successfully applied application pipeline-visualization-service" time="2020-08-11T04:51:26Z" level=info msg="Deploying application profiles" time="2020-08-11T04:51:26Z" level=info msg="ownerReferences added for resource profiles.kubeflow.org." time="2020-08-11T04:51:26Z" level=info msg="ownerReferences added for resource profiles-controller-service-account.kubeflow" time="2020-08-11T04:51:26Z" level=info msg="ownerReferences added for resource profiles-cluster-role-binding." time="2020-08-11T04:51:26Z" level=info msg="ownerReferences added for resource profiles-profiles-parameters-5c86m8kfb8.kubeflow" time="2020-08-11T04:51:26Z" level=info msg="ownerReferences added for resource profiles-kfam.kubeflow" time="2020-08-11T04:51:26Z" level=info msg="ownerReferences added for resource profiles-deployment.kubeflow" time="2020-08-11T04:51:26Z" level=info msg="ownerReferences added for resource profiles.kubeflow" time="2020-08-11T04:51:26Z" level=info msg="ownerReferences added for resource kfam.kubeflow" customresourcedefinition.apiextensions.k8s.io/profiles.kubeflow.org configured serviceaccount/profiles-controller-service-account unchanged clusterrolebinding.rbac.authorization.k8s.io/profiles-cluster-role-binding unchanged configmap/profiles-profiles-parameters-5c86m8kfb8 unchanged service/profiles-kfam unchanged deployment.apps/profiles-deployment configured application.app.k8s.io/profiles configured virtualservice.networking.istio.io/kfam unchanged time="2020-08-11T04:51:26Z" level=info msg="Successfully applied application profiles" time="2020-08-11T04:51:26Z" level=info msg="Deploying application seldon-core-operator" time="2020-08-11T04:51:26Z" level=info msg="ownerReferences added for resource seldondeployments.machinelearning.seldon.io." time="2020-08-11T04:51:26Z" level=info msg="ownerReferences added for resource seldon-mutating-webhook-configuration-kubeflow." time="2020-08-11T04:51:26Z" level=info msg="ownerReferences added for resource seldon-manager.kubeflow" time="2020-08-11T04:51:26Z" level=info msg="ownerReferences added for resource seldon-leader-election-role.kubeflow" time="2020-08-11T04:51:26Z" level=info msg="ownerReferences added for resource seldon-manager-cm-role.kubeflow" time="2020-08-11T04:51:26Z" level=info msg="ownerReferences added for resource seldon-manager-role-kubeflow." time="2020-08-11T04:51:26Z" level=info msg="ownerReferences added for resource seldon-manager-sas-role-kubeflow." time="2020-08-11T04:51:26Z" level=info msg="ownerReferences added for resource seldon-leader-election-rolebinding.kubeflow" time="2020-08-11T04:51:26Z" level=info msg="ownerReferences added for resource seldon-manager-cm-rolebinding.kubeflow" time="2020-08-11T04:51:26Z" level=info msg="ownerReferences added for resource seldon-manager-rolebinding-kubeflow." time="2020-08-11T04:51:26Z" level=info msg="ownerReferences added for resource seldon-manager-sas-rolebinding-kubeflow." time="2020-08-11T04:51:26Z" level=info msg="ownerReferences added for resource seldon-config.kubeflow" time="2020-08-11T04:51:26Z" level=info msg="ownerReferences added for resource seldon-webhook-service.kubeflow" time="2020-08-11T04:51:26Z" level=info msg="ownerReferences added for resource seldon-controller-manager.kubeflow" time="2020-08-11T04:51:26Z" level=info msg="ownerReferences added for resource seldon-core-operator.kubeflow" time="2020-08-11T04:51:26Z" level=info msg="ownerReferences added for resource seldon-serving-cert.kubeflow" time="2020-08-11T04:51:26Z" level=info msg="ownerReferences added for resource seldon-selfsigned-issuer.kubeflow" time="2020-08-11T04:51:26Z" level=info msg="ownerReferences added for resource seldon-validating-webhook-configuration-kubeflow." customresourcedefinition.apiextensions.k8s.io/seldondeployments.machinelearning.seldon.io configured mutatingwebhookconfiguration.admissionregistration.k8s.io/seldon-mutating-webhook-configuration-kubeflow configured serviceaccount/seldon-manager unchanged role.rbac.authorization.k8s.io/seldon-leader-election-role unchanged role.rbac.authorization.k8s.io/seldon-manager-cm-role configured clusterrole.rbac.authorization.k8s.io/seldon-manager-role-kubeflow configured clusterrole.rbac.authorization.k8s.io/seldon-manager-sas-role-kubeflow configured rolebinding.rbac.authorization.k8s.io/seldon-leader-election-rolebinding unchanged rolebinding.rbac.authorization.k8s.io/seldon-manager-cm-rolebinding unchanged clusterrolebinding.rbac.authorization.k8s.io/seldon-manager-rolebinding-kubeflow unchanged clusterrolebinding.rbac.authorization.k8s.io/seldon-manager-sas-rolebinding-kubeflow unchanged configmap/seldon-config unchanged service/seldon-webhook-service unchanged deployment.apps/seldon-controller-manager configured application.app.k8s.io/seldon-core-operator configured certificate.cert-manager.io/seldon-serving-cert unchanged issuer.cert-manager.io/seldon-selfsigned-issuer unchanged validatingwebhookconfiguration.admissionregistration.k8s.io/seldon-validating-webhook-configuration-kubeflow configured time="2020-08-11T04:51:27Z" level=info msg="Successfully applied application seldon-core-operator" time="2020-08-11T04:51:27Z" level=info msg="Deploying application oidc-authservice" time="2020-08-11T04:51:27Z" level=info msg="ownerReferences added for resource oidc-authservice-parameters.istio-system" time="2020-08-11T04:51:27Z" level=info msg="ownerReferences added for resource authservice.istio-system" time="2020-08-11T04:51:27Z" level=info msg="ownerReferences added for resource authservice.istio-system" time="2020-08-11T04:51:27Z" level=info msg="ownerReferences added for resource oidc-authservice.istio-system" time="2020-08-11T04:51:27Z" level=info msg="ownerReferences added for resource authn-filter.istio-system" time="2020-08-11T04:51:27Z" level=info msg="ownerReferences added for resource authservice-pvc.istio-system" configmap/oidc-authservice-parameters unchanged service/authservice unchanged statefulset.apps/authservice configured application.app.k8s.io/oidc-authservice unchanged envoyfilter.networking.istio.io/authn-filter unchanged persistentvolumeclaim/authservice-pvc unchanged time="2020-08-11T04:51:27Z" level=info msg="Successfully applied application oidc-authservice" time="2020-08-11T04:51:27Z" level=info msg="KubeFlow Deployment Completed."

Hmr-ramzi commented 4 years ago

@adrian555 Thank you for your reply. I added the logs in a previous commit. The main problem that i observed regarding the profiles is that the custom resource definition of "profiles.kubeflow.org" somehow on my kubernetes cluster was created at the same time of the reconciling. Is there any chance that the operator deletes the crd of recreates it somehow ?

adrian555 commented 4 years ago

@Hmr-ramzi thanks for uploading the log. From the log, there was only one resource been created: application.app.k8s.io/kubeflow created. Given that the operator is watching over the deletion event for these resources, it is fair to assume that the reconciling routine was kicked in due to the kubeflow Application object been deleted. Other resources are either configured or unchanged, from operator perspective, it seemed to just re-apply the vanilla configuration to the deployed objects, which is what the operator expected to work.

Is this log all you have from the kubectl logs deployment/kubeflow-operator command? If it happens you can retrieve older logs from the operator deployment, we may be able to figure out more events that happened. BTW, if you have more logs, please put in a google doc and share me the link. It will be easier to read. :)

And wrt the Profile crd, as shown in the log customresourcedefinition.apiextensions.k8s.io/profiles.kubeflow.org configured, it was only configured at time="2020-08-11T04:51:26Z". So I would only expect that the generation changed but not the creationTimeStamp.

A couple of things to verify: 1) run kubectl get kfdef --all-namespaces to see whether you have one or more KfDef CRs. Also look at the KfDef CR's creationTimeStamp to see whether it matched the time you first deployed the Kubeflow. If not, it might somehow have been deleted and recreated. 2) run kubeflow get deployment/kubeflow-operator --all-namespaces to see whether you have one or more Kubeflow operators running.

Hmr-ramzi commented 4 years ago

@adrian555 Thank you for your reply.

Unfortunately, i don't have access to the previous logs currently but here are points that are worth mentioning:

I saw in the logs at the time of the problem that the profile application is of status "Created" not configured or unchanged.
The timestamp (5 days) of the creation of the customresoucedefinition profiles is matching the timestamp where the problem started not when i deployed the system 50 days ago.
kubectl get kfdef --all-namespaces only has one kfdef which of timestamp 50 days ago
kubeflow get deployment/kubeflow-operator --all-namespaces (I think you are referring to the operator instance). I made sure i only have one operator instance.
I just realized that the problem was not just with profiles. The whole kubeflow was redeployed . All my pipelines, experiments are also gone.
The namespace kubeflow has age 50 days ago. However, all the resources in that namespace have timestamp 5 days ago where the problem started.
The time stamp of the kfdef is 50 days ago.

Is it by any chance that the operator deleted these resources for any reason? The only logical reason that i know of for the operator to redeploy is that some entity tried to recreate the kfdef. However, the kfdef is still with creation timestamp 50 days ago. So i am struggling now to know what caused this vanilla redeployment of all components.

Any ideas?

Thanks and Best Regards, Hisham

adrian555 commented 4 years ago

@Hmr-ramzi I was hoping that we could have some previous logs to tell us more. If the kfdef CR is up for 50 days, this becomes more difficult to seek a possible reason.

Several things to verify further:

Could you please run kubectl get kfdef kubeflow -n kubeflow -o yaml > kfdef.yaml and post the content here?
Also which branch of operator code did you run to install the Kubeflow operator?
And could you please check whether the metadata.ownerReferences.uid of any kubeflow applications (for example, deployment/ml-pipeline) matches the current kfdef's uid?
What is the life time for the cert-manager namespace? I would assume it is only 5 days old?
Is there any pod/deployment in the kubeflow namespace created 50 days ago? Want to figure out if the kubeflow namespace where the kfdef instance resides is special.
Has any event happened to your Kubernetes cluster, such as restart of master node etc?

The reason we added the ownerReference to Kubeflow applications is to depend on the Kubernetes garbage collector to handle the instance tear-down. And therefore, inside the operator code, including the manager, watcher or reconcile, there is no a single line directly deletes a Kubeflow resource.

Hmr-ramzi commented 4 years ago

apiVersion: v1 items:

apiVersion: kfdef.apps.kubeflow.org/v1 kind: KfDef metadata: annotations: meta.helm.sh/release-name: kubeflow meta.helm.sh/release-namespace: kubeflow creationTimestamp: "2020-06-22T14:57:33Z" finalizers:
- kfdef-finalizer.kfdef.apps.kubeflow.org generation: 1 labels: app: raw app.kubernetes.io/managed-by: Helm chart: raw-0.2.3 heritage: Helm release: kubeflow name: kubeflow namespace: kubeflow resourceVersion: "66439087" selfLink: /apis/kfdef.apps.kubeflow.org/v1/namespaces/kubeflow/kfdefs/kubeflow uid: 7609c45c-ca41-497e-ba83-4167f1a71104 spec: applications:
- kustomizeConfig: overlays:
  - istio parameters:
  - name: namespace value: auth
  - name: issuer value: http://dex.auth.svc.cluster.local:5556/dex
  - name: client_id value: kubeflow-oidc-authservice
  - name: oidc_redirect_uris value: '["/login/oidc"]'
  - name: static_email repoRef: name: manifests path: dex-auth/dex-crds name: dex
- kustomizeConfig: parameters:
  - name: namespace value: istio-system repoRef: name: manifests path: istio/kfserving-gateway name: kfserving-gateway
- kustomizeConfig: parameters:
  - name: clusterRbacConfig value: "OFF" repoRef: name: manifests path: istio/istio name: istio
- kustomizeConfig: parameters:
  - name: namespace value: istio-system repoRef: name: manifests path: istio/add-anonymous-user-filter name: add-anonymous-user-filter
- kustomizeConfig: repoRef: name: manifests path: application/application-crds name: application-crds
- kustomizeConfig: overlays:
  - application repoRef: name: manifests path: application/application name: application
- kustomizeConfig: parameters:
  - name: namespace value: cert-manager repoRef: name: manifests path: cert-manager/cert-manager-crds name: cert-manager-crds
- kustomizeConfig: parameters:
  - name: namespace value: kube-system repoRef: name: manifests path: cert-manager/cert-manager-kube-system-resources name: cert-manager-kube-system-resources
- kustomizeConfig: overlays:
  - self-signed
  - application parameters:
  - name: namespace value: cert-manager repoRef: name: manifests path: cert-manager/cert-manager name: cert-manager
- kustomizeConfig: repoRef: name: manifests path: metacontroller name: metacontroller
- kustomizeConfig: overlays:
  - istio
  - application repoRef: name: manifests path: argo name: argo
- kustomizeConfig: repoRef: name: manifests path: kubeflow-roles name: kubeflow-roles
- kustomizeConfig: overlays:
  - istio
  - application repoRef: name: manifests path: common/centraldashboard name: centraldashboard
- kustomizeConfig: overlays:
  - application repoRef: name: manifests path: admission-webhook/bootstrap name: bootstrap
- kustomizeConfig: overlays:
  - application repoRef: name: manifests path: admission-webhook/webhook name: webhook
- kustomizeConfig: overlays:
  - istio
  - application parameters:
  - name: userid-header value: kubeflow-userid repoRef: name: manifests path: jupyter/jupyter-web-app name: jupyter-web-app
- kustomizeConfig: overlays:
  - application repoRef: name: manifests path: spark/spark-operator name: spark-operator
- kustomizeConfig: overlays:
  - istio
  - application
  - db repoRef: name: manifests path: metadata name: metadata
- kustomizeConfig: overlays:
  - istio
  - application repoRef: name: manifests path: jupyter/notebook-controller name: notebook-controller
- kustomizeConfig: overlays:
  - application repoRef: name: manifests path: pytorch-job/pytorch-job-crds name: pytorch-job-crds
- kustomizeConfig: overlays:
  - application repoRef: name: manifests path: pytorch-job/pytorch-operator name: pytorch-operator
- kustomizeConfig: overlays:
  - application repoRef: name: manifests path: kfserving/kfserving-crds name: kfserving-crds
- kustomizeConfig: overlays:
  - application repoRef: name: manifests path: kfserving/kfserving-install name: kfserving-install
- kustomizeConfig: overlays:
  - application parameters:
  - name: usageId value:
  - name: reportUsage value: "true" repoRef: name: manifests path: common/spartakus name: spartakus
- kustomizeConfig: overlays:
  - istio repoRef: name: manifests path: tensorboard name: tensorboard
- kustomizeConfig: overlays:
  - application repoRef: name: manifests path: tf-training/tf-job-crds name: tf-job-crds
- kustomizeConfig: overlays:
  - application repoRef: name: manifests path: tf-training/tf-job-operator name: tf-job-operator
- kustomizeConfig: overlays:
  - application repoRef: name: manifests path: katib/katib-crds name: katib-crds
- kustomizeConfig: overlays:
  - application
  - istio repoRef: name: manifests path: katib/katib-controller name: katib-controller
- kustomizeConfig: overlays:
  - application repoRef: name: manifests path: pipeline/api-service name: api-service
- kustomizeConfig: overlays:
  - application parameters:
  - name: minioPvcName value: minio-pv-claim repoRef: name: manifests path: pipeline/minio name: minio
- kustomizeConfig: overlays:
  - application parameters:
  - name: mysqlPvcName value: mysql-pv-claim repoRef: name: manifests path: pipeline/mysql name: mysql
- kustomizeConfig: overlays:
  - application repoRef: name: manifests path: pipeline/persistent-agent name: persistent-agent
- kustomizeConfig: overlays:
  - application repoRef: name: manifests path: pipeline/pipelines-runner name: pipelines-runner
- kustomizeConfig: overlays:
  - istio
  - application repoRef: name: manifests path: pipeline/pipelines-ui name: pipelines-ui
- kustomizeConfig: overlays:
  - application repoRef: name: manifests path: pipeline/pipelines-viewer name: pipelines-viewer
- kustomizeConfig: overlays:
  - application repoRef: name: manifests path: pipeline/scheduledworkflow name: scheduledworkflow
- kustomizeConfig: overlays:
  - application repoRef: name: manifests path: pipeline/pipeline-visualization-service name: pipeline-visualization-service
- kustomizeConfig: overlays:
  - application
  - istio parameters:
  - name: admin value: johnDoe@acme.com repoRef: name: manifests path: profiles name: profiles
- kustomizeConfig: overlays:
  - application repoRef: name: manifests path: seldon/seldon-core-operator name: seldon-core-operator
- kustomizeConfig: overlays:
  - application parameters:
  - name: namespace value: istio-system
  - name: userid-header value: kubeflow-userid
  - name: oidc_provider value: http://dex.auth.svc.cluster.local:5556/dex
  - name: oidc_redirect_uri value: /login/oidc
  - name: oidc_auth_url value: /dex/auth
  - name: skip_auth_uri value: /dex
  - name: client_id value: kubeflow-oidc-authservice repoRef: name: manifests path: istio/oidc-authservice name: oidc-authservice repos:
- name: manifests uri: https://github.com/kubeflow/manifests/archive/v1.0.2.tar.gz version: v1.0.2 kind: List metadata: resourceVersio

Hmr-ramzi commented 4 years ago

@adrian555 Please, find the kfdef used above.
I used release 1.0.2 of the operator.
I believe it is the same id
Yup 5 days old (today 6 days old)
Nop. All of them created 5 days ago
We deploy our cluster on azure so we basically don't get to control or restart our nodes.

adrian555 commented 4 years ago

Thanks @Hmr-ramzi.

Based on the diag info we can get so far, we did not find the evidence that the Kubeflow deployment has been deleted and namespaces were deleted because of that. The operator reconciling routine did respond to resources been deleted and re-deploy Kubeflow.

While trying to figure out why the namespaces were deleted, we found a blog linking the garbage collection with the kube-controller-manager pod been restarted. The blog referenced the garbage collection doc that cross-namespace owner references are disallowed and found that when that pod restarted, the namespaces were deleted. The blog did not go further why the namespaces were deleted. But since the current operator code applies the namespace scoped kfdef CR to resources in other namespaces, this might have also happened.

We tried to repro on minikube and openshift clusters where we have the control to restart the master node, but unfortunately we could not reproduce the problem. So we suspect this may happen on a particular version of Kubernetes and/or certain cloud Kubernetes services. What Kubernetes version is your cluster running on?

Since the official garbage collection document disallows the use of cross-namespace ownerreferences, we are thinking of proactively modifying the implementation, just like other projects are starting to do so. For example, using custom labels or annotations besides the ownerReference could be one solution.

Hmr-ramzi commented 4 years ago

@adrian555 Thanks Adrian for the deep investigation. Very much appreciated!

We are using an azure environment with kubernetes server version 1.16.9.

Hmr-ramzi commented 4 years ago

@adrian555 Do you know if there is a way to get through this and make sure it does not happen again on version kubeflow and kfctl 1.0.2? Maybe removing the operator after the installtion of the kfdef application? or maybe even applying some post installation precautions ? Maybe patching some namespaces to remove some cross referencing in order to avoid the issue till the change is implemented ?

animeshsingh commented 4 years ago

thanks @adrian555 for detailed analysis. @Hmr-ramzi he is also documenting some possible steps to take in operator codebase https://docs.google.com/document/d/1VBcf0MKebZiwpxnzXwBiEDFCVOEtTFJpzX7i-wLIXLE/edit?usp=sharing

cc @vpavlin

adrian555 commented 4 years ago

@Hmr-ramzi I hope we know more about the direct cause these namespaces were deleted. If it is related to garbage collection, removing the operator after the deployment won't help. However, it can prove that it is not the operator deletes the resources. Also it will prevent the re-deployment with the vanilla configurations which you do not want. So I would go with this short term measure for now.

And from your response above that all resources in the kubeflow namespace are also re-created, it is hard to reason that the impact is only on those namespaces other than kubeflow. So patching those namespaces to remove the ownerReferences may not avoid the issue totally. Since we suspect the combination of cross-namespace ownerReferences and the master pod restart could be the culprit (but we have not confirmed yet), keeping the ownerReferences may help if it happens again.

BTW, does your azure cluster have some diagnosis logging enabled? If not, maybe you could enable it and next time this happens we can have more diagnosis info? Here is a link for setting and viewing logs https://docs.microsoft.com/en-us/azure/aks/view-master-logs.

aslom commented 4 years ago

I have the same issues happening in openshift 4.4 in IBM cloud. It seems ot be triggered by upgrade of master node 4.4.11_1511 --> 4.4.17_1515_openshift (pending)

I could see that pods were restarted

 oc get pods --namespace=kubeflow
NAME                                   READY   STATUS    RESTARTS   AGE
argo-ui-5b5dc4f45c-tmfhk               1/1     Running   0          76s
centraldashboard-69fcb6c867-lf6mp      1/1     Running   0          74s
kubeflow-operator-687fd7c966-x8hw6     1/1     Running   0          41h
workflow-controller-6645854dbb-v79jj   1/1     Running   0          76s

and eventually

oc get pods --namespace=kubeflow
NAME                                                           READY   STATUS    RESTARTS   AGE
argo-ui-5b5dc4f45c-tmfhk                                       1/1     Running   0          8m26s
centraldashboard-69fcb6c867-lf6mp                              1/1     Running   0          8m24s
jupyter-web-app-deployment-6fbbf58dbd-lbknx                    1/1     Running   0          5m44s
katib-controller-8f59dff56-5wbwf                               1/1     Running   1          5m32s
katib-db-manager-67c7476896-pdlqs                              1/1     Running   1          5m32s
katib-mysql-66df8d4564-fc62x                                   1/1     Running   0          5m32s
katib-ui-6c97bf6c98-vjqwb                                      1/1     Running   0          5m32s
kubeflow-operator-687fd7c966-x8hw6                             1/1     Running   0          41h
metadata-db-bd9c4fc94-4kpbv                                    1/1     Running   0          5m42s
metadata-deployment-664d84664c-pz82g                           1/1     Running   0          5m42s
metadata-envoy-deployment-69766744b5-67p6c                     1/1     Running   0          5m42s
metadata-grpc-deployment-76fb558d75-76v76                      1/1     Running   5          5m42s
metadata-ui-57f9b8d667-5hgxx                                   1/1     Running   0          5m42s
minio-784784b9bb-kcj6x                                         1/1     Running   0          5m29s
ml-pipeline-568cf4d8d-qlnbs                                    1/1     Running   0          5m30s
ml-pipeline-ml-pipeline-visualizationserver-5c5b674b9c-c25xs   1/1     Running   0          5m12s
ml-pipeline-persistenceagent-99f744f78-djx8p                   1/1     Running   1          5m27s
ml-pipeline-scheduledworkflow-5c657bfbf6-6hj6h                 1/1     Running   0          5m13s
ml-pipeline-ui-65f65bcc7f-b826k                                1/1     Running   0          5m18s
ml-pipeline-viewer-controller-deployment-fd8777966-nsrwf       1/1     Running   0          5m16s
mysql-bbbb6f6b5-9rf7k                                          1/1     Running   0          5m28s
notebook-controller-deployment-7f75f68b59-7lcc9                1/1     Running   0          5m40s
profiles-deployment-5575d8bbb5-gsb2t                           2/2     Running   0          5m11s
pytorch-operator-7b685b5f6c-xcv2f                              1/1     Running   0          5m38s
seldon-controller-manager-74ff8889dc-grwxd                     1/1     Running   0          5m8s
tensorboard-6549cd78c9-jzbbk                                   1/1     Running   0          5m37s
tf-job-operator-5f7d85c579-cj9xs                               1/1     Running   0          5m35s
workflow-controller-6645854dbb-v79jj                           1/1     Running   0          8m26s

all new except kubeflow-operator-687fd7c966-x8hw6

and the user namespace that containe notebooks was completely cleaned

k -n anonymous get all
No resources found in anonymous namespace.

I am attchng log kubectl logs deployment/kubeflow-operator -n kubeflow|less > log-kubeflow-operator1.txt log-kubeflow-operator1.txt

adrian555 commented 4 years ago

Thanks @aslom and @Hmr-ramzi . So now I have a reliable way to reproduce this and I have been digging more and trying different tests on this garbage collection related issue. This open issue https://github.com/kubernetes/kubernetes/issues/65200 in the Kubernetes project likely explains what happens when the master is updated. Some sorts of fix or checking is expected for K8S 1.20 from that issue's comment. So based on that, I will have to drop the ownerReferences approach in our operator. I am working on a fix.

Hmr-ramzi commented 3 years ago

@adrian555 Is there any update regarding the refactoring in order to prevent the issue from happening again ?

adrian555 commented 3 years ago

@Hmr-ramzi yes, the PR https://github.com/kubeflow/kfctl/pull/411 is waiting to be merged and should resolve this issue.

Hmr-ramzi commented 3 years ago

Thanks @adrian555 for your effort! Very much appreciated!

kubeflow / kfctl

Operator reconciling and redeploying the kfdef without known reason #393