TfJob creation failed due to webhook validation failure

[nagar-ajay]

What happened?

• Tried creating a TfJob using the following manifest:

  ❯ kubectl apply -n kubeflow-user-example-com -f - <<EOF
  apiVersion: kubeflow.org/v1
  kind: TFJob
  metadata:
  name: tfjob-simple
  spec:
  tfReplicaSpecs:
  PS:
    replicas: 1
    restartPolicy: OnFailure
    template:
      metadata:
        annotations:
          sidecar.istio.io/inject: "false"
      spec:
        containers:
          - name: tensorflow
            image: kubeflow/tf-mnist-with-summaries:latest
            command:
              - "python"
              - "/var/tf_mnist/mnist_with_summaries.py"
  Worker:
    replicas: 2
    restartPolicy: OnFailure
    template:
      metadata:
        annotations:
          sidecar.istio.io/inject: "false"
      spec:
        containers:
          - name: tensorflow
            image: kubeflow/tf-mnist-with-summaries:latest
            command:
              - "python"
              - "/var/tf_mnist/mnist_with_summaries.py"
  EOF

❯ k get ValidatingWebhookConfiguration validator.tfjob.training-operator.kubeflow.org -o yaml
Error from server (NotFound): validatingwebhookconfigurations.admissionregistration.k8s.io "validator.tfjob.training-operator.kubeflow.org" not found

Not sure but there might be some flakiness in the training-operator kubeflow mode installation.

What did you expect to happen?

• TfJob should have been created but create failed with the following error

  Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "validator.tfjob.training-operator.kubeflow.org": failed to call webhook: Post "[https://training-operator.kubeflow.svc:443/validate-kubeflow-org-v1-tfjob?timeout=10s](https://training-operator.kubeflow.svc/validate-kubeflow-org-v1-tfjob?timeout=10s)": context deadline exceeded

Environment

• Installed kubeflow 1.9-rc1 using the following command present in kubeflow/manifest repo.

  while ! kustomize build example | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 20; done