Open Barteus opened 1 year ago
Thanks for the input @Barteus!
Looks like this is an Istio issue, since in the kubeflow
namespace we don't have any Istio sidecars but in admin
the Istio sidecar is injected. We'll need to fully understand the communication path from the PaddlePaddle Pods and (most probably) create corresponding AuthorizationPolicies to allow some of the communication.
Or understand if the upstream Kubeflow Training Operator project never expects to run these Jobs with Istio, which is the case with Katib Experiment CRs
Or another possible issue, which I need to confirm, is whether the Pods in the same namespace can't talk to each other (again due to Istio sidecars).
If that's the case though we'll need to prioritize this more, since it can be a blocker for some use-cases.
Linking also to some issues that seem to have relevant context https://github.com/canonical/seldon-core-operator/issues/110 https://github.com/canonical/seldon-core-operator/issues/109
The istio sidecar being injected is the issue, training jobs don't work with istio sidecar injection. From the upstream documentation:
If you are using Kubeflow with Istio, you have to disable sidecar injection.
We should update our documentation to say that, too.
You can disable istio sidecar injection by running:
yq -i '.spec.paddleReplicaSpecs.Worker.template.metadata.annotations."sidecar.istio.io/inject" = "false"' simple-cpu.yaml
where simple-cpu.yaml
is the PaddleJob definition
This should then succeed in the user namespace.
After discussing with @NohaIhab we decided to submit a PR to the upstream docs to expose this detail. Noha found that this is already exposed for TFJobs, so we'll need to make sure this is exposed for PaddlePaddle as well https://www.kubeflow.org/docs/components/training/tftraining/
Ideally we'd like upstream to support running Training Jobs and Experiments alongside Istio https://github.com/kubeflow/training-operator/issues/1681 https://github.com/kubeflow/katib/issues/1638
Again, the issue is that upstream is not working with Istio sidecars. The above example mentioned in the issue was supposed to be run in the kubeflow
namespace, and without a sidecar.
To move forward, we will do the effort to include how to deploy PaddlePaddle and other jobs in non-kubeflow namespace. Removing the bug
label and marking it as a docs
issue.
Discussed this with Noha. It is expected behaviour of upstream Kubeflow to require disabling sidecar injection for any training operator jobs. This is documented in TensorFlow, but not in PaddlePaddle.
Therefore, this is a documentation issue with upstream Kubeflow.
We currently don't have any CKF docs on training operator (or PaddlePaddle). If we did, we'd need to document this (as we do in Katib)
So the only action to take for now is submit a PR to upstream docs.
Reproduce
The example used:
https://raw.githubusercontent.com/kubeflow/training-operator/master/examples/paddlepaddle/simple-cpu.yaml
Pods:
Logs from Pods in 'kubeflow' namespace:
Waiting Pods in the admin namespace: