Docs: reference architecture for fault tolerance capabilities

[StefanoFioravanzo]

What you would like to be added?

Since @andreyvelich commented:

Unfortunately, we don't have good docs right now about our ElasticPolicy: https://github.com/kubeflow/training-operator/blob/master/pkg/apis/kubeflow.org/v1/pytorch_types.go#L98 and restart policy APIs: https://github.com/kubeflow/training-operator/blob/master/pkg/apis/kubeflow.org/v1/common_types.go#L170

We should write some reference architecture docs to expose these features to our users.

Why is this needed?

Users do not have a reference to understand and appreciate the fault tolerance capabilities offered by training operator

Love this feature?

Give it a 👍 We prioritize the features with most 👍

[andreyvelich]

Thank you for creating this @StefanoFioravanzo! /good-first-issue

[google-oss-prow[bot]]

@andreyvelich: This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed by commenting with the /remove-good-first-issue command.

In response to [this](https://github.com/kubeflow/training-operator/issues/2157): >Thank you for creating this @StefanoFioravanzo! >/good-first-issue Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

[LogicalGuy77]

Hi, new to kubeflow, would like to work on this. Couple of questions:

1. Will docs be contributed to kubeflow-website repo? (so PR will be created there?)
2. Should I create a google doc first and have it reviewed iteratively?
3. I'm new to kubeflow ecosystem so it'd be great if you can point me towards the resources/pointers to fully understand the above mentioned policies.

Thank you for your time. @andreyvelich @StefanoFioravanzo

[aryan-py]

I want to take this up, and would love any advice @StefanoFioravanzo @andreyvelich

[StefanoFioravanzo]

@LogicalGuy77 @aryan-py Thanks for stepping up!

Docs will be contributed to the Kubeflow Website, more specifically under Katib's Reference section here https://www.kubeflow.org/docs/components/katib/reference/

I would recommend starting with a google doc, especially since you are not familiar with these concepts. This will allow the project owners to review faster. You can then move the content to a PR once it's in a good state. I suggest sharing the Google Doc with the whole google discuss group, with commenter privilege (if you do that, remember to un-tick the option to notify the recipient, otherwise everyone in the Kubeflow google group will be spammed).

[andreyvelich]

Thanks for your interest @LogicalGuy77 and @aryan-py!

Yes, @StefanoFioravanzo is right, we are planning to contribute this docs to the Kubeflow website: https://github.com/kubeflow/website

Just a small correction, we should use Training Operator user-guides section to explain how various APIs work with Training Operator to achieve fault tolerance: https://www.kubeflow.org/docs/components/training/user-guides/

I would suggest to start with RestartPolicy API to handle ML training Pod restarts, and Elastic Policy API for fault-tolerant PyTorch on Kubernetes. cc @kubeflow/wg-training-leads

[StefanoFioravanzo]

@andreyvelich ops, sorry indeed we are talking about training-operator. But shouldn't this go under Reference? I think we are talking about how fault-tolerance is designed in the operator.

What kind of user guides are you thinking about?

[andreyvelich]

I guess, we can add two things:

• Explain users how they can leverage various training Job run policies API for large-scale distributed training
• Add diagrams to the reference docs to explain how does it work

[LogicalGuy77]

Update:

I've been going through lots of code and documentation and have prepared an initial draft for Restart Policy: Google Doc. I've provided commenter access to kubeflow-discuss google group. I would love to have your guidance to improve it further. I was thinking of dividing the task into three parts:

1. Restart Policy
2. Elastic Policy
3. Job Run Policies for Large-Scale Distributed Training: like clean pod and scheduling policy.

Could you elaborate more on what kind of diagrams are you looking for?

Thank you for your time. @andreyvelich @StefanoFioravanzo

[LogicalGuy77]

Hi @andreyvelich @StefanoFioravanzo,

Just wanted a follow up on this issue. I've added docs for elastic policy as well. Would you be able to take a look and provide feedback when you have a chance?

Thanks!