Create a reference guide for Kubeflow Pipelines

[sarahmaddox]

We need to create and publish an SDK reference for Kubeflow Pipelines. The API info is too scattered at the moment. For example:

• The Build your own component page mentions kfp.dsl.ContainerOp.
• The Lightweight Python components page mentions kfp.components.func_to_container_op(func).

It's good to have tutorials like the above. But as a developer, when I want to see everything I can do with kfp.dsl.ContainerOp or with kfp.dsl, etc, I need a reference.

Ideally we should auto-generate the reference from comments in the code files. Of course, this isn't something we can do quickly, but I do think it has high priority for Q1.

[sarahmaddox]

Some of the work items required before we can generate a reference:

• Clean up the namespaces.
• Get the Pipelines SDK preinstalled in KubeFlow JupyterHub (team is working on this).
• Create a PyPI entry.
• Make sure the code comments are suitable for doc generation.
• Set up a doc generation process.

[sarahmaddox]

/area docs /area api /area pipelines /area 1.0.0 /priority p0

[sarahmaddox]

/cc @abhi-g /assign @qimingj

[sarahmaddox]

It'd be good if we can have this by the 0.4 release.

/area 0.4.0 /remove-area 1.0.0

[sarahmaddox]

Info on generating reference docs from Python doc strings, using Pydoc: https://docs.python.org/3.7/library/pydoc.html

We need to think about how we'll include the generated reference docs in our Kubeflow website. As an example, see Kubernetes:

• Has a “Reference” page within the website framework, listing the various reference docs: https://kubernetes.io/docs/reference/
• Then branches out to a separate look and feel for the generated reference: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.13/

[sarahmaddox]

There's a REST API that you can use to, for example, create a new pipeline. Here is the swagger definition for the API: https://github.com/kubeflow/pipelines/tree/master/backend/api/swagger

The Python SDK supports submitting pipeline runs programmatically: https://github.com/kubeflow/pipelines/blob/master/sdk/python/kfp/_client.py

[sarahmaddox]

From @hongye-sun in issue https://github.com/kubeflow/website/issues/518 (marked as duplicate of this issue).

We need to build or generate SDK documentation from current python code. The documentation should cover:

• The public surface of DSL SDK code
• The public surface of pipeline client SDK code
• The public surface of component sdk which user can use inside their container. code

The best thing we can do is to generate the python docs from docstring, but we need to figure out how to integrate with the kubeflow/website structure.

[sarahmaddox]

To integrate the generated reference docs with the Kubeflow website structure:

• Generate the doc files as Markdown pages. It's fine to have the content as HTML, but use the file extension .md mot .html.

• Use the following file path in the kubeflow/website repo: content/docs/references/API-NAME/VERSION/RESOURCE-NAME/...

• Add metadata to the top of each file, similar to the example below. This will make the website treat the file as a web page that needs the header and navigation chrome around it:

+++
title = "YOUR PAGE TITLE"
description = "Reference documentation for YOUR SDK NAME"
weight = 10
+++

The weight determines the page order within a given directory. A good numbering sequence is 10, 20, 30, and so on.

See the following PR as an example: https://github.com/kubeflow/website/pull/516

[sarahmaddox]

/assign