search

kubeflow / testing

Test infrastructure and tooling for Kubeflow.
Apache License 2.0
63 stars 89 forks source link

readme

Table of Contents generated with DocToc

Test Infrastructure

There are two test infrastructures exist in the Kubeflow community:

If you are interested in oss-test-infra, please find useful resources here.
If you are interested in optional-test-infra, please find useful resources here

We use Prow, K8s' continuous integration tool.

We use Prow to run:

Here's high-level idea about how it works

Quick Links

Anatomy of our Tests

Writing An Argo Workflow For An E2E Test

This section provides guidelines for writing Argo workflows to use as E2E tests

This guide is complementary to the E2E testing guide for TFJob operator which describes how to author tests to performed as individual steps in the workflow.

Some examples to look at

Adding an E2E test to a repository

Follow these steps to add a new test to a repository.

Python function

  1. Create a Python function in that repository and return an Argo workflow if one doesn't already exist

    • We use Python functions defined in each repository to define the Argo workflows corresponding to E2E tests

    • You can look at prow_config.yaml (see below) to see which Python functions are already defined in a repository.

  2. Modify the prow_config.yaml at the root of the repo to trigger your new test.

    • If prow_config.yaml doesn't exist (e.g. the repository is new) copy one from an existing repository (example).

    • prow_config.yaml contains an array of workflows where each workflow defines an E2E test to run; example

      workflows:
- name: workflow-test
  py_func: my_test_package.my_test_module.my_test_workflow
  kwargs:
      arg1: argument
      • py_func: Is the Python method to create a python object representing the Argo workflow resource
      • kwargs: This is an array of arguments passed to the Python method
      • name: This is the base name to use for the submitted Argo workflow.

  3. You can use the e2e_tool.py to print out the Argo workflow and potentially submit it

  4. Examples

ksonnet

Using ksonnet is deprecated. New pipelines should use python.

  1. Create a ksonnet App in that repository and define an Argo workflow if one doesn't already exist

    • We use ksonnet apps defined in each repository to define the Argo workflows corresponding to E2E tests

    • If a ksonnet app already exists you can just define a new component in that app

      1. Create a .jsonnet file (e.g by copying an existing .jsonnet file)
      1. Update the params.libsonnet to add a stanza to define params for the new component

    • You can look at prow_config.yaml (see below) to see which ksonnet apps are already defined in a repository.

  2. Modify the prow_config.yaml at the root of the repo to trigger your new test.

    • If prow_config.yaml doesn't exist (e.g. the repository is new) copy one from an existing repository (example).

    • prow_config.yaml contains an array of workflows where each workflow defines an E2E test to run; example

      workflows:
- app_dir: kubeflow/testing/workflows
  component: workflows
  name: unittests
  job_types:
    - presubmit
  include_dirs:
    - foo/*
    - bar/*
      params:
  params:
    platform: gke
    gkeApiVersion: v1beta1

      • app_dir: Is the path to the ksonnet directory within the repository. This should be of the form ${GITHUB_ORG}/${GITHUB_REPO_NAME}/${PATH_WITHIN_REPO_TO_KS_APP}

      • component: This is the name of the ksonnet component to use for the Argo workflow

      • name: This is the base name to use for the submitted Argo workflow.

        • The test infrastructure appends a suffix of 22 characters (see here)

        • The result is passed to your ksonnet component via the name parameter

        • Your ksonnet component should truncate the name if necessary to satisfy K8s naming constraints.

        • e.g. Argo workflow names should be less than 63 characters because they are used as pod labels

      • job_types: This is an array specifying for which types of prow jobs this workflow should be triggered on.

        • Currently allowed values are presubmit, postsubmit, and periodic.

      • include_dirs: If specified, the pre and postsubmit jobs will only trigger this test if the PR changed at least one file matching at least one of the listed directories.

        • Python's fnmatch function is used to compare the listed patterns against the full path of modified files (see here)

        • This functionality should be used to ensure that expensive tests are only run when test impacting changes are made; particularly if its an expensive or flaky presubmit

        • periodic runs ignore include_dirs; a periodic run will trigger all workflows that include job_type periodic

      • A given ksonnet component can have multiple workflow entries to allow different triggering conditions on pre/postsubmit

      • For example, on presubmit we might run a test on a single platform (GKE) but on postsubmit that same test might run on GKE and minikube

      • this can be accomplished with different entries pointing at the same ksonnet component but with different job_types and params.

      • params: A dictionary of parameters to set on the ksonnet component e.g. by running ks param set ${COMPONENT} ${PARAM_NAME} ${PARAM_VALUE}

Using pytest to write tests

Prow Variables

Argo Spec

Creating K8s resources in tests.

Tests often need a K8s/Kubeflow deployment on which to create resources and run various tests.

Depending on the change being tested

To connect to the cluster:

NFS Directory

An NFS volume is used to create a shared filesystem between steps in the workflow.

Step Image

Checking out code

Building Docker Images

There are lots of different ways to build Docker images (e.g. GCB, Docker in Docker). Current recommendation is