search

canonical / seldon-core-operator

Seldon Core Operator
Apache License 2.0
5 stars 10 forks source link

Integration tests for Tensorflow Seldon servers #151

Closed i-chvets closed 1 year ago

i-chvets commented 1 year ago

Description

Integration tests for Tensorflow Seldon severs are required to validate that Seldon Deployment will be successful. Integration tests for these servers should be a part of Seldon integration tests suite and run on every PR. These tests will be reused in ROCKs testing.

The following Tesnorflow prepackaged Seldon servers should be covered in test:

        "TENSORFLOW_SERVER": {
          "protocols" : {
            "tensorflow": {
              "image": "tensorflow/serving",
              "defaultImageVersion": "2.1.0"
              },
            "seldon": {
              "image": "seldonio/tfserving-proxy",
              "defaultImageVersion": "1.15.0"
              }
            }
        },

Design

Use pamatetirized tests to execute same test for different Seldon servers. To test use specified test deployments from:

For seldon protocol, server container image in test response data is updated based on deployed seldon-config ConfigMap. This ensures that test response data contains properm image and version.

For one of the Tensorflow tests request and response data are relatively large. That data is stored in files and common paramterized test is modified to be able to read data from file (JSON) or use it as is (JSON object). Eg.:

    # prepare request data:
    # - if it is string, load it from file specified in by that string
    # - otherwise use it as JSON object
    if isinstance(request_data, str):
        # response test data contains file with JSON data
        with open(f"tests/data/{request_data}") as f:
            request_data = json.load(f)

For mlflowserver test ephemeral-sorage=2G requirement/limit needs to be added to ensure it runs in GH runner.

Testing

Integration testing should pass when implemented.

i-chvets commented 1 year ago

Merged. Closing.