search

awslabs / multi-model-server

Multi Model Server is a tool for serving neural net models for inference
Apache License 2.0
998 stars 230 forks source link

Performance regression suite Cut 2 #926

Closed maheshambule closed 4 years ago

maheshambule commented 4 years ago

Before or while filing an issue please feel free to join our slack channel to get in touch with development team, ask questions, find out what's cooking and more!

Issue #, if available:

Description of changes:

This PR removes the duplicate code in the test case yamls and modularizes test cases. This will help in maintaining test cases and adding new test cases.

A Test Case consists of the test.yaml, test.jmx, environment/*.yaml files and a global_config.yaml.

  1. The global_config.yaml will now act as a master template and will be shared across test cases..
  2. The global_config contains all common yaml sections and boilerplate.
  3. The global_config contains common criteria and monitoring metrics which well get applied to all the test cases.
  4. The global_config contains variables for metrics and test specific attributes.
  5. The test case environment yaml files contain values for the variables mentioned in the global_config and test.yaml.
  6. The test case test.yaml inherits the global_config yaml. And it usually contains the scenario, specific preprocessing commands (if any), and special criteria (if any) applicable for that test case only.
  7. If you want a behavior other than defined in the master template, It is possible to override sections of global_config.yaml in the individual test case. The global_config.yaml's top-level sections can be overridden, merged, or appended based on below rules:
    • By default the dictionaries get merged.
    • If the dictionary key is prepended with '~' it will get overridden.
    • The list gets appended.

Testing done:

To run CI tests on your changes refer README.md

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.