search

awslabs / multi-model-server

Multi Model Server is a tool for serving neural net models for inference
Apache License 2.0
995 stars 231 forks source link

performance regression suite #919

Closed maheshambule closed 4 years ago

maheshambule commented 4 years ago

Contributors: @dhruvaray @prashantsail, @maheshambule

Issue #, if available:

Description of changes:

It adds a performance monitoring test suite. This suite enables to quickly add test cases and specify pass fail criteria for different metrics. It creates a standard XML and HTML report with JUnit standard specifications. Refer README for more details.

Components added:

TODO-

  1. Add more metrics including GPU metrics
  2. Auto Threshold calculation, per environment threshold
  3. Environment profiles
  4. Add more tests
  5. Enable test case variables such as host, port to be specified from suite runner script

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

dhanainme commented 4 years ago

Can you please add the CircleCI logs / screenshots if possible.

maheshambule commented 4 years ago

@dhanainme, There is a separate PR for Circle CI integration. However, we had tested Circle CI and performance suite integration.

Below are the links of Circle CI job:

https://app.circleci.com/pipelines/github/prashantsail/multi-model-server/224/workflows/b4180b58-d141-4bef-9776-114763384e4c/jobs/563/steps

HTML Report: https://563-265769776-gh.circle-artifacts.com/0/tmp/mms-performance-regression/junit.html

Circle CI Job: Circle CI Job

HTML report: html_report

dhanainme commented 4 years ago

Follow up on in-person feedback from the call :

Documentation 

* Add details on Motivation & Scope of the tests in README.md
* Cookbook for running test  - Explain config / result / metrics / HTML report for a Single test

Simple Code Refactors

* Verbose explanation of comparison test result - Enabling easy understanding / ability to tune thresholds 
* Docs for all python classes   
    - Document & split classes to separate files  - specifically - Monitoring Agent / Comparison / Suite
    - Interaction with Taurus APIs  / JMX
    - Refactor / Break down ~150 line compare_artifacts python method

Eliminate Redundancy

* Eliminate redundancy across tests - consider generating the taurus / jmeter configs on the fly
dhanainme commented 4 years ago

This looks good to me. I ll have this merged tomorrow.