search

awslabs / multi-model-server

Multi Model Server is a tool for serving neural net models for inference
Apache License 2.0
998 stars 230 forks source link

CircleCI + Performance Regression #925

Open prashantsail opened 4 years ago

prashantsail commented 4 years ago

Description of changes:

Primary goal of this PR is -

  1. Integration of performance regression suite with CircleCI
    • Add new performance regression job in circleci config
    • Add corresponding shell script for job execution
    • Added new _ci_linuxmedium environment for comparing against thresholds specific to CircleCI env.
      • Renamed xlarge environment to _macosxlarge (more specific)

Along with the primary goal, we have identified and fixed following -

  1. Movement of API tests(newman) suite from test/ to tests/api/
  2. CircleCI's API tests(newman) job now continues with execution of all 3 collections(management, inference, https), even if any one of them fails.
  3. CircleCI's shell scripts tweaked to exit with error when any of the important commands fail.
  4. CircleCI Jobs now copy the entire MMS's logs directory along with console logs (instead of only copying mms console logs)
  5. Removed - _/performanceregression. The directory was part of POC and is no longer required.

All Jobs are working and are successful after these changes - Build #443

Screenshot 2020-07-09 at 15 05 33

Performance regression Job Performance regression logs and results

Note:
The commit count is high because the previous PRs of CircleCI and Performance regression have been squashed & merged.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.