[RFC] Amplify Video Test Suite

[smp]

As we look at the next stage of maturity for Amplify Video, we have some housekeeping we'd like to address in order to...

• Increase confidence in project stability and usability for production environments (not just prototyping)
• Confidence Amplify Projects with Video resources maintain stability over the course of their life
• Prevent regressions from being introduced
• Ease of testing for new features
• Identify issues prior to releasing to npm/users
• Provide recommended version of upstream Amplify-CLI version compatibility with A/V

This ticket is a call to help us scope the test suite sprint for A/V. Currently we are thinking about 3 key test categories - integration tests, system tests, and usability tests.

Integration Testing

Integration tests should validate that the plugin produces cloud resources that successfully deploy and operate within the context of an Amplify project. Examples of integration tests

• Does Amplify-CLI recognize an Installation of Amplify Video?
• Do all versions of the Video resource managed by A/V deploy successfully upon running 'amplify push'
• Does a change introduce new service or increase Resource security surface?
• Does the change use new interfaces exposed by Amplify-CLI?
• Have Amplify-CLI functions used by Amplify Video been modified?
• Do Amplify-CLI commands work as expected with Amplify Video managed Resources?
• Do user inputs successfully store and push with the project?

System Testing

System tests should validate that the functionality implemented by the entire Amplify Framework work as intended to the end viewer or application consumer (React Component, A/V SDK, etc).

• Does video play?
• Does the Video Management System (VMS) Schema still compile?
• Does the VMS respond to queries?
• How do Resource Categories created by Amplify Video, which Video Resources depend on, still exist?

Usability Testing

As a stretch goal, the test suite should deploy and maintain an active website using AWS Amplify and Amplify Video so that maintainers can see the published results of the build process.

[nathanagez]

Hi @smp , I was reading the contribution guidelines and noticed that actually on a pull request, contributors must have to manually run ESLint before submitting their PR (https://github.com/awslabs/amplify-video/blob/master/CONTRIBUTING.md#pull-requests).

It is human to forgot sometimes, adding a tool doing it automatically (using git hooks) could be really useful and time saving. Husky is a great open source library. We can use it to:

• Validate commit messages (if we have later a specific convention)
• Lint code on a pre-commit stage
• Run tests

Just before submitting the pull request. If it make sens, I can open a PR and set it up.