Added support for automatic Unit Tests and Code Coverage Report generation.

[dustuu]

This pull request adds optional support for automatically running Unit Tests via GitHub Actions and generating Code Coverage Reports from those tests. Configuration steps for this feature are outlined here. If this feature is not configured by a user, it will simply be skipped over during workflow execution.

Two configurable badges have also been added to the README file; one badge which displays the package's current VPM version, and another which shows the results of the Code Coverage Report (if the new automatic unit tests feature is enabled). These badges are outlined here. When set up properly, they look like this:

• 
• 

Clicking on the VPM badge opens the repository's VPM Listing Page, while clicking on the Code Coverage badge opens the the generated Code Coverage Report.

:warning: Note: The Code Coverage badge currently does not render in my forked repository's README: This is because I have already changed it to point back to the source vrchat-community/template-package repository in preparation for being merged. If merged, this badge will render properly in the vrchat-community/template-package repository's README after the new Build Release workflow has been run at least one time with the Automatic Unit Tests feature enabled. Users who don't use the Automatic Unit Testing feature can simply remove this badge from their README files.

Additionally, this pull request removes the need for users to manually edit any of the workflow files. It does this by using repository variables to pass values into the workflows. Users can now create a repository variable named PACKAGE_NAME with the name of their package, such as com.vrchat.demo-template in this demo, instead of manually editing the workflow files. This step has been added to the README file here.

If automatic unit testing is enabled, repository secrets can also be used for Unity License configuration, which are outlined here. There are also several more non-secret repository variables to be entered at this step.

In this template repository, these are the current configurations of all of the repository variables used:	Variable Name	Value
PACKAGE_NAME	com.vrchat.demo-template
ASSEMBLY_NAME	VRChatPackageTemplate
REQUIRED_COVERAGE	100
UNITY_VERSION	2019.4.31f1

If the user requests for Code Coverage percentages to be enforced by setting the REQUIRED_COVERAGE variable to something greater than 0, the Build Release workflow will enforce their configured Code Coverage target. It does this by using my own custom fork of a GitHub Action that runs in a Docker Container: Kemocade.Unity.Coverage.Action. This Container is built during the start of the Build Release workflow's test job, and executed later in that same job's Enforce Code Coverage step.

The Build Release workflow previously only contained one job, build. It now contains three jobs: config, test, and build:

• config
  • Checks which repository variables and repository secrets have been configured.
• test
  • Executes the Game CI Test Runner GitHub Action and builds a Code Coverage report. If all required configuration was not found in the config job, this job will be skipped.
• build
  • Publishes a release of the package. This job only requires the PACKAGE_NAME variable to be configured, all other variables and secrets are optional. This job will run even if the test job is skipped.

From this template repository, I've created 8 example repositories that show how the updated workflows respond to various possible configurations of repository variables, asset changes, or both:

• template-package-unconfigured shows what happens when the workflows are run with no configuration.

  • Without a PACKAGE_NAME variable, the Build Release workflow will skip both the test and build jobs:
  • 
  • The Build Repo Listing workflow will be trigged to run as well, but it will fail as expected because there are no published releases.
  • Both badges will be broken because there are no releases:
  • 
  • 

• template-package-skip-tests shows what happens when only the PACKAGE_NAME variable is set, but none of the variables or secrets needed for running Unit Tests are. This is how workflows will run for users who don't opt-in to the new automatic Unit Tests feature, which is likely to be most users.

  • The Build Release workflow will skip the test job because it is missing the necessary configuration, but will run the build job and make a release:
  • 
  • The Build Repo Listing workflow will be trigged to run and will succeed.
  • The VPM Badge will work because we had a successful release, but the Code Coverage badge will be broken because we did not generate a Code Coverage Report:
  • 
  • 

• template-package-full shows what happens when all repository variables and secrets are configured properly, and Unit Tests are present. This version uses all available new features.

  • The Build Release workflow will pass on all three jobs:
  • 
  • The Build Repo Listing workflow will be trigged to run and will succeed, just as before. Additionally, this copies the previously created Code Coverage report onto the GitHub Pages website. This report can then be accessed via the Code Coverage badge in the README.
  • Both badges will be configured properly:
  • 
  • 

• template-package-template-package-fail-tests shows what happens when all repository variables and secrets are configured properly, but one or more Unit Tests are failing.

  • The Build Release workflow will fail because Unit Tests are present and configured to run by the user via repository variables, but failing:
  • 
  • The Build Repo Listing workflow will fail as expected because there are no published releases.
  • Both badges will be broken because there are no releases:
  • 
  • 

• template-package-no-tests shows what happens when all repository variables and secrets are configured properly, but the actual Unit Test files are removed from the repository.

  • The Build Release workflow will fail because the user has indicated that Unit Tests are present, but they are not:
  • 
  • The Build Repo Listing workflow will fail as expected because there are no published releases.
  • Both badges will be broken because there are no releases:
  • 
  • 

• template-package-runtime-tests-only-fail shows what happens when all repository variables and secrets are configured properly, but Unit Tests are only present for testing the package's Editor Scripts.

  • Because REQUIRED_COVERAGE is still set to 100, the Build Release workflow will fail due to the Unit Tests no longer covering the Editor Scripts:
  • 
  • The Build Repo Listing workflow will fail as expected because there are no published releases.
  • Both badges will be broken because there are no releases:
  • 
  • 

• template-package-runtime-tests-only shows the same scenario as template-package-runtime-tests-only-fail, but this time has REQUIRED_COVERAGE set to 0.

  • The Build Release workflow will pass on all three jobs. even without Editor Tests. This is because the new configuration no longer requires full code coverage:
  • 
  • The Build Repo Listing workflow will be trigged to run and will succeed.
  • Both badges will be configured properly:
  • 
  • 

• template-package-editor-tests-only shows the same scenario as template-package-runtime-tests-only, but with only Editor Tests instead of only Runtime Tests.

  • Just as before, the Build Release workflow will pass on all three jobs:
  • 
  • The Build Repo Listing workflow will be trigged to run and will succeed.
  • Both badges will be configured properly:
  • 
  • 

Beyond the main features of Unit Tests, Code Coverage, and Repository Variable Configuration, this pull request also includes a few small additional changes.

Previously, zip releases were made with a third-party GitHub Action that used relative paths to back outwards from the package itself:

- name: Create Zip
  uses: thedoctor0/zip-release@09336613be18a8208dfa66bd57efafd9e2685657
  with:
    type: "zip"
    directory: "Packages/${{env.packageName}}/"
    filename: "../../${{env.zipFile}}" # make the zip file two directories up, since we start two directories in above

I have changed this to use the native Bash zip command, as well as absolute paths built from ${{ github.workspace }}.

- name: Create Package Zip
  working-directory: "${{ env.packagePath }}"
  run: zip -r "${{ github.workspace }}/${{ env.zipFile }}" .

This new approach is also used to build the Code Coverage Report zip, if Automatic Unit Testing is enabled.

Finally, names and comments have been added to all workflow steps which did not previously have them. All workflow steps which use other actions via the uses keyword have also been set to use their source's commit ID instead of tags, keeping in line with how this was done in most of the original workflows.

This pull request has been built on top of the existing work done in the approved but still-pending PR #11. Therefore, this pull request can be merged in place of or on top of that one with no merge conflicts.

[momo-the-monster]

First - thank you very much @dustuu for submitting this helpful and very well documented PR! I'm happy with the changes to use Repository Variables instead of requiring code changes. I'm a little worried that the automation and information about the test suite will be an overload for most of our package creators, who are not likely to create and run tests like this for their packages. I don't want anyone scared away because of how complicated it may look. Any thoughts on that, @Faxmashine?

[dustuu]

@momo-the-monster No worries! I thought that might be the case.

I can open a new PR with a subset of the changes from this one.

Would you like me to include only the Repository Variable changes, or are you also interested in including the workflow cleanup changes such as the new zip step or the updated comments & action versions?

[momo-the-monster]

@dustuu - I'm interested in all the workflow cleanup changes too, thanks!

[dustuu]

@momo-the-monster I've opened a new PR #13 which has the reduced scope you're looking for. I'll go ahead and close this PR now. 🍍👍