Add end-to-end tests

[vdye]

Closes #23

Summary

This pull request creates a Cucumber/TypeScript end-to-end testing framework for the Git Bundle Server (see #23 for background on the framework & reasoning for why TypeScript was selected). Two classes of tests are created to start with: "basic" tests (demonstrating expected typical workflows) and "performance" tests (comparing performance with & without use of the bundle server on large repositories).

The commits are broken down as follows:

• Commit 1 sets up a TypeScript Cucumber project with no tests
• Commits 2-4 create the basic scenario tests, including the corresponding test step definitions & helper classes.
• Commit 5 adds the performance tests, as well as the filters to prevent running those tests by default
• Commit 6 updates all of the workflows to use the 'stable' version of Go (in preparation for a new workflow that does the same)
• Commit 7 adds an e2e-test target to the Makefile
• Commit 8 adds a new test.yml workflow

Performance results

The performance tests are quite slow, so they aren't run by default in the Actions workflow. For reference, some example results on my machine were:

Clone execution time for https://github.com/git/git.git: 20.70720636224747s (bundle URI) vs. 27.499098003387452s (no bundle URI)
Clone execution time for https://github.com/torvalds/linux.git: 474.9542240447998s (bundle URI) vs. 623.6930130381584s (no bundle URI)

Note that this only compares the clone time; it also took torvalds/linux ~10 minutes to clone & generate its 4.1Gb(!) base bundle. Since that scale is more reflective of the monorepos this bundle server would be used with than the small repos used in the "basic" tests, it's probably worth doing some more detailed profiling on the bundle server. Hopefully, we'd find some places where we can improve performance.

[derrickstolee]

I played around with extending your tests, so I could get a feel of how it works. The result is de0924472e261b8967e8c35763b7c8cc0d7d1e7d, which let me repeat the performance tests, but used a second machine with existing bundles instead of creating new ones from scratch. (Perhaps these should be the same, with the environment variable switching modes at runtime instead of failing without it.)

[vdye]

I ended up making more substantial changes than I initially intended. 😅 Beyond the suggestions in @derrickstolee's review:

• changed setup-go to use version in go.mod rather than the stable.
• split the performance test into @slow and not-@slow, adding the non-@slow tests to the default end-to-end execution.
• rather than create a new npm script per profile, pass the profiles as command line args in run-e2e-tests.sh.
• related to ^, change TEST_E2E_PERF to the more generic E2E_FLAGS, with documentation for the valid options.

If anyone's interested, here's the range-diff!

Range diff

```diff 1: a817a95 ! 1: e8a38b0 e2e: add initial bundle server clone test @@ Commit message repository initialized in a bundle server can be cloned with '--bundle-uri' to download bundles. + Helped-by: Derrick Stolee Signed-off-by: Victoria Dye ## .gitignore ## @@ test/e2e/features/step_definitions/repository.ts (new) + const actualURI = result.stdout.toString().trim() + assert.strictEqual(actualURI, this.bundleServer.bundleUri()) + -+ result = clonedRepo.runGit("for-each-ref", "--format=%(refname)") ++ result = clonedRepo.runGit("for-each-ref", "--format=%(refname)", "refs/bundles/*") + utils.assertStatus(0, result, "git for-each-ref failed") + -+ const bundleRefRegex = new RegExp('^refs/bundles/.*$') -+ const bundleRefs = result.stdout.toString().split("\n").filter(function (line) { -+ return bundleRefRegex.test(line) ++ const bundleRefs = result.stdout.toString().split("\n").filter(function(line) { ++ return line.trim() != "" + }) + assert.strict(bundleRefs.length > 0, "No bundle refs found in the repo") +}) 2: 99d8db1 ! 2: 368fa56 e2e: add tests updating the remote & bundle server @@ Commit message bundle server. As part of this update, add a custom Cucumber expression for matching "are/are not" to a boolean. + Helped-by: Derrick Stolee Signed-off-by: Victoria Dye ## test/e2e/cucumber.js ## @@ test/e2e/features/support/world.ts: export class BundleServerWorld extends World + + const clonedRepo = this.getRepo(user) + -+ // TODO: figure out a better way to check whether the repo is empty -+ if (clonedRepo.runGit("show").status != 0) { ++ const result = clonedRepo.runGit("rev-list", "--all", "-n", "1") ++ if (result.stdout.toString().trim() == "") { + // Repo is empty, so make sure we're on the right branch + utils.assertStatus(0, clonedRepo.runGit("branch", "-m", branch)) + } else { 3: 40dd79c ! 3: c9ee08f e2e: add performance test @@ Commit message e2e: add performance test Add a test that captures timing measurements (in 'console.log()' printouts) - for cloning various large repositories. Because these tests can be - exceptionally slow, tag them with '@slow' and make sure they are not run in - the 'default' profile. Add an 'all' profile that runs the performance tests - alongside the "regular" tests. + for cloning various large repositories. The tests are divided into two + example sets: one for tests that run on modern hardware with a good internet + connection for a *total* of less than 15 minutes (currently take ~7 + minutes), and the other for tests that take longer than that. The latter set + is tagged with '@slow' and the 'default' profile is updated to skip them. + Add an 'all' profile that runs these slow performance tests alongside the + "regular" tests. + Helped-by: Derrick Stolee Signed-off-by: Victoria Dye ## test/e2e/cucumber.js ## @@ test/e2e/features/performance.feature (new) + Background: The bundle web server is running + Given the bundle web server was started at port 8080 + -+ @slow + Scenario Outline: Comparing clone performance + Given a remote repository '' + Given the bundle server has been initialized with the remote repo @@ test/e2e/features/performance.feature (new) + Then I compare the clone execution times + + Examples: -+ | repo | -+ | https://github.com/git/git.git | # takes ~2 minutes -+ | https://github.com/torvalds/linux.git | # takes ~30(!) minutes ++ | repo | ++ | https://github.com/git/git.git | ++ | https://github.com/kubernetes/kubernetes.git | ++ ++ @slow ++ Examples: ++ | repo | ++ | https://github.com/homebrew/homebrew-core | ++ | https://github.com/torvalds/linux.git | ## test/e2e/features/step_definitions/repository.ts ## @@ test/e2e/features/step_definitions/repository.ts: When('I clone from the remote repo with a bundle URI', async function (this: Bun @@ test/e2e/features/step_definitions/repository.ts: Then('my repo\'s bundles {bool + utils.assertStatus(0, myClone.cloneResult) + utils.assertStatus(0, otherClone.cloneResult) + -+ console.log(`\nClone execution time for ${this.remote!.remoteUri}: ${myClone.cloneTimeMs / 1000}s (bundle URI) vs. ${otherClone.cloneTimeMs / 1000}s (no bundle URI)`) ++ console.log(`\nClone execution time for ${this.remote!.remoteUri}: ${(myClone.cloneTimeMs / 1000).toFixed(2)}s (bundle URI) vs. ${(otherClone.cloneTimeMs / 1000).toFixed(2)}s (no bundle URI)`) +}) - - ## test/e2e/package.json ## -@@ - "description": "", - "main": "index.js", - "scripts": { -- "test": "cucumber-js" -+ "test": "cucumber-js", -+ "test-all": "cucumber-js -p all" - }, - "license": "MIT", - "devDependencies": { 4: 3d477c9 < -: ------- workflows: update 'go-version' to 'stable' -: ------- > 4: 9c7860f e2e: add profile for running offline tests -: ------- > 5: 704f46f workflows: use go version specified in 'go.mod' 5: cfcf0b0 ! 6: a6ff664 Makefile: add 'e2e-test' target @@ Commit message Add a target to the Makefile that installs the end-to-end test NPM dependencies then runs the tests. By default, this will only run tests matching the "default" profile of the tests (excluding performance tests). - The 'TEST_E2E_PERF' flag, if it has a value, will instead run the tests with - the 'all' profile (including the performance tests). + The 'E2E_FLAGS' variable can be used to specify the profile(s) used by the + tests (e.g., '--all' to include the tests excluded by default). + + Add documentation on how to run the end-to-end tests to 'README.md'. Signed-off-by: Victoria Dye @@ Makefile: PACKAGE_ARCH := $(GOARCH) APPLE_APP_IDENTITY = APPLE_INST_IDENTITY = APPLE_KEYCHAIN_PROFILE = -+TEST_E2E_PERF= ++E2E_FLAGS= # Build targets .PHONY: build @@ Makefile: doc: + @echo + @echo "======== Running end-to-end tests ========" + $(RM) -r $(TESTDIR) -+ @if [ -n "$(TEST_E2E_PERF)" ]; then \ -+ ./scripts/run-e2e-tests.sh --all; \ -+ else \ -+ ./scripts/run-e2e-tests.sh; \ -+ fi ++ @scripts/run-e2e-tests.sh $(E2E_FLAGS) + # Installation targets .PHONY: install @@ Makefile: clean: $(RM) -r $(DOCDIR) + $(RM) -r $(TESTDIR) + ## README.md ## +@@ README.md: $ go build -o bin/ ./... + + ### Testing and Linting + +-To run the project's unit tests, navigate to the repository root directory and +-run `go test -v ./...`. ++Unless otherwise specified, run commands from the repository root. + +-To run the project's linter, navigate to the repository root directory and run +-`go vet ./...`. ++#### Unit tests ++ ++``` ++go test -v ./... ++``` ++ ++#### Linter ++ ++``` ++go vet ./... ++``` ++ ++#### End-to-end tests ++ ++In order to run these tests, you need to have a recent version of ++[Node.js](https://nodejs.org) (current LTS version is a pretty safe bet) and NPM ++installed. ++ ++For the standard set of tests (i.e., excluding exceptionally slow tests), run: ++ ++``` ++make e2e-test ++``` ++ ++To configure the test execution and filtering, set the `E2E_FLAGS` build ++variable. The available options are: ++ ++* `--offline`: run all tests except those that require internet access. ++* `--all`: run all tests, including slow performance tests. ++ ++The above modes are mutually exclusive; if multiple are specified, only the last ++will be used. For example, `E2E_FLAGS="--offline --all"` is equivalent to ++`E2E_FLAGS="--all"`. ++ ++:warning: The performance tests that are excluded by default clone very large ++repos from the internet and can take anywhere from ~30 minutes to multiple hours ++to run, depending on internet connectivity and other system resources. + + ## License + + ## scripts/run-e2e-tests.sh (new) ## @@ +#!/bin/bash @@ scripts/run-e2e-tests.sh (new) +TESTDIR="$THISDIR/../test/e2e" + +# Defaults -+NPM_TARGET="test" ++ARGS=() + +# Parse script arguments +for i in "$@" +do +case "$i" in ++ --offline) ++ ARGS+=("-p" "offline") ++ shift # past argument ++ ;; + --all) -+ NPM_TARGET="test-all" ++ ARGS+=("-p" "all") + shift # past argument + ;; + *) @@ scripts/run-e2e-tests.sh (new) +cd "$TESTDIR" + +npm install -+npm run $NPM_TARGET ++npm run test -- ${ARGS:+${ARGS[*]}} 6: 617da16 ! 7: f0b17de workflows: add a workflow to run end-to-end tests @@ Commit message Add a dedicated 'test.yml' workflow that executes the end-to-end tests. Unlike CI, these tests are only run automatically for pull requests (although they can be manually triggered via workflow dispatch). By default, - the tests will run with the "default" (no perf test) profile; this can be - changed via an input variable in the workflow dispatch. + the tests will run with the "default" profile; this can be changed via an + input variable in the workflow dispatch. Because the end-to-end tests deal with external dependencies (namely, Git), add an 'install-dependencies.sh' script to install and configure the @@ .github/workflows/test.yml (new) + pull_request: + workflow_dispatch: + inputs: -+ test-perf: ++ run-all: + type: boolean -+ description: 'Whether to run the (slow) end-to-end perf tests' ++ description: 'Include tests that are excluded by default due to slowness' + default: false + +jobs: @@ .github/workflows/test.yml (new) + runs-on: ubuntu-latest + + steps: ++ - name: Clone repository ++ uses: actions/checkout@v3 ++ + - name: Setup Go + uses: actions/setup-go@v3 + with: -+ go-version: 'stable' ++ go-version-file: 'go.mod' + + - name: Setup Node.js + uses: actions/setup-node@v3 + with: + node-version: 18 + -+ - name: Clone repository -+ uses: actions/checkout@v3 -+ + - name: Install system dependencies for end-to-end tests + run: build/ci/install-dependencies.sh + shell: bash + + - name: Enable perf tests -+ if: ${{ github.event.inputs.test-perf }} -+ run: echo "TEST_E2E_PERF=1" >> $GITHUB_ENV ++ if: ${{ github.event.inputs.run-all }} ++ run: echo "E2E_FLAGS='--all'" >> $GITHUB_ENV + + - name: Run end-to-end tests + env: -+ TEST_E2E_PERF: ${{matrix.jobs.goarch}} -+ run: make e2e-test TEST_E2E_PERF="$TEST_E2E_PERF" ++ E2E_FLAGS: ${{env.E2E_FLAGS}} ++ run: make e2e-test E2E_FLAGS="$E2E_FLAGS" ## build/ci/install-dependencies.sh (new) ## @@ ```