Massive test suite overhaul [Phase 1]

[toptobes]

Major changes to the astra-db-ts test suite

This is the first stage (of two) of major test overhauls to the astra-db-ts test suite. The second will come as tests for Data API Tables are added.

Rationale

Beyond testing itself, astra-db-ts, along with the other Data API clients, serve as the last line of defense when it comes to testing the Data API—thus, it's important to have a comprehensive test suite that covers as much surface area as it can.

However, in our particular case, its's difficult to do so in an efficient, clean, and scalable manner with the default tooling Mocha provides out of the box. What this update does is write a light wrapper around the basic Mocha API to provide some baked-in functionality focused around the following things:

• Parallelism
• Eliminating Data-API-testing-related boilerplate
• More parallelism
• More intuitive test filtering
• And some more parallelism

While admittedly adding in some getting-started-complexity, by virtue of having to learn a somewhat-new testing API, these changes have eliminated hundreds of lines of boilerplate throughout the actual test code, and, most importantly, has made the full astra-db-ts test ~500% faster, in a manner that is very generic and easily appliable to any and all future tests.

(The few fewer tests are because some duplicate/extraneous tests were removed)

Changes to the test suite itself

Preface

Before getting into it, here are some "common terms" that may be used throughout the rest of my rambling in this section

• "common fixtures" - These are the { client, db, collection, dbAdmin } objects which are most commonly used in tests
• "tag filtering" - This checks for "tags" in the test names, such as (LONG) createCollection tests, or (ADMIN) AstraAdmin tests, and checks if these tags are enabled in the current test suite run—skips the test/suite if not

Mocha-API-inspired functions

The most prominent changes are the introduction of 5 new Mocha-API-esque functions (two of which are overhauls)

• describe - An overhaul to the existing dynamic block
  • Provides fresh instances of the "common fixtures" in its callback
  • Performs "tag filtering" on the suite names
  • Some suite options to reduce boilerplate
    • truncateColls: 'default' - Does deleteMany({}) on the default collection in the default namespace after each test case
    • truncateColls: 'both' - Does deleteMany({}) on the default collection in both test namespaces after each test case
    • dropEphemeral: 'after' - Drops all non-default collections in both test namespaces after all of the test cases in the suite
    • dropEphemeral: 'afterEach' - Drops all non-default collections in both test namespaces each test case
• it - An overhaul to the existing it block
  • Performs "tag filtering" on the test names
  • Provides unique string keys for every test case
• parallel - A wrapper around describe which runs all of its test cases in parallel
  • Only allows it, before, after, and a single layer of describe functions
  • Will run all tests simultaneously in a before hook, capture any exceptions, and rethrow them in reconstructed it/describe blocks for the most native-like behavior
  • Performs tag and test filtering as normal
  • Nearly all integration tests have been made parallel
• background - A version of describe which runs in the background while all of the other test cases run
  • Only allows it blocks
  • Will run the test at the very start of the test script, capture any exceptions, and rethrow them in reconstructed it/describe blocks for the most native-like behavior at the end of the test script
  • Performs tag and test filtering as normal
  • Meant for independent tests that take a very long time to execute (such as the integration.devops.db-admin lifecycle test)

These are not globals like Mocha's—rather, they are imported, like so:

import { background, describe, it, parallel } from '@/tests/testlib';

Examples

You can find examples of usages of each in most, if not all, test files, such as:

• /tests/integration/miscs/timeouts.test.ts (describe, parallel, it)
• /tests/integration/devops/lifecycle.test.ts (background)

Custom test filtering*

Before even implementing any of this, Mocha's native test filtering was always iffy with dynamically generated test cases (i.e. for vectorize), and with the addition of custom test description blocks, native test filtering would've been even more broken.

Instead, astra-db-ts now has a Mocha-inspired, yet enhanced, test-filtering implementation.

Usage is like so:

scripts/test.sh [~f/F <match_filter>] [~g/G <regex_filter] [-fand | -for] # (-fand is default)

You can use multiple filters, with -fand (the default) requiring them all to be true for a test to run, or -for requiring just one. You can also negate a filter (requiring it to NOT match) using ~f instead of -f. ~ takes precedence over - if filters overlap.

Examples:

scripts/test.sh -f VECTORIZE -F openai # Run only vectorize tests, except for openai
scripts/test.sh -f createCollection -f dropCollection -for # Run all tests for either createCollection or dropCollection

*Test filtering refers to filtering tests by their name—tag filtering is an astra-db-ts-specific concept that refers specifically to filtering by tags in the name, such as (LONG) createCollection tests

Custom error reporter

A custom error reporter, which extends the default spec error reporter, was also added, which logs the complete errors thrown in each test run to the ./etc/test-reports directory in md format. This uses util.inspect on the error to print the full error object for the most possible debuggability.

Changes to the test script

Preface

There have been so many changes to the test script, there's no point trying to list them out. So I'll just give a quick overview of the new API.

One important thing though, is the test script is no longer an alternative to using the Mocha command directly—you need to use the test script, as it performs some operations Mocha simply can't/won't.

Script commands

There are two ways to use the script:

# Running the test cases
scripts/test.sh [-all | -light | -coverage] [-fand | -for] [-f/F <filter>]+ [-g/G <regex>]+ [-w/W <vectorize_whitelist>] [-b | -bail] [-R | -no-report] [-c <http_client>] [-e <environment>]

or

# Linting + typechecking
scripts/test.sh [-lint] [-tc]

You can check out the DEVGUIDE for a full explanation of the above.

Changes to the env vars

The env vars names' have all been changed, the most prominent change of which is them all being prefixed with CLIENT_ now. Check the .env.example file for the new names.

That being said, beyond the CLIENT_APPLICATION_URI and CLIENT_APPLICATION_TOKEN, the .env.example file shouldn't be used much except for setting default values for certain env vars—using flags on the test script is more recommended.

Other changes

There are likely quite a few smaller changes I've missed or not covered here, but in general, reading the DEVGUIDE should tell you everything you need to know to run and write your own astra-db-ts tests.