The initial plan was to enhance the unified test runner to support the new search index management commands. There ended up being complications integrating support for the new commands into the UTR (the largest issue was that the commands are asynchronous, and return from the server before the changes have finished). The unified tests were already written and expected errors when run against non-Atlas deployments, so they effectively serve as unit tests for drivers without writing another specialized test format. The new prose tests add end-to-end functionality for drivers' implementations.
Why setup/teardown a cluster for each test?
Drivers already spinup/teardown resources for existing tests (serverless, gcp/azure kms, FAAS testing), so additional CI time was not a concern.
We have tooling in drivers-evergreen-tools to setup and teardown Atlas clusters so the setup was easy to reuse.
Provisioning a cluster on-demand reduces the changes for collisions and the need to ensure driver use unique databases / collections / commit hashes to avoid CI runs interfering on the same cluster.
CSOT
These commands should work with CSOT as per the CSOT spec, but looking through the specs repo we don't seem to have tests for every driver helper that supports timeoutMS. Consequently, I chose not to add tests for these helpers with timeoutMS.
Please complete the following before merging:
[x] Update changelog.
[x] Make sure there are generated JSON files from the YAML test files.
[x] Test changes in at least one language driver.
[x] Test these changes against all server versions and topologies (including standalone, replica set, sharded clusters, and serverless).
This PR does three things:
The third bullet can be separated into a different PR if necessary.
Node POC: https://github.com/mongodb/node-mongodb-native/pull/3736
Notes
Why have unified tests and prose tests?
The initial plan was to enhance the unified test runner to support the new search index management commands. There ended up being complications integrating support for the new commands into the UTR (the largest issue was that the commands are asynchronous, and return from the server before the changes have finished). The unified tests were already written and expected errors when run against non-Atlas deployments, so they effectively serve as unit tests for drivers without writing another specialized test format. The new prose tests add end-to-end functionality for drivers' implementations.
Why setup/teardown a cluster for each test?
CSOT
These commands should work with CSOT as per the CSOT spec, but looking through the specs repo we don't seem to have tests for every driver helper that supports
timeoutMS. Consequently, I chose not to add tests for these helpers withtimeoutMS.Please complete the following before merging: