DRIVERS-2533: add RunCursorCommand API

[nbbeeken]

Summary

Adds a new runCursorCommand database API that can handle running a generic command that returns a cursor from the server. MongoDB cursors need the client to track some pieces of metadata that are generally internal to a driver implementation (session.lsid, cursor.id, server/connection, namespace) so this API is aware that the command being run needs to maintain these components for subsequent iteration of the cursor, via getMore commands.

Changes

• Update the find_getmore_killcursors spec
  • Describe the shape of a getMore command
  • Add more description to what a tailable / tailable_await cursor is
  • Add a requirement to update id and ns upon every getMore response
• Update the run-command spec
  • Add note about gossiping $clusterTime - an additional property the driver MUST attach to the command sent to the server on behalf of the user
  • Add runCursorCommand specification
  • The options interface
  • Handling of: Sessions, Server Selection, Load Balancers, Iteration, Tail-ability, Resource Clean up, and CSOT
• Add yaml tests for the various options supported by the API
  • successfully executes checkMetadataConsistency cursor creating command
  • errors if the command response is not a cursor
  • creates an implicit session that is reused across getMores
  • accepts an explicit session that is reused across getMores
  • returns pinned connections to the pool when the cursor is exhausted
  • returns pinned connections to the pool when the cursor is closed
  • supports configuring getMore batchSize
  • supports configuring getMore maxTimeMS
  • supports configuring getMore comment
  • does not close the cursor when receiving an empty batch
• Update the Unified Test Format spec
  • New runCursorCommand operation that iterates a cursor to exhaustion in one operation
  • New createCommandCursor operation that saves an entity to be iterated later
  • close and iterateUntilDocumentOrError are expected to support the new entity
• Add valid-pass tests
  • runCursorCommand creates and exhausts cursor by running getMores
  • createCommandCursor creates a cursor and stores it as an entity that can be iterated one document at a time
  • createCommandCursor cursor can be closed and will perform a killCursors operation
• Add CSOT tests
  • errors if timeoutMode is set without timeoutMS
  • error if timeoutMode is cursorLifetime and cursorType is tailableAwait
  • error if getMore maxTimeMS setting is greater than timeoutMS
  • error if getMore maxTimeMS setting is equal to timeoutMS
  • Non-tailable cursor lifetime remaining timeoutMS applied to getMore if timeoutMode is unset
  • Non=tailable cursor iteration timeoutMS is refreshed for getMore if timeoutMode is iteration - failure
  • Tailable cursor iteration timeoutMS is refreshed for getMore - failure
  • Tailable cursor awaitData iteration timeoutMS is refreshed for getMore - failure

Tasks

• [x] Update changelog.
• [x] Make sure there are generated JSON files from the YAML test files.
• [x] Test changes in Node.js: https://github.com/mongodb/node-mongodb-native/pull/3655
• [x] Test change in Go:
• [x] Test these changes against all server versions and topologies.

[nbbeeken]

@alcaeus

allow customization of batchSize, comment, and maxTimeMS for getMore command invocations: does this apply to all cursors or only those created through runCursorCommand?

It MAY only apply to this API, I don't see a reason other than expanding the scope of existing APIs why driver's should or should not allow getMore fields to be configurable for all cursors. It's likely that there is no need in other cursors intended to provide a high level API where getMores are a hidden implementation detail. For this API, the user is seeking the ability to control command construction, so the motivation is to try and provide the knobs that also impact getMore command construction.

Some more discussion here: https://github.com/mongodb/specifications/pull/1412#discussion_r1190202154

Use case examples here: https://mongodb.slack.com/archives/C7WJZNUTA/p1675879216249919 https://jira.mongodb.org/browse/GODRIVER-1605

Given that context do you think there is something to add? I wouldn't recommend it for other cursor implementations but I don't see anything wrong with it either.

[alcaeus]

@nbbeeken thanks for the explanation. I don't think we need to add it to the rest of the cursor-generating API at this time. We may want to consider that in the future, at which point we should also revisit how these options are set: whether it's in the original call that produces the cursor or on the resulting cursor itself.