Queryable Encryption is also referred to as QE or FLE2.
The versions of the QE protocols are noted as QEv1 and QEv2.
QEv1 was introduced in 6.0.0. QEv2 is introduced in 7.0.0.
Removal of eccCollection
This is requested by DRIVERS-2524.
In QEv1, each QE collection required three additional metadata collections: eccCollection, ecocCollection, and escCollection.
In QEv2, the eccCollection is no longer required.
Removal of documenting escCollection and ecocCollection
This is requested by DRIVERS-2586.
Addition of Wire Version check
QEv2 is a backwards breaking change to QEv1. Drivers supporting QEv2 will not support QEv1. Drivers supporting QEv1 will not support QEv2. QEv2 is introduced in server 7.0.0. I expect QEv2 will not be backported.
Sending an QEv1 payload to mongod7.0.0-alpha-1139-gc9e3390 results in a server error. Example:
(Location7292602) Encountered a Queryable Encryption find payload type that is no longer supported: 5
Sending a QEv2 payload to mongod6.1.0 results in a server error. Example:
Enumeration value '11' for field 'subtype' is not a valid value.
Creating a QE collection using encryptedFields without including eccCollection does not result in a server error from 6.1.0. Instead, the eccCollection is implicitly created by the server. The wire version check is intended to give users a helpful error if attempting to use a QEv2 driver on a QEv1 server. Without the wire version check, the user may not discover the incompatibility until a QEv2 payload is sent. This may result in the QE collections being created but being unusable.
Creating a collection with encryptedFields with eccCollection does not result in a server error from 7.0.0-alpha-1139-gc9e3390. SERVER-75683 requests returning a helpful error if attempting to create a QEv1 collection on a QEv2 server.
Interaction with createEncryptedCollection
The wire version check is proposed in createCollection. createEncryptedCollection calls createCollection after creating the data keys. If the wire version check fails, this may result in data keys being created and returned. This behavior is expected to be improved by the proposal in DRIVERS-2540 to create data keys and collections in a transaction.
Please complete the following before merging:
[x] Update changelog.
[x] Make sure there are generated JSON files from the YAML test files.
[x] Test changes in at least one language driver. Tested in C driver here- [] Test these changes against all server versions and topologies (including standalone, replica set, sharded clusters, and serverless).QE is skipped on serverless pending DRIVERS-2589. C driver does not currently test QE on sharded
Summary
eccCollection
.escCollection
andecocCollection
not be documented.eccCollection
is not created.Additional Improvements
encryptedFields
.encryptedFieldsMap with cyclic entries does not loop
encryptedFields are consulted for metadata collection names
Changes have been tested in the C driver here: https://github.com/mongodb/mongo-c-driver/pull/1232
Tests require libmongocrypt 1.8.0-alpha0 or newer. Binaries for 1.8.0-alpha0 are available here: https://spruce.mongodb.com/task/libmongocrypt_publish_upload_all_042603b1d72f49d9034e7059b82a65e843c7e38a_23_03_29_15_26_54/logs?execution=0
Background & Motivation
Terminology
Queryable Encryption is also referred to as QE or FLE2. The versions of the QE protocols are noted as QEv1 and QEv2. QEv1 was introduced in 6.0.0. QEv2 is introduced in 7.0.0.
Removal of eccCollection
This is requested by DRIVERS-2524. In QEv1, each QE collection required three additional metadata collections:
eccCollection
,ecocCollection
, andescCollection
. In QEv2, theeccCollection
is no longer required.Removal of documenting
escCollection
andecocCollection
This is requested by DRIVERS-2586.
Addition of Wire Version check
QEv2 is a backwards breaking change to QEv1. Drivers supporting QEv2 will not support QEv1. Drivers supporting QEv1 will not support QEv2. QEv2 is introduced in server
7.0.0
. I expect QEv2 will not be backported.Sending an QEv1 payload to
mongod
7.0.0-alpha-1139-gc9e3390
results in a server error. Example:(Location7292602) Encountered a Queryable Encryption find payload type that is no longer supported: 5
Sending a QEv2 payload to
mongod
6.1.0
results in a server error. Example:Enumeration value '11' for field 'subtype' is not a valid value.
Creating a QE collection using
encryptedFields
without includingeccCollection
does not result in a server error from6.1.0
. Instead, theeccCollection
is implicitly created by the server. The wire version check is intended to give users a helpful error if attempting to use a QEv2 driver on a QEv1 server. Without the wire version check, the user may not discover the incompatibility until a QEv2 payload is sent. This may result in the QE collections being created but being unusable.Creating a collection with
encryptedFields
witheccCollection
does not result in a server error from7.0.0-alpha-1139-gc9e3390
. SERVER-75683 requests returning a helpful error if attempting to create a QEv1 collection on a QEv2 server.Interaction with
createEncryptedCollection
The wire version check is proposed in
createCollection
.createEncryptedCollection
callscreateCollection
after creating the data keys. If the wire version check fails, this may result in data keys being created and returned. This behavior is expected to be improved by the proposal in DRIVERS-2540 to create data keys and collections in a transaction.Please complete the following before merging:
- [] Test these changes against all server versions and topologies (including standalone, replica set, sharded clusters, and serverless).QE is skipped on serverless pending DRIVERS-2589. C driver does not currently test QE on sharded