createIndex `background` option is obsolete for mongo 4.2+

[mattcollier]

See: https://docs.mongodb.com/manual/reference/method/db.collection.createIndex/#options-for-all-index-types

Server reports when it sees the option

2021-03-11T15:06:45.853+0000 I  INDEX    [conn1439] ignoring obselete { background: false } index build option because all indexes are built in the background with the hybrid method

We can start removing our usage of the backround option.

[dlongley]

We will need to be more careful now with this new index building behavior. Previously:

Foreground index builds were fast and produced more efficient index data structures, but required blocking all read-write access to the parent database of the collection being indexed for the duration of the build.

This is no longer an option. This means that the index constraints (such as uniqueness) will not be enforced until the index is finished building and, while it's being built, those constraints will not be enforced:

Similarly, an application can successfully write documents to the inventory collection with duplicate values of product_sku while the index build is in progress. If any violations still exist at the end of the build, the mongod terminates the build and throws an error.

https://docs.mongodb.com/manual/core/index-creation/#constraint-violations-during-index-build

I'm not sure what this means for our applications during startup, particularly those that rely on duplicate exceptions. We haven't seen any problems yet, but it sounds like that's only because of a combination of timing and the limited number of cases where we do that sort of thing.

[mattcollier]

I have just confirmed that the mongodb 3.5.x driver functions as expected. That is, that the following will throw an error if there are database entries to violate a new index. The MongoDB docs state that the server builds the index and THEN checks for key violations. This suggests to me that awaiting the createIndex operation as we do on bedrock startup is the correct approach. This of course says nothing about cases where there may be multiple processes accessing the same database as in a K8S deployment.

await database.createIndex({
  controller: 1, 'config.referenceId': 1
}, {
  partialFilterExpression: {
    'config.referenceId': {$exists: true}
  }, 
  unique: true, background: false
};

An error like this one is properly produced if there is a unique key violation:

MongoError: E11000 duplicate key error collection: bedrock_kms_http_test.kmsKeystore index: controller_1_config.referenceId_1 dup key: { controller: "+rCwVnrMecakDSIvDOvlYd1SRffwPB/VhBeHBznkpwA=", config.referenceId: "urn:uuid:7c8b5e04-bbeb-4267-bdf3-b8ea425f9e32" }
    at MessageStream.messageHandler (/home/matt/bedrock-dev/bedrock-index-test/node_modules/mongodb/lib/cmap/connection.js:268:20)
    at MessageStream.emit (events.js:315:20)
    at MessageStream.EventEmitter.emit (domain.js:467:12)
    at processIncomingData (/home/matt/bedrock-dev/bedrock-index-test/node_modules/mongodb/lib/cmap/message_stream.js:144:12)
    at MessageStream._write (/home/matt/bedrock-dev/bedrock-index-test/node_modules/mongodb/lib/cmap/message_stream.js:42:5)
    at writeOrBuffer (internal/streams/writable.js:358:12)
    at MessageStream.Writable.write (internal/streams/writable.js:303:10)
    at Socket.ondata (internal/streams/readable.js:719:22)
    at Socket.emit (events.js:315:20)
    at Socket.EventEmitter.emit (domain.js:467:12) {
  ok: 0,
  code: 11000,
  codeName: 'DuplicateKey',
  keyPattern: { controller: 1, 'config.referenceId': 1 },
  keyValue: {
    controller: '+rCwVnrMecakDSIvDOvlYd1SRffwPB/VhBeHBznkpwA=',
    'config.referenceId': 'urn:uuid:7c8b5e04-bbeb-4267-bdf3-b8ea425f9e32'
  }
}

As for errors that are thrown in Bedrock event handlers where our mongo indexing takes place, we have long standing issues there, see: https://github.com/digitalbazaar/bedrock/issues/78