MongoDB service startup initialization

[philipmjohnson]

Please do work for this task in a branch called issue-235.

(Based upon comments below, I've edited this issue description and title.)

Upon MongoDB service startup via Docker, it would be useful to perform several system startup tasks in an initialization script/file.

To reflect the first-class status of the MongoDB service in OPQ Cloud, please remove the util/mongo directory and create a top-level opq/mongodb directory in which to put these scripts and files. The following should be accomplished by the initialization scripts/files:

• [ ] Ensure that all required collections are defined (if they are not already defined).
• [ ] Attach mongodb schemas to these collections to ensure that they conform to our data model documentation. If not already defined.
• [ ] Ensure that appropriate indexes are created for all collections (if not already defined).
• [ ] Ensure that appropriate TTL for each type of entity (measures, events, incidents, phenomena) are defined.
• [ ] Ensure that this script is invoked each time the MongoDB service starts up.
• [ ] Provide a link in the data model documentation to the scripts to make it easy to ensure that the documentation and the schema definitions stay in sync.

[philipmjohnson]

Looks like we already have a "database bootstrap script" which ensures that indexes are created. This might be the place to add collection definitions along with their associated schemas?

[anthonyjchriste]

It's just a JS file that creates the indexes: https://github.com/openpowerquality/opq/blob/master/util/mongod/ensure_indexes.js

I'm not sure if it's a great place to store the schemas or not.

[anthonyjchriste]

There is an associated script as well: https://github.com/openpowerquality/opq/blob/master/util/mongod/ensure_indexes.sh

It's currently not integrated with the mongo docker. Should probably somehow be integrated with the mongo docker to be called every time the mongo docker image starts.

[philipmjohnson]

Interesting. A couple of thoughts:

• Perhaps we need a top-level directory called "mongodb" in which stuff related to the :mango: service is stored. Since Mongo is a first class OPQ Cloud service, it seems like it deserves its own top-level directory in opq.

• The ensure-indexes.js file seems to assume that the collections have already been defined? Perhaps we need to rename this file to "mongodb-init.js", and have it ensure that the collections have been defined, plus ensure that the schemas are attached, plus ensure that indexes are defined?

[philipmjohnson]

Agree that this "Mongo Initialization Script" needs to be integrated into Docker.

[philipmjohnson]

Looks like the measurements collection needs an index on its timestamp_ms field.