loopbackio / loopback-connector-elastic-search

Strongloop Loopback connector for Elasticsearch
MIT License
79 stars 55 forks source link

Multi Index usage #52

Closed pulkitsinghal closed 8 years ago

pulkitsinghal commented 8 years ago

The goal is to implement and document how to use this connector with more than one index.

pulkitsinghal commented 8 years ago

It seems to me that after raising Issue #20, the solution provided for it in PR #23 by @fabien ... already had laid the groundwork for configuring and supporting each model with its own index. For example this became possible as a result of the work done at that time:

      User = db.define('User', {
        ...
      }, {
        elasticsearch: {
          index: red
        }
      });

      Customer = db.define('Customer', {
        ...
      }, {
        elasticsearch: {
          index: green
        }
      });

BUT there were several commits after that which may have undone some of the work!

For example this commit by @mrfelton makes changes under the assumption that there is only one index ... with that assumption:

pulkitsinghal commented 8 years ago

So far reviewing the code and experimentation has shown me that calling setupIndices instead of setupIndex in ESConnector.prototype.connect has no merit because the _models array (modelClass is looked up here by addDefaults) is empty so essentially the same thing happens, no matter which method we call.

pulkitsinghal commented 8 years ago

Maybe we can move the setup for index and mappings to run when Connector.prototype.define is called? I searched github with keywords: connector call define user:strongloop extension:js and found several clues.

  1. while mysql takes care of everything upfront
  2. instead clodant seems to have a more dynamic approach ... which is the sort of dynamic hook I'd been looking for ... to make sure its been done before.
pulkitsinghal commented 8 years ago

So I think overriding define would be a bad idea because the framework that calls will not wait for any of the async operations like es-index-creation or es-mappings-creation to finish.

ESConnector.prototype.define = function(modelDefinition) {
  var self = this;

  // call parent method
  ESConnector.super_.prototype.define.apply(self, arguments);

  // add additional logic
  // TODO: add indexes
  // TODO: add mappings
  // BUT above-TODO logic would be async whereas this code block will return right away!!!
  // which adds a certain level of unpredictability if we accidently end up working on the model
  // before indexes are created or mappings are done
};
pulkitsinghal commented 8 years ago

To summarize scenario-A:

  1. default index is still required and can be configured in datasource..json
  2. we no longer attempt to create ANY indices or mappings when the connector is initialized
  3. we only create an index and mapping when a modelDefinition is first provided to our connector - this has a huge CON that there is no way to tell the framework to wait for the async methods!
  4. automigrate continues to leverage Connector._models to take care of everything in one fell swoop.
pulkitsinghal commented 8 years ago

I've decided to go with scenario-B:

  1. any model specific indices or mappings should be setup when the connector is initialized
  2. mappigns array in datasource.<env>.json may contain the index and type properties, which will be used to setup that model's index and mappings during connector initialization
pulkitsinghal commented 8 years ago

released in v1.1.0 on sept 15 2016