Trying to create an elasticsearch index I get this error: "must specify non-negative number of shards for index [my_search]"

[AlexisWilke]

I turned on support for ElasticSearch.

    const options = {
            // See: https://docs.datastax.com/en/developer/nodejs-driver/3.3/api/type.ClientOptions/
            clientOptions: {
                contactPoints: [
                    '127.0.0.1'
                ],
                protocolOptions: {
                    port: 9042
                },
                keyspace: 'my_search',
                queryOptions: {
                    consistency: models.consistencies.one
                },
                elasticsearch: {
                    host: 'http://127.0.0.1:9200',
                    apiVersion: '5.5',
                    sniffOnStart: true
                }
            },

            // See: https://express-cassandra.readthedocs.io/en/stable/usage/#explanations-for-the-options-used-to-initialize
            ormOptions: {
                defaultReplicationStrategy: {
                    class: 'NetworkTopologyStrategy'
                },
                migration: 'alter',
                manageESIndex: true
            }
        }

But when I try to run for the first time (I just started with a brand new database), I get the following error about shards:

{ Error: [illegal_argument_exception] must specify non-negative number of shards for index [my_search]
    at respond (/my_search/node_modules/elasticsearch/src/lib/transport.js:308:15)
    at checkRespForFailure (/my_search/node_modules/elasticsearch/src/lib/transport.js:267:7)
    at HttpConnector.<anonymous> (/my_search/node_modules/elasticsearch/src/lib/connectors/http.js:165:7)
    at IncomingMessage.wrapper (/my_search/node_modules/lodash/lodash.js:4935:19)
    at IncomingMessage.emit (events.js:187:15)
    at endReadableNT (_stream_readable.js:1094:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  status: 400,
  displayName: 'BadRequest',
  message:
   '[illegal_argument_exception] must specify non-negative number of shards for index [my_search]',
  path: '/my_search',
  query: {},
  body:
   '{"settings":{"keyspace":"my_search"},"index":{"number_of_shards":1}}',
  statusCode: 400,
  response:
   '{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"must specify non-negative number of shards for index [my_search]"}],"type":"illegal_argument_exception","reason":"must specify non-negative number of shards for index [my_search]"},"status":400}',
  toString: [Function],
  toJSON: [Function] }

Looking for a solution, I saw a few things such as this command line:

curl -XPUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -d '{
  "index.number_of_shards" : "1"
}'

But that fails too:

{"error":
  {"root_cause":
    [
      {
        "type":"index_not_found_exception",
        "reason":"no such index",
        "resource.type":"index_or_alias",
        "resource.id":"_all",
        "index_uuid":"_na_",
        "index":"_all"
      }
    ],
    "type":"index_not_found_exception",
    "reason":"no such index",
    "resource.type":"index_or_alias",
    "resource.id":"_all",
    "index_uuid":"_na_",
    "index":"_all"
  },
  "status":404
}

I also tried with "my_search" as the index name (instead of "_all"). Same results.

Looking at the Elassandra code, I see that this is what breaks. The create_index() call.

ElassandraBuilder.prototype = {
  create_index(keyspaceName, indexName, callback) {
    debug('creating elassandra index: %s', indexName);
    this._client.indices.create({
      index: indexName,
      body: {
        settings: {
          keyspace: keyspaceName,

          // various attempts to get that number_of_shards to work...
          index: { number_of_shards: 1 }
          //number_of_shards: 1
          //"index.number_of_shards": 1
        },
      }
    }, function (err) {
      if (err) {
        callback(err);
        return;
      }

      callback();
    });
  },
[...snip...]

[AlexisWilke]

I found the location of the error in the elasticsearch code and reported it here:

https://github.com/elastic/elasticsearch/issues/36234

It's actually the number of Replicas that is the cause of the error and NOT the number of Shards. So the message is confusing at the moment. The bug report is asking the authors of elasticsearch to fix that glitch.

[AlexisWilke]

Okay, I found my problem. I had to use the data center information in the ORM options as follow:

ormOptions: {
  defaultReplicationStrategy: {
    class: 'NetworkTopologyStrategy',
    DC1: 1
  }
}

When you switch from the default to using elasticsearch, you need to switch to NetworkTopologyStrategy and to specify the replication you need to do it here with the name of the data center and the replication factor as shown above.

You can't use the replication_factor along NetworkTopologyStrategy so that's why you have to name the data center. I though that would come directly from the setup files (namely cassandra-topology.properties and cassandra-rackdc.properties) but that's not the case. You have to be explicit in the ORM options as well.

When you do not include that replication factor information, as in:

ormOptions: {
  defaultReplicationStrategy: {
    class: 'NetworkTopologyStrategy'
  }
}

you get a replication factor of zero (0) and then get the error I mentioned above.

(See also #167)