Feature request: Specify ES settings and mappings

[jmrichardson]

I have 4 requirements to configure specific aspects of ES:

1. Set field limit to 2000
2. Create custom analyzers and tokenizers
3. Create nested fields
4. Set specific field properties (document.reference to indexed with offsets and custom analyzer)

I currently do all of the above by sending the following settings/mappings to ES prior to starting Norconex jobs:

PUT wmsearch
{
  "settings": {
    "index.mapping.total_fields.limit": 2000,
    "analysis": {
      "analyzer": {
        "custom": {
          "type": "custom",
          "tokenizer": "custom_token",
          "filter": [
            "lowercase"
          ]
        },
        "custom2": {
          "type": "custom",
          "tokenizer": "keyword",
          "filter": [
            "lowercase"
          ]
        }
      },
      "tokenizer": {
        "custom_token": {
          "type": "ngram",
          "min_gram": 1,
          "max_gram": 30
        }
      }
    }
  },
  "mappings": {
    "doc": {
      "properties": {
        "document": {
          "properties": {
            "reference": {
              "type": "text",
              "index_options": "offsets",
              "analyzer": "custom"
            }
          }
        },
        "scope" : {
          "type" : "nested",
          "properties" : {
            "level" : { 
              "type" : "integer"
            },
            "ancestors" : { 
              "type" : "keyword",
              "index" : "true"
            },
            "value" : { 
              "type" : "keyword",
              "index" : "true"
            },
            "order" : {
              "type" : "integer"
            }    
          }
        }
      }
    }
  }
}

This does the job. However, it would be nice to specify these settings in the xml file. Perhaps a committer tag with the above pasted?

Thanks for your consideration.

[essiembre]

It is not the goal of a Committer to "configure" its target, but that could be handy nonetheless, as long as it does not create new problems.

What would happen if we submit the mappings again every time the collector runs? Or if you make changes to the mappings between two runs? Or worse, what if someone changed the schema manually on ES and this new option overwrite such changes?

If none of the above is a concern, we can consider adding this option.

[jmrichardson]

What would happen if we submit the mappings again every time the collector runs?

If the index already exists, ES will complain. I think the collector should ignore and log this specific issue and continue. At least, this is what I have seen other tools do.

Or if you make changes to the mappings between two runs?

Since the index won't be changed because it already exists, collector will continue to run as normal again ignoring/logging the ES complaint. It would up to the user to delete the index to allow the committer apply the new configuration.

Or worse, what if someone changed the schema manually on ES and this new option overwrite such changes?

Once the mapping is indexed, it cannot be changed unless the index is deleted. You can however add new mappings (as it hasn't been indexed) but this request is to just apply the initial mapping before it is indexed.

IMHO, none of the issues are of concern. However, the current method of applying an initial index mapping manually is working fine. It's just an extra step that could be added as a feature if you think it is worth it. Either way, I am very impressed with this product and really appreciate your support. I will continue to let you know if I run into any other issues. Thanks again.

[essiembre]

Many thanks for your great feedback. I am sold. :-)

[jmrichardson]

I think there is a better way of doing this... I have been using templates to specify my mappings which provides the functionality described above without having to send any initial mappings. I am not sure if you have begun work on this but may want to reconsider. Thanks

[essiembre]

Thanks for the update. If there is a better way then we'll forget this. Can you elaborate on your template solution?

[jmrichardson]

I think it is safe to forget about this request because in ES you can specify your mappings and settings in a template. I am able to accomplish 1-4 above with the following template:

PUT _template/wmsearch
{
  "template": "wmsearch",
  "settings": {
    "index.mapping.total_fields.limit": 1000,
    "analysis": {
      "analyzer": {
        "custom": {
          "type": "custom",
          "tokenizer": "custom_token",
          "filter": [
            "lowercase"
          ]
        },
        "custom2": {
          "type": "custom",
          "tokenizer": "keyword",
          "filter": [
            "lowercase"
          ]
        }
      },
      "tokenizer": {
        "custom_token": {
          "type": "ngram",
          "min_gram": 1,
          "max_gram": 30
        }
      }
    }
  },
  "mappings": {
    "Documents": {
      "properties": {
        "content": {
          "type": "text",
          "term_vector" : "with_positions_offsets",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "title": {
          "type": "text",
          "term_vector" : "with_positions_offsets",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "document": {
          "properties": {
            "reference": {
              "type": "text",
              "index_options": "offsets",
              "analyzer": "custom",
              "term_vector" : "with_positions_offsets",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "filename": {
              "type": "text",
              "index_options": "offsets",
              "analyzer": "custom",
              "term_vector" : "with_positions_offsets",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "generatedTitle": {
              "type": "text",
              "index_options": "offsets",
              "analyzer": "custom",
              "term_vector" : "with_positions_offsets",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "collector": {
          "properties": {
            "filesize": {
              "type": "long",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "lastmodified": {
              "type": "long",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "scope" : {
          "type" : "nested",
          "properties" : {
            "level" : { 
              "type" : "integer"
            },
            "ancestors" : { 
              "type" : "keyword",
              "index" : "true"
            },
            "value" : { 
              "type" : "keyword",
              "index" : "true"
            },
            "order" : {
              "type" : "integer"
            }    
          }
        }
      }
    }
  }
}

With the above template saved, whenever I delete and re-index the above settings and mappings are automatically applied for "wmsearch". Therefore, I don't need to send an initial call to ES to set the above when re-indexing (deleting and creating).

I am not sure if anyone else has a use case for sending an initial settings/mappings or other call to ES prior to indexing, but mine are handled with the template feature of ES.

Let me know if you need more detail. Thanks.