CSM BDS: Change allocation index mapping of text field to date type

[thanh-lam]

Describe the bug Current cast-allocation index contains two string fields that are used for time range query:

data.begin_time
data.history.end_time

With elasticsearch 7.5.1+, time range queries on text fields do not work. These fields need to be mapped to date type.

It's possible that the index cast-allocation didn't have a customized mapping of data fields initially. So, filebeat reads the allocation records in csm_transaction log and puts them into elasticsearch indexes as text or string type.

This defect documents the procedure of loading a customized index mapping template for cast-allocation using the filebeat and curl commands.

To Reproduce Steps to reproduce the behavior:

1. Log in to a BDS node to access elasticsearch service (local or remote).
2. Run curl to display cast-allocation index mapping:

   curl http://XX.X.X.XX:9200/cast-allocation/_mapping?pretty

3. In the output, look for begin_time and end_time
4. Note that they are defined as text, for example:

           "begin_time" : {
             "type" : "text",
             "fields" : {
               "keyword" : {
                 "type" : "keyword",
                 "ignore_above" : 256
               }
             }
           },

Expected behavior Those date fields should be defined as date, for example:

            "begin_time" : {
              "type" : "date"
            },

Screenshots If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• Machine [CSM BDS, IST BDS clusters]
• Version [CSM 1.8.2]

Additional context With these fields changed to date type, the date format to be written into them needs to have correct format as well. See #992.

Issue Source: Beside addressing the index mapping issue, the procedure of how to use the filebeat command to load customized mapping template should be in CSM BDS readthedoc for when there's need in the future.

[thanh-lam]

Following are the steps to load a customized index mapping for cast-allocation using the filebeat command.

1. /etc/filebeat/filebeat.ym contains following at the end (no changes). Note: The section under "Elasticsearch template setting" could be added automatically by one of the commands.

   
   name: "master"
   setup.kibana:
   host: "X.XXX.XX.XX:5601"

output.logstash: hosts: [ "cXXXfXn0X:10523" ]

logging.level: info logging.to_files: true logging.files: path: /var/log/filebeat name: filebeat keepfiles: 7 permissions: 0644

==================== Elasticsearch template setting ==========================

setup.template.name: "filebeat-*" setup.template.fields: "filebeat_fields.yml" setup.template.overwrite: true

2. Obtain current `cast-allocation` index mapping from `elasticsearch` and save the output to a file (`cast-allocation-map.json`:

curl http://XX.X.X.XX:9200/cast-allocation/_mapping?pretty > cast-allocation-map.json

3. Edit `cast-allocation-map.json` and change the two fields: `data.begin_time` and `data.history.end_time` to type `date` as shown in defect description.
4. Copy `cast-allocation-map.json` to `cast-allocation-template.json`. Edit the new file and make sure the "index_patterns" attribute is assigned the value "cast-allocation" before the "mappings" section.

{ "index_patterns": [ "cast-allocation" ], "mappings": {

5. Run `filebeat` to disable `logstash` output and enable `elasticsearch` output:

filebeat setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["XX.X.X.XX:9200"]'

6. Run `curl` to load the customized template:

curl -XPUT -H 'Content-Type: application/json' http://XX.X.X.XX:9200/_template/cast-allocation -d@cast-allocation-template.json

7. Verify that the template was applied:

curl http://XX.X.X.XX:9200/_template/cast-allocation?pretty

8. Login to `kibana` GUI to delete existing `cast-allocation` index in `elasticsearch` and `kibana`. Then, create some new csm allocations to see new `cast-allocation` index in `Elasticsearch`. Now, create the index pattern in `kibana` to access new data.

[thanh-lam]

Edited the steps to add the last step because it's necessary to delete the old cast-allocation index in elasticsearch and kibana. Then, the new data will be mapped to the new data type.