Re-add Index Mappings

[mymasse]

Biggest impact of not having this is the dynamic_template for String type.

Fixes cloudfoundry-community/logsearch-for-cloudfoundry#345

[TheMoves]

What OPs files are you testing with? For some reason i'm getting this error:

++ curl -s -H 'Content-Type: application/json' -w '%{http_code}' -o /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt -X PUT --data-binary @/var/vcap/jobs/elasticsearch_config/index-templates/index-mappings.json http://INTERNALIP:9200/_template/index-mappings
+ HTTP_CODE=400
+ grep --quiet errors:true /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
+ '[' 400 -le 300 ']'
+ cat /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters:  [doc : {dynamic_templates=[{string_fields={mapping={index=true, omit_norms=true, type=keyword}, match_mapping_type=string, match=*}}], _all={enabled=false}, properties={geoip={dynamic=true, type=object, properties={location={type=geo_point}}}, @raw={norms={enabled=false}, index=true, type=text}, @version={index=true, type=keyword}}}]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters:  [doc : {dynamic_templates=[{string_fields={mapping={index=true, omit_norms=true, type=keyword}, match_mapping_type=string, match=*}}], _all={enabled=false}, properties={geoip={dynamic=true, type=object, properties={location={type=geo_point}}}, @raw={norms={enabled=false}, index=true, type=text}, @version={index=true, type=keyword}}}]","caused_by":{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters:  [doc : {dynamic_templates=[{string_fields={mapping={index=true, omit_norms=true, type=keyword}, match_mapping_type=string, match=*}}], _all={enabled=false}, properties={geoip={dynamic=true, type=object, properties={location={type=geo_point}}}, @raw={norms={enabled=false}, index=true, type=text}, @version={index=true, type=keyword}}}]"}},"status":400}+ exit 400

piped the above file to jq, cat /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt | jq ..

{
  "error": {
    "root_cause": [
      {
        "type": "mapper_parsing_exception",
        "reason": "Root mapping definition has unsupported parameters:  [doc : {dynamic_templates=[{string_fields={mapping={index=true, omit_norms=true, type=keyword}, match_mapping_type=string, match=*}}], _all={enabled=false}, properties={geoip={dynamic=true, type=object, properties={location={type=geo_point}}}, @raw={norms={enabled=false}, index=true, type=text}, @version={index=true, type=keyword}}}]"
      }
    ],
    "type": "mapper_parsing_exception",
    "reason": "Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters:  [doc : {dynamic_templates=[{string_fields={mapping={index=true, omit_norms=true, type=keyword}, match_mapping_type=string, match=*}}], _all={enabled=false}, properties={geoip={dynamic=true, type=object, properties={location={type=geo_point}}}, @raw={norms={enabled=false}, index=true, type=text}, @version={index=true, type=keyword}}}]",
    "caused_by": {
      "type": "mapper_parsing_exception",
      "reason": "Root mapping definition has unsupported parameters:  [doc : {dynamic_templates=[{string_fields={mapping={index=true, omit_norms=true, type=keyword}, match_mapping_type=string, match=*}}], _all={enabled=false}, properties={geoip={dynamic=true, type=object, properties={location={type=geo_point}}}, @raw={norms={enabled=false}, index=true, type=text}, @version={index=true, type=keyword}}}]"
    }
  },
  "status": 400
}

I pulled down the latest of this and then cherry-picked your commit/PR to test. What you are doing makes sense, but I don't have enough experience with ELK to troubleshoot or make sense of the error

[mymasse]

All I did was put back what was there in the previous release. I tested part of it by adding the index mappings directly in Kibana. But these files used to work in ELK 6.8

[mymasse]

I'm away until Thursday I'll try to debug it then.

[TheMoves]

I believe the template needs to look like this...i've tested it my lab and it works. Kinda annoying that KIBANA freaks out spewing errors until it starts to index data. Please note I have almost 0 experience with ELK and 0 experience outside of the below at working with indexes/mappings/etc... Just spend 3 hours this morning reading the ELK doc for 7.6 and reading blog posts...

The stuff I removed, per my understanding, is deprecated in 7.6.

My understanding is the

{
  "template": "<%= p('elasticsearch_config.index_prefix') %>*",
  "order": 102,
  "settings": {
    "index": {
      "query": {
        "default_field": "@raw"
      }
    }
  },
  "mappings": {
    "dynamic_templates": [
      {
        "string_fields": {
          "match": "*",
          "match_mapping_type": "string",
          "mapping": {
            "type": "keyword",
            "index": true
          }
        }
      }
    ],
    "properties": {
      "@version": {
        "type": "keyword",
        "index": true
      },
      "@raw": {
        "type": "text",
        "index": true
      },
      "geoip": {
        "type": "object",
        "dynamic": true,
        "properties": {
          "location": {
            "type": "geo_point"
          }
        }
      }
    }
  }
}

[TheMoves]

Also need to update the logsearch-for-cloudfoundry templates for ELK 7.6 i think

[mymasse]

I made the fixes to the mappings, kept the norms fields but adapted to the correct syntax. Also made a fix in the logsearch-for-cloudfoundry mappings. Can somebody give it a test I'm not setup at all to test this (currently at home for the next few days)

[axelaris]

LGTM

[axelaris]

Thank you @mymasse !

[akhettal]

I'm trying to upgrade logsearch from v210.5.0 to v210.5.2 and using elasticsearch_config.templates with those custom template logservice template

Here is my elasticsearch_config maintenace job manifest :

properties:
      elasticsearch_config:
        index_prefix: logs-
        templates:
        - shards-and-replicas: /var/vcap/jobs/elasticsearch_config/index-templates/shards-and-replicas.json
        - index-settings: /var/vcap/jobs/elasticsearch_config/index-templates/index-settings.json
        - index-mappings: /var/vcap/jobs/elasticsearch_config/index-templates/index-mappings.json
        - index-mappings-logservice: /var/vcap/jobs/logservice-logsearch-filters/index-mappings.json
        - index-mappings-app-logservice: /var/vcap/jobs/logservice-logsearch-filters/index-mappings-app.json
        - index-mappings-platform-logservice: /var/vcap/jobs/logservice-logsearch-filters/index-mappings-platform.json

The upgrade seems no te be working and the bosh logs are :

Error: 'maintenance/768ccaf8-0f81-466e-9164-bc486f9e64c3 (0)' is not running after update. Review logs for failed jobs: elasticsearch_config
Task 524125 | 12:32:39 | Error: 'maintenance/768ccaf8-0f81-466e-9164-bc486f9e64c3 (0)' is not running after update. Review logs for failed jobs: elasticsearch_config

The errors jobs logs are :

+ URL=http://172.17.8.37:9200/_template/shards-and-replicas
+ CONTENT_PATH=/var/vcap/jobs/elasticsearch_config/index-templates/shards-and-replicas.json
++ curl -s -H 'Content-Type: application/json' -w '%{http_code}' -o /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt -X PUT --data-binary @/var/vcap/jobs/elasticsearch_config/index-templates/shards-and-replicas.json http://172.17.8.37:9200/_template/shards-and-replicas
+ HTTP_CODE=200
+ grep --quiet errors:true /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
+ '[' 200 -le 300 ']'
+ cat /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
+ URL=http://172.17.8.37:9200/_template/index-settings
+ CONTENT_PATH=/var/vcap/jobs/elasticsearch_config/index-templates/index-settings.json
++ curl -s -H 'Content-Type: application/json' -w '%{http_code}' -o /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt -X PUT --data-binary @/var/vcap/jobs/elasticsearch_config/index-templates/index-settings.json http://172.17.8.37:9200/_template/index-settings
+ HTTP_CODE=200
+ grep --quiet errors:true /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
+ '[' 200 -le 300 ']'
+ cat /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
+ URL=http://172.17.8.37:9200/_template/index-mappings
+ CONTENT_PATH=/var/vcap/jobs/elasticsearch_config/index-templates/index-mappings.json
++ curl -s -H 'Content-Type: application/json' -w '%{http_code}' -o /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt -X PUT --data-binary @/var/vcap/jobs/elasticsearch_config/index-templates/index-mappings.json http://172.17.8.37:9200/_template/index-mappings
+ HTTP_CODE=200
+ grep --quiet errors:true /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
+ '[' 200 -le 300 ']'
+ cat /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
+ URL=http://172.17.8.37:9200/_template/index-mappings-logservice
+ CONTENT_PATH=/var/vcap/jobs/logservice-logsearch-filters/index-mappings.json
++ curl -s -H 'Content-Type: application/json' -w '%{http_code}' -o /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt -X PUT --data-binary @/var/vcap/jobs/logservice-logsearch-filters/index-mappings.json http://172.17.8.37:9200/_template/index-mappings-logservice
+ HTTP_CODE=400
+ grep --quiet errors:true /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
+ '[' 400 -le 300 ']'
+ cat /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
tail: elasticsearch_config.stderr.log: file truncated
{"error":{"root_cause":[{"type":"parse_exception","reason":"request body is required"}],"type":"parse_exception","reason":"request body is required"},"status":400}+ exit 400

It's not taking the custom templates.

Note that when i'm exuciting the script manually it seems to be good :

/var/vcap/jobs/elasticsearch_config/bin
+ export DATA_DIR=/var/vcap/data/elasticsearch_config
+ DATA_DIR=/var/vcap/data/elasticsearch_config
+ URL=http://172.17.8.37:9200
++ curl -s http://172.17.8.37:9200/_cat/templates
++ awk '{print $1}'
+ for template in '$(curl -s ${URL}/_cat/templates | awk '\''{print $1}'\'')'
+ grep -qw security-index-template /dev/fd/63
++ echo 'shards-and-replicas index-settings index-mappings index-mappings-logservice index-mappings-app-logservice index-mappings-platform-logservice'
+ curl -XDELETE http://172.17.8.37:9200/_template/security-index-template
{"acknowledged":true}+ for template in '$(curl -s ${URL}/_cat/templates | awk '\''{print $1}'\'')'
+ grep -qw .ml-config /dev/fd/63
++ echo 'shards-and-replicas index-settings index-mappings index-mappings-logservice index-mappings-app-logservice index-mappings-platform-logservice'
+ curl -XDELETE http://172.17.8.37:9200/_template/.ml-config
{"acknowledged":true}+ for template in '$(curl -s ${URL}/_cat/templates | awk '\''{print $1}'\'')'
+ grep -qw index-mappings-logservice /dev/fd/63
++ echo 'shards-and-replicas index-settings index-mappings index-mappings-logservice index-mappings-app-logservice index-mappings-platform-logservice'
+ for template in '$(curl -s ${URL}/_cat/templates | awk '\''{print $1}'\'')'
+ grep -qw shards-and-replicas /dev/fd/63
++ echo 'shards-and-replicas index-settings index-mappings index-mappings-logservice index-mappings-app-logservice index-mappings-platform-logservice'
+ for template in '$(curl -s ${URL}/_cat/templates | awk '\''{print $1}'\'')'
+ grep -qw index-mappings-app-logservice /dev/fd/63
++ echo 'shards-and-replicas index-settings index-mappings index-mappings-logservice index-mappings-app-logservice index-mappings-platform-logservice'
+ for template in '$(curl -s ${URL}/_cat/templates | awk '\''{print $1}'\'')'
+ grep -qw index-settings /dev/fd/63
++ echo 'shards-and-replicas index-settings index-mappings index-mappings-logservice index-mappings-app-logservice index-mappings-platform-logservice'
+ for template in '$(curl -s ${URL}/_cat/templates | awk '\''{print $1}'\'')'
+ grep -qw index-mappings /dev/fd/63
++ echo 'shards-and-replicas index-settings index-mappings index-mappings-logservice index-mappings-app-logservice index-mappings-platform-logservice'
+ for template in '$(curl -s ${URL}/_cat/templates | awk '\''{print $1}'\'')'
+ grep -qw .ml-anomalies- /dev/fd/63
++ echo 'shards-and-replicas index-settings index-mappings index-mappings-logservice index-mappings-app-logservice index-mappings-platform-logservice'
+ curl -XDELETE http://172.17.8.37:9200/_template/.ml-anomalies-
{"acknowledged":true}+ for template in '$(curl -s ${URL}/_cat/templates | awk '\''{print $1}'\'')'
+ grep -qw security_audit_log /dev/fd/63
++ echo 'shards-and-replicas index-settings index-mappings index-mappings-logservice index-mappings-app-logservice index-mappings-platform-logservice'
+ curl -XDELETE http://172.17.8.37:9200/_template/security_audit_log
{"acknowledged":true}+ for template in '$(curl -s ${URL}/_cat/templates | awk '\''{print $1}'\'')'
+ grep -qw .ml-notifications /dev/fd/63
++ echo 'shards-and-replicas index-settings index-mappings index-mappings-logservice index-mappings-app-logservice index-mappings-platform-logservice'
+ curl -XDELETE http://172.17.8.37:9200/_template/.ml-notifications
{"acknowledged":true}+ for template in '$(curl -s ${URL}/_cat/templates | awk '\''{print $1}'\'')'
+ grep -qw index-mappings-platform-logservice /dev/fd/63
++ echo 'shards-and-replicas index-settings index-mappings index-mappings-logservice index-mappings-app-logservice index-mappings-platform-logservice'
+ for template in '$(curl -s ${URL}/_cat/templates | awk '\''{print $1}'\'')'
+ grep -qw .ml-meta /dev/fd/63
++ echo 'shards-and-replicas index-settings index-mappings index-mappings-logservice index-mappings-app-logservice index-mappings-platform-logservice'
+ curl -XDELETE http://172.17.8.37:9200/_template/.ml-meta
{"acknowledged":true}+ for template in '$(curl -s ${URL}/_cat/templates | awk '\''{print $1}'\'')'
+ grep -qw logstash-index-template /dev/fd/63
++ echo 'shards-and-replicas index-settings index-mappings index-mappings-logservice index-mappings-app-logservice index-mappings-platform-logservice'
+ curl -XDELETE http://172.17.8.37:9200/_template/logstash-index-template
{"acknowledged":true}+ for template in '$(curl -s ${URL}/_cat/templates | awk '\''{print $1}'\'')'
+ grep -qw .ml-state /dev/fd/63
++ echo 'shards-and-replicas index-settings index-mappings index-mappings-logservice index-mappings-app-logservice index-mappings-platform-logservice'
+ curl -XDELETE http://172.17.8.37:9200/_template/.ml-state
{"acknowledged":true}+ curl -f 'http://172.17.8.37:9200/_cluster/health?wait_for_status=yellow&timeout=10m'
{"cluster_name":"logsearch","status":"green","timed_out":false,"number_of_nodes":7,"number_of_data_nodes":3,"active_primary_shards":854,"active_shards":1708,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":3,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":15,"active_shards_percent_as_number":100.0}+ URL=http://172.17.8.37:9200/_template/shards-and-replicas
+ CONTENT_PATH=/var/vcap/jobs/elasticsearch_config/index-templates/shards-and-replicas.json
++ curl -s -H 'Content-Type: application/json' -w '%{http_code}' -o /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt -X PUT --data-binary @/var/vcap/jobs/elasticsearch_config/index-templates/shards-and-replicas.json http://172.17.8.37:9200/_template/shards-and-replicas
+ HTTP_CODE=200
+ grep --quiet errors:true /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
+ '[' 200 -le 300 ']'
+ cat /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
{"acknowledged":true}+ URL=http://172.17.8.37:9200/_template/index-settings
+ CONTENT_PATH=/var/vcap/jobs/elasticsearch_config/index-templates/index-settings.json
++ curl -s -H 'Content-Type: application/json' -w '%{http_code}' -o /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt -X PUT --data-binary @/var/vcap/jobs/elasticsearch_config/index-templates/index-settings.json http://172.17.8.37:9200/_template/index-settings
+ HTTP_CODE=200
+ grep --quiet errors:true /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
+ '[' 200 -le 300 ']'
+ cat /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
{"acknowledged":true}+ URL=http://172.17.8.37:9200/_template/index-mappings
+ CONTENT_PATH=/var/vcap/jobs/elasticsearch_config/index-templates/index-mappings.json
++ curl -s -H 'Content-Type: application/json' -w '%{http_code}' -o /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt -X PUT --data-binary @/var/vcap/jobs/elasticsearch_config/index-templates/index-mappings.json http://172.17.8.37:9200/_template/index-mappings
+ HTTP_CODE=200
+ grep --quiet errors:true /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
+ '[' 200 -le 300 ']'
+ cat /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
{"acknowledged":true}+ URL=http://172.17.8.37:9200/_template/index-mappings-logservice
+ CONTENT_PATH=/var/vcap/jobs/logservice-logsearch-filters/index-mappings.json
++ curl -s -H 'Content-Type: application/json' -w '%{http_code}' -o /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt -X PUT --data-binary @/var/vcap/jobs/logservice-logsearch-filters/index-mappings.json http://172.17.8.37:9200/_template/index-mappings-logservice
+ HTTP_CODE=200
+ grep --quiet errors:true /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
+ '[' 200 -le 300 ']'
+ cat /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
{"acknowledged":true}+ URL=http://172.17.8.37:9200/_template/index-mappings-app-logservice
+ CONTENT_PATH=/var/vcap/jobs/logservice-logsearch-filters/index-mappings-app.json
++ curl -s -H 'Content-Type: application/json' -w '%{http_code}' -o /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt -X PUT --data-binary @/var/vcap/jobs/logservice-logsearch-filters/index-mappings-app.json http://172.17.8.37:9200/_template/index-mappings-app-logservice
+ HTTP_CODE=200
+ grep --quiet errors:true /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
+ '[' 200 -le 300 ']'
+ cat /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
{"acknowledged":true}+ URL=http://172.17.8.37:9200/_template/index-mappings-platform-logservice
+ CONTENT_PATH=/var/vcap/jobs/logservice-logsearch-filters/index-mappings-platform.json
++ curl -s -H 'Content-Type: application/json' -w '%{http_code}' -o /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt -X PUT --data-binary @/var/vcap/jobs/logservice-logsearch-filters/index-mappings-platform.json http://172.17.8.37:9200/_template/index-mappings-platform-logservice
+ HTTP_CODE=200
+ grep --quiet errors:true /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
+ '[' 200 -le 300 ']'
+ cat /var/vcap/data/elasticsearch_config/elasticsearch_config-response.txt
{"acknowledged":true}