managing index templates in elasticsearch

[WassimDhib]

Hi,

When using this jmeter plugin, I noticed that the mapping in elasticsearch indexes was not consistent. Sometimes date fields are mapped to long type.

 "Timestamp" : {
          "type" : "long"
        },

I suggest to use an index template to define mapping explicitly.

 PUT _template/jmeter-2.6.10
  {
    "index_patterns":[
         "jmeter-2.6.10-*"
      ],
    "aliases" : { },
    "mappings" : {
      "properties" : {
        "AllThreads" : {
          "type" : "long"
        },
        "BodySize" : {
          "type" : "long"
        },
        "Bytes" : {
          "type" : "long"
        },
        "ConnectTime" : {
          "type" : "long"
        },
        "ContentType" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "DataType" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "ElapsedTime" : {
          "type" : "date"
        },
        "ErrorCount" : {
          "type" : "long"
        },
        "FailureMessage" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "GrpThreads" : {
          "type" : "long"
        },
        "IdleTime" : {
          "type" : "long"
        },
        "InjectorHostname" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "Latency" : {
          "type" : "long"
        },
        "ResponseCode" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "ResponseTime" : {
          "type" : "long"
        },
        "SampleCount" : {
          "type" : "long"
        },
        "SampleEndTime" : {
          "type" : "date"
        },
        "SampleLabel" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "SampleStartTime" : {
          "type" : "date"
        },
        "SentBytes" : {
          "type" : "long"
        },
        "Success" : {
          "type" : "boolean"
        },
        "TestElement" : {
          "properties" : {
            "name" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "TestStartTime" : {
          "type" : "date"
        },
        "ThreadName" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "Timestamp" : {
          "type" : "date"
        },
        "URL" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }
    },
    "settings" : {
      "index" : {
        "number_of_shards" : "1",
        "number_of_replicas" : "1"
        }
    }
}

[anthonygauthier]

Hi @WassimDhib,

The field timestamp should technically always be a long - it's a timestamp in millisecond. For a full list of types used when sending the documents to the index:

https://github.com/delirius325/jmeter-elasticsearch-backend-listener/blob/master/src/main/java/io/github/delirius325/jmeter/backendlistener/elasticsearch/ElasticSearchMetric.java#L52-L101

As well as JMeter's documentation:

https://jmeter.apache.org/api/org/apache/jmeter/samplers/SampleResult.html

[WassimDhib]

Yes, from the point of view of jmeter, the timestamp is always a long But, if you don't provide mapping information, elasticsearch won't be able to map this field to a date (the behavior may be inconsistent).

from elasticsearch docs :

You know more about your data than Elasticsearch can guess, so while dynamic mapping can be useful to get started, at some point you will want to specify your own explicit mappings.

It's a problem since your time series require a field with date type to create charts/visualizations

[anthonygauthier]

Are you using Kibana or Grafana to generate your dashboard? Because I have been using Grafana with the dynamic mapping and never really had any issues.

[WassimDhib]

I use kibana

[anthonygauthier]

kibana being generally less aimed towards time series graphs, I'd suggest using Grafana to create your dashboards.

As for the change, you are more than welcome to suggest the changes through a pull request :smile:

[bratwurzt]

I had the same problem, solved it with a template. Would like to point out that while kibana is geared towards time series, its grouping/bucketing of graphs is quite ahead of Graphana.

[WassimDhib]

It's not only a kibana problem, since the mapping is also responsible of optimizing the disk space allocated to indices in elasticsearch . You can also specify index settings in template to improve indexing speed (number of primary shards, refresh interval ... ).

I'll take the time to make a PR to be able to manage templates.

[gholtzhausen]

Thanks, the timeout issue is resolved. What would be the way to resolve the timestamp as it is displayed in ms to be used in Kibana? E.G. currently "Timestamp": 1590995355655 Can we get the same format - yyyy-MM-dd'T'HH:mm:ss.SSSZZ which would also apply to "ElapsedTime": 1577829723000, "TestStartTime": 1590995232368,

[clefego]

I'm new with ELK. I have already activated the k8s pods on ELK and Grafana. I have already created a test plan with the plugin. Grafana can communicate with ELK, but I don't know how to import this model index. Could someone help me?