anilganipineni
commented
5 years ago 
Closed anilganipineni closed 5 years ago
anilganipineni
commented
5 years ago
anilganipineni
commented
5 years ago @sivasamyk , even after adding the "keyword_suffix" : "" to logtrail.josn, no luck.
Here is my logtrail.json configuration.
{
"version" : 2,
"index_patterns" : [
{
"es": {
"default_index": "logstash-*",
"allow_url_parameter": false
},
"tail_interval_in_seconds": 10,
"es_index_time_offset_in_seconds": 0,
"display_timezone": "local",
"display_timestamp_format": "YYYY-MMM-DD HH:mm:ss",
"max_buckets": 500,
"default_time_range_in_days" : 0,
"max_hosts": 100,
"max_events_to_keep_in_viewer": 5000,
"default_search": "",
"fields" : {
"mapping" : {
"timestamp" : "@timestamp",
"display_timestamp" : "@timestamp",
"hostname" : "host",
"program": "program",
"message": "log_message"
},
"message_format": "[GSP] : {{{log_message}}}",
"keyword_suffix" : ""
},
"color_mapping" : {
"field" : "log_level",
"mapping" : {
"ERROR": "#ff3232",
"WARN": "#ff7f24",
"DEBUG": "#ffb90f",
"TRACE": "#a2cd5a"
}
}
}
]
}@anilganipineni Can you please try the solution mentioned here - #347 (comment) ?
I tried, but no luck, can you please help
sivasamyk
commented
5 years ago Can you please share a sample ES document from logstash index? Also please share the mapping for the index.
On Wed, Apr 17, 2019 at 16:26 Anil Ganipineni notifications@github.com wrote:
@anilganipineni https://github.com/anilganipineni Can you please try the solution mentioned here - #347 (comment) https://github.com/sivasamyk/logtrail/issues/347#issuecomment-477927336 ?
I tried, but no luck, can you please help
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/sivasamyk/logtrail/issues/351#issuecomment-484036214, or mute the thread https://github.com/notifications/unsubscribe-auth/ABZXet3zI3jNNTVG5ln-PLnZ6oirkQpzks5vhv3cgaJpZM4c0uHG .
-- Regards, Siva.
anilganipineni
commented
5 years ago Mapping :
"logstash-2019-04": {
"mappings": {
"_default_": {
"dynamic_templates": [
{
"message_field": {
"path_match": "message",
"match_mapping_type": "string",
"mapping": {
"norms": false,
"type": "text"
}
}
},
{
"string_fields": {
"match": "*",
"match_mapping_type": "string",
"mapping": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"norms": false,
"type": "text"
}
}
}
],
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "keyword"
},
"geoip": {
"dynamic": "true",
"properties": {
"ip": {
"type": "ip"
},
"latitude": {
"type": "half_float"
},
"location": {
"type": "geo_point"
},
"longitude": {
"type": "half_float"
}
}
}
}
},
"doc": {
"dynamic_templates": [
{
"message_field": {
"path_match": "message",
"match_mapping_type": "string",
"mapping": {
"norms": false,
"type": "text"
}
}
},
{
"string_fields": {
"match": "*",
"match_mapping_type": "string",
"mapping": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"norms": false,
"type": "text"
}
}
}
],
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "keyword"
},
"agent": {
"properties": {
"ephemeral_id": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"hostname": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"id": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"type": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"version": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"ecs": {
"properties": {
"version": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"geoip": {
"dynamic": "true",
"properties": {
"ip": {
"type": "ip"
},
"latitude": {
"type": "half_float"
},
"location": {
"type": "geo_point"
},
"longitude": {
"type": "half_float"
}
}
},
"host": {
"properties": {
"name": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"input": {
"properties": {
"type": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"log": {
"properties": {
"file": {
"properties": {
"path": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"flags": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"offset": {
"type": "long"
}
}
},
"log_message": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"message": {
"type": "text",
"norms": false
},
"tags": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}Sample Document:
{
"_index": "logstash-2019-04",
"_type": "doc",
"_id": "Iv-0KWoBRPHNe7j-VN0F",
"_score": 1,
"_source": {
"@version": "1",
"tags": [
"beats_input_codec_plain_applied"
],
"log": {
"file": {
"path": "/var/log/nginx/access.log"
},
"offset": 2677219
},
"log_message": "[17/Apr/2019:10:40:43 +0530] [200] [612] [POST /gsp/authenticate?grant_type=token HTTP/1.1] [-]-[-]-[-] 172.25.10.73 - 1315A57FD01440C8AEEBDA957B0D3E70 \"-\" \"Java/1.7.0_21\" \"203.175.185.56\"",
"agent": {
"type": "filebeat",
"id": "14edbcf2-aa91-47d3-bac5-e7a49ae505b6",
"hostname": "ip-172-25-1-12.ap-south-1.compute.internal",
"version": "7.0.0",
"ephemeral_id": "2afdf83d-c916-45d5-8104-40ab8267ec14"
},
"host": {
"name": "ip-172-25-1-12.ap-south-1.compute.internal"
},
"@timestamp": "2019-04-17T05:10:43.900Z",
"message": "[17/Apr/2019:10:40:43 +0530] [200] [612] [POST /gsp/authenticate?grant_type=token HTTP/1.1] [-]-[-]-[-] 172.25.10.73 - 1315A57FD01440C8AEEBDA957B0D3E70 \"-\" \"Java/1.7.0_21\" \"203.175.185.56\"",
"ecs": {
"version": "1.0.0"
},
"input": {
"type": "log"
}
}
}I see your ES doc has 2 fields hostname and a nested field host.name. You logtrail.json uses host as mapping for hostname. Please use either of the above fields.
anilganipineni
commented
5 years ago Hi @sivasamyk ,
I tried, host.name , agent.hostname, host and hostname. None of the things worked for me.
Error when i use host.name: Cannot fetch hosts : [illegal_argument_exception] Fielddata is disabled on text fields by default. Set fielddata=true on [host.name] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
Error when i use agent.hostname: Cannot fetch hosts : [illegal_argument_exception] Fielddata is disabled on text fields by default. Set fielddata=true on [agent.hostname] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
Just want to highlight regarding host and hostname in ES doc. Both are like below.
"agent": { "hostname": "ip-172-25-1-11.ap-south-1.compute.internal", "type": "filebeat", "ephemeral_id": "0a2889bd-5cd7-4504-a9aa-1acfbde749ec", "id": "2b0dcc53-e994-42c3-af71-4a104a349256", "version": "7.0.0" }
"host": { "name": "ip-172-25-1-11.ap-south-1.compute.internal" }
anilganipineni
commented
5 years ago @sivasamyk , Please let me know if required more details / Please suggest the configuration which i need to check
sivasamyk
commented
5 years ago What is the value present in keyword_suffix field of logtrail.json? It should be keyword.
anilganipineni
commented
5 years ago @sivasamyk , Perfect, now it's working for me.
Thank You!
anilganipineni
commented
5 years ago One more question
Can I remove time stamp in the log trail out put as My log is already having the time of application??
sivasamyk
commented
5 years ago No. You cannot remove the timestamp in logtrail. You can strip the timestamp from actual message using logstash filters.
@anilganipineni Can you please try the solution mentioned here - https://github.com/sivasamyk/logtrail/issues/347#issuecomment-477927336 ?